Fabric - Airflow Jobs - Notebook Access to Key Vault -mssparkutils.credentials.getSecret returns 403

[kaouter]

Hi all,

I've been hitting a 403 Forbidden error when calling mssparkutils.credentials.getSecret from a Fabric notebook triggered via Apache Airflow. The error message includes a caller appid (caxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx) that I can't find anywhere in Entra ID : not in App Registrations, not in Enterprise Applications.

Here's the relevant part of the error:

Caller: appid=caxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx;oid=4exxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx Action: Microsoft.KeyVault/vaults/secrets/getSecret/action innererror: ForbiddenByRbac

I decoded the JWT token used for the Key Vault call and confirmed idtyp: app — so it's not a user identity. But it doesn't match the Workspace Identity either (different appid).

My understanding so far: Fabric generates a spark runtime identity automatically when a notebook runs, especially when triggered externally via Airflow. This identity isn't directly visible in Entra ID, which makes RBAC assignment tricky.

Has anyone run into this? 🤔

Worth noting : Airflow triggers the notebook by calling the Fabric API using a SPN. That SPN has enough permissions to kick off the job, but apparently that's where its authority ends — the identity that actually runs inside the notebook and tries to read from Key Vault is something else entirely. So even if your SPN is correctly configured, the notebook runtime uses its own identity for downstream calls, and that identity has no Key Vault access.
@deborshi_nag @NaveenUpadhye  @tayloramy  @v-nmadadi-msft @Olufemi7 @Lodha_Jaydeep @ssrithar @arabalca @Shreya_Barhate 

[tayloramy]

Hi @kaouter, 

For that suggestion, I recommend you post an Idea and see what happens.  https://community.fabric.microsoft.com/t5/Fabric-Ideas/idb-p/fbc_ideas

If it gains traction here then the product team will notice it. 

[tayloramy · Accepted solution]

Hi @kaouter, 

As we discussed on Discord, Fabric uses the security context of the identity that calls the API to trigger the notebook to access the keyvault. 

In your case, when you have an airflow SPN triggering the notebook, that SPN is the identity that needs permissions on the keyvault for the notebook to be able to successfully authenicate to the keyvault. 

It was great connecting with you on Discord and being able to help! 

[kaouter]

Hi everyone,
Quick update to close this out.

We ended up granting Key Vault permissions directly to the SPN that Airflow uses to trigger the notebook. Once the SPN had the Key Vault Secrets User role assigned, everything worked.
Thanks to everyone who jumped in on this thread, and a special shoutout to Taylor, your root cause breakdown was really helpful.
One thing I’d love to see from Microsoft in the future: when a notebook is triggered externally via a SPN, it should be possible to configure it to run under the Workspace Managed Identity instead. This would be the cleaner approach from a security standpoint : Airflow stays responsible for orchestration only, the SPN never touches the data or the secrets, and the notebook runs under a Workspace Identity that already has the right Key Vault permissions. Clean separation between the orchestration layer and the data layer.
Right now that separation isn’t possible when triggering notebooks directly via the Fabric API, which forces you to give Key Vault access to the orchestration SPN (not ideal).
Hope this thread helps someone else. Thanks again everyone!​​​​​​​​​​​​​​​​

@tayloramy 😉

[v-ssriganesh]

Hi @kaouter,

Thank you for posting your query in the Microsoft Fabric Community Forum, and thanks to @tayloramy for sharing valuable insights.

Could you please confirm if your query has been resolved by the provided solutions? This would be helpful for other members who may encounter similar issues.

Thank you for being part of the Microsoft Fabric Community.

[tayloramy]

Hi @ssrithar, 

---

Instead, Fabric generates a session-specific identity that has no Key Vault permissions.

Are you able to provide a source for this behaviour? THis is the first time I've heard of Fabric creating it's own identities. A documentation link would be helpful to verify this. 

[ssrithar]

Hi @tayloramy ,

Please refres to the below links for the details from where I was able to provide the details

https://community.fabric.microsoft.com/t5/Data-Science/How-to-Access-Azure-Key-Vault-Secrets-in-Fabr...

https://peerinsights.emono.dk/whos-calling

https://www.syntera.ch/blog/2023/10/18/how-to-access-azure-key-vault-secrets-from-fabric-notebook/

Please let me know if further details are required on the same

[tayloramy]

Hi @ssrithar, 

None of those links make any reference to Fabric creating new session specific identities for auth. 

Please fact check before you post content on the forums, especially if it is generated from an LLM.  Spreading misinformation is not helpful for the community. 

[tayloramy]

Hi @kaouter, 

Notebooks will run under the context of the current user, or the notebook author if it is scheduled. 
If it is run with a pipeline, then it can run under a service principal or workspace identity I believe. 

Make sure the notebook creator has permissions on the keyvault. Also make sure the keyvault is either open to the public internet, or you have created a private endpoint from the fabric workspace to the keyvault. 

[kaouter]

Hi, thanks for the response!
The notebook is triggered externally via Apache Airflow, not through a Fabric Pipeline or the built-in scheduler. Airflow calls the Fabric API using a SPN to start the job. So the “notebook author” context doesn’t seem to apply here — the identity that shows up in the 403 error is neither my user account nor the SPN Airflow uses to trigger it.
That’s the confusing part. The caller appid in the token doesn’t match anything in Entra ID — not the triggering SPN, not the Workspace Identity. It looks like Fabric generates a separate runtime identity for the Spark session itself when the notebook is triggered externally.

WN: executing notebook manually or with pipeline or scheduler in fabric works perfects but i need that my notebook get the key vault when it is triggered by apache airflow job in fabric !!!
Has anyone specifically documented how identity works when Fabric notebooks are triggered via external API calls rather than Pipelines or the scheduler?