Access Key-vault in notebooks

[AdarshPanasri]

Hello Everyone, I have few secrets in my Key vault and want to access those in fabric notebook. I dont want users to have access to key vault. I can create SPN and use that to access the vault but for that I would need to authenticate using the credentials.

I am aware I can use the below utility but how does the authentication work in backend, do we need to provide some level of permissions on KV ?

mssparkutils.credentials.getSecret('https://<name>.vault.azure.net/', 'secret name')

[do-sc]

We now have the option to use a service principal! Fabric Notebook: Key Vault Access with Service Principal

[x_mark_x]

@do-sc , this is a great article but even after following the steps to the T, i still get a "403 Caller is not authorized to perform action on resource" error when running the notebook in the pipeline.
Do you have any suggestions on how to solve/troubleshoot this issue?

[do-sc]

Oh hi Mark!

I assume you have read the section about troubleshooting in the blog? Sometimes it can take a while to reflect the permissions from the Key Vault. Double check the permission level of the Service Principal on the Key Vault and ensure that the Pipeline is using the Notebook connection of your Service Principal. If you run the Notebook in the context of your user without having given additional permissions it can also result in the 403. Just beeing the owner on the Key Vault will not allow you to get a secret!

If it is still not working I would try to call the Key Vault with your Service Principal credentials directly to see if you could get through. Maybe some other policy like a firewall is preventing you from accessing the secret and everything else is set up correctly: https://learn.microsoft.com/en-us/azure/key-vault/general/developers-guide#authenticate-to-key-vault...

[x_mark_x]

@do-sc , thank you for your reply.

After some further trials and errors, I got it working based on the article mentioned above.

I noticed that while I followed every step, I used my own code for the retrieval and more precisely I used the notebookutils.mssparkutils.credentials.getSecret method instead of using 

notebookutils.credentials.getSecret.

As mssparkutils is an older library, and notebookutils practically replaced it (https://learn.microsoft.com/en-us/fabric/data-engineering/notebook-utilities), I guess that the getSecret method got an update as well, so it can use the custom connections, while the old version does not support that feature.

Anyways, thank you for your help 🙂

[jagdishwagh0909]

For access keyvault follow below blog. I have create one service priciple and followed below steps mention in blog and access the keyvault everything working fine as we exptected. Might be Fabric will come with better approach in future. 
Reading and Writing Secrets in Azure Key Vault Using Microsoft Fabric Notebooks

[Harshadeep21]

Access Azure Key vault secrets from MS Fabric Notebooks by Harshadeep 

This post might be helpful

[jagdishwagh0909]

In how we can proceed with this, Currently I have extract data from API and for connectivity we are using userid and password but we cant not put those details in notebook as hardcodeed. We need keyvault feature as soon as possible otherwise its major showstopper for many use cases. 

[PanuO]

Well currently you have to give key vault access to the Owner of the notebook, then you can use notebookutils.credentials.getSecret

[umeshr]

This may not be possible as organization policy, individual access to Key Vault is restricted. 

[jagdishwagh0909]

I have tested that workspace identity works only for ADLS connectivity and not with Key Vault. Additionally, I am not looking to use external libraries for accessing Key Vault in Fabric. In the blog mentioned above, they are using the external Trident library to access Key Vault, but from a security perspective, I believe it's not a good idea to rely on such libraries.

Please feel free to correct me if I am mistaken.

[PanuO]

So summary of this: You cannot currently use Workspace Identity to grant access into Azure Key Vault. Well you can, but nothing supports it. You cannot run Notebook with Workspace Identity. Hope this feature would come quickly as this would improve the security of Notebooks.

[Anonymous]

Hi @AdarshPanasri ,

Any update on this? Did the above suggestions help with your scenario? if that is the case, you can consider Kudo or Accept the helpful suggestions to help others who faced similar requirements.

If these also don't help, please share more detailed information and description to help us clarify your scenario to test.

How to Get Your Question Answered Quickly 

Regards,

Xiaoxin Sheng

[APG]

As an alternative to SPNs you can use Workspace Identity to authenticate. You will need to provide some permissions on the key vault to the workspace identity, for example:

• the "Key Vault Secrets User" role if your key vault is configured to use RBAC
• or an access policy with get/list permissions on secrets if your key vault is configured to use access policies

If you don't want to use workspace identities you could also authenticate with the user that created the notebooks; of course you still need to grant roles/access policies to that user in the key vault

[frithjof_v]

Very interesting! That is really exciting. Have you tested it? I thought workspace identity was only for authenticating to ADLS at the moment, ref. the docs you linked to:

"Fabric workspaces with a workspace identity can securely read or write to firewall-enabled Azure Data Lake Storage Gen2 accounts through trusted workspace access for OneLake shortcuts. In the future, Fabric items will be able to use the identity when connecting to resources that support Microsoft Entra authentication."

Thanks.

[mrtnhd]

I am also interested in this. Would it be possible to provide some example code of how I use the workspace identity to access an Azure key vault in a Fabric notebook?

[AndyDDC · Accepted solution]

Hi @AdarshPanasri @can you try this blog and see if it fits your scenario? https://www.syntera.ch/blog/2023/10/18/how-to-access-azure-key-vault-secrets-from-fabric-notebook/

[umeshr]

This is great stuff to use key vault using individua's id to connect to key vault.

Still awaiting for solution to connect to key vault using Fabric's workspace identity.

Please let us know in case there is any solution available.