Azure Key Vaults for secret or credential protection at Fabric Notebooks

[ibarrau]

What is the Azure Key Vault service?

In Microsoft’s own words: “Azure Key Vault is a cloud service for securely storing and accessing secrets. A secret is anything that you want to tightly control access to, such as API keys, passwords, certificates, or cryptographic keys.”

Although the service supports various operations, we’ll focus on the concept that it allows us to store a key or password in an encapsulated form. Only users with read access to the keys in the service can use this encapsulated value. This way, we ensure that only approved users can use it. Let’s see how to create this secret before using it in a Fabric Notebook.

Creating the service is quite simple, you just select the subscription, resource group, name, region, and plan:

As mentioned earlier, the service includes many more capabilities than we’ll use here. We’ll now focus on “Secrets,” which is what we’re interested in.

Here, for example, we already have three secrets that we’ll use to connect to the Power BI REST API. We’ve saved secrets for the tenant ID, app ID, and secret value of our app registered in Azure.

Let’s see how to create a new one. It’s as simple as giving it a name and defining what we want to encapsulate. We can also define it as temporary if desired:

That’s how we create a new secret in our key vault. What comes next? We must allow read access to whoever is going to use it.

Azure resources use RBAC (Role-Based Access Control) permissions. You can find these in the “Access Control (IAM)” section.
Open the permissions menu and add the role called “Key Vault Secrets User.” The Entra ID account (formerly Azure AD) with that permission will be able to call the stored secret from code.

This process is very important. Imagine being able to grant a developer permission to build a process without ever knowing the original credentials.

How to call it from Fabric

To use this service from a Fabric Notebook with Python, we’ll take advantage of Microsoft’s library that provides many convenient interaction features.
You can read more details here: Microsoft Spark Utilities documentation

Inside our notebook, we’ll start by importing SimplePBI to connect to the Power BI REST API. Then we’ll import the necessary libraries. The key part is in our third cell, here we’ll see how to call the secret we just stored:

We use the getSecret method, which requires two parameters: First "Vault URI" found in the Overview of your Azure Key Vault resource. Second, the name of the secret you defined earlier.

mssparkutils.credentials.getSecret('https://casa.vault.azure.net/', 'TenantId')

Nowadays Microsoft is recommending moving to a different library. It's just NotebookUtils. The code is almost the same:

notebookutils.credentials.getSecret('https://casa.vault.azure.net/', 'TenantId')

In this way, we store the result in a variable and can continue the API authentication process in the following lines, creating a token and requesting the top 5 workspaces. Remember that our focus here is security: not only avoiding exposing passwords in code but also preventing anyone from viewing the secret’s content. If a developer tries to read the variable directly, they’ll face restrictions:

* NOTE: This third cell (where the secret is requested) can only be executed by a Fabric-logged account with the “Key Vault Secrets User” role in our Key Vault. Otherwise, it will fail due to forbidden access.

If you want to improve the security even more, consider configuring a private endpoint between the workspace and the Key Vault. This step improves security by ensuring traffic stays within the private network instead of going over the public internet.

I hope this helps you automate workflows more securely using Fabric Notebooks.

[nielsvdc]

Hi @ibarrau,

Firstly, in your code samples you’re still using mssparkutils. Why not switch to notebookutils? It’s the newer approach, automatically available in Fabric notebooks without any extra imports, and it represents the future direction for notebook development. See the NotebookUtils documentation.

Secondly, when explaining how to set up Azure Key Vault and use it in Fabric, consider adding details about configuring a private endpoint between the workspace and the Key Vault. This step significantly improves security by ensuring traffic stays within the private network instead of going over the public internet.

[ibarrau]

@nielsvdc thanks for sharing your ideas. NotebookUtils was mentioned in the doc I have shared and I have added now an additional line of code showing how to get secret with that one too. Regarding private endpoint, I think it could be a different post. I wanted to keep it simple. I have shared your idea at the article saying if the want even more security they can read about that.

Regards

[Srisakthi]

Hi @ibarrau ,

Great article! One additional point to consider: Fabric currently supports connecting to an Azure Key Vault that is publicly accessible. However, many customers are unlikely to accept having their Key Vault exposed publicly.

Regards,

Srisakthi