pass KeyVault values OUT OF Notebook to pipeline to consume - securely

[Maverikk]

I have a fabric pipeline that initiates a Notebook that calls to KeyVault to return a bearer_token, this is [REDACTED], but when I use mssparkutils.notebook.exit in my Notebook it passes out the string [REDACTED], NOT the value.  Using a Web activity isnt acceptable as it is using Service Principal, whereas the Notebook uses Workspace Identity, which is more secure.

What is the recommended approach to securely use the keyvault in a pipeline?  I dont have to use a Notebook if there is a better methodology.

[apturlov]

@Maverikk from your explanation you are using a data pipeline (Copy activity, etc.) to retrieve data from a commercial Web API. I assume that the Web API requires authentication and because you mention a Bearer token I also assume it is a JWT token. Another assumption is you need an API key for authentication which is a secret value. It absolutely makes sense to want to store the API key in Azure KeyVault. You can use Web Activity in a pipeline to connect to AKV and retrieve a secret during a pipeline execution, which is currently a recommended approach in a data pipeline in Microsoft Fabric.

So, your data pipeline logic will look like this:

1. Web activity: retrieve a secret value for API key from Azure Key Vault using a service principla authentication. Service principal credentials for such connection will be stored in Fabric in a secure way. Use the secret value for activity output as a SecureString to avoid exposure of the sensitive value in the pipeline logs.

2. Web activity: acquire an authentication token from your API authentication point using the API key from the previous activity output. Because the outpu is secure the API key won't be visible or exposed. Return the token as activity output as SecureString. Use this activity output in an Authorization header in the next Copy activity.

3. Copy activity: retrieve data from your API endpoint using a Bearer token from the previous activity.

This approach is well known and well tested pattern in data pipeline in both Azure Data Factory and Microsoft Fabric. Feel free to ask more questions.

[apturlov]

Hi @Maverikk Microsoft Fabric specifically prevent any secrets obtained in a Notebook to be leaked outside of the Notebook. You can read more about that technique here https://learn.microsoft.com/en-us/fabric/data-engineering/author-execute-notebook#secret-redaction.

This answer does not solve your questions, unfortunately, but this is not a problem with your solution but a current technical limitation in Fabric. Fabric allows you to create a Key Vault reference but it's not supported in data pipelines and it also does not support yet a private networking for Azure Key Vaults.

So, there is no generic recommended approach yet for your question. We maight be able to find a solution for your specific scenario, but you need to provide an explanation on why do you need to use a secret in a pipeline.
I personally always start from trying to avoid using secrets in the data pipeline in Fabric.

Hopefully that explains the situation even though does not answer your question. Feel free to continue this discussion but please explain your scenario.

[Maverikk]

Thankyou @apturlov for taking the time to reply.  I am trying to securely store my API Bearer Token inside KeyVault, and then access and use it inside a CopyActivity for an API call.  The API call holds commercially sensitive data, so naturally I want to keep the "secret" under wraps.  Perhaps you can suggest a way that you would approach this, maybe there is a better methodology than I am using?  Thankyou for your time

[BhaveshPatel]

Hi @Maverikk 

You should think about Notebooks because everybody in the world uses Linux System. Use Spark and bit of study..

Python --> Data Lake ( Aache Spark ) --> Delta Lake.  It is just using Python, and it is open source.

[Maverikk]

Thanks @BhaveshPatel , I am not sure you have answered my question tbh.  I am saying that I AM using a Notebook and Python, but am struggling to pass the [REDACTED] OUT of the Notebook and back into a Fabric pipeline.

[v-tejrama]

Hi @Maverikk ,

You’re absolutely right to point out that the issue isn’t about using notebooks or Spark in general, but about how to securely pass a value retrieved from Key Vault out of a Fabric notebook so the pipeline can use it. The behavior you’re seeing where the value is replaced with [REDACTED] is expected, as Fabric intentionally masks sensitive information to prevent secrets from being exposed in pipeline logs or variables.

This means values obtained using Workspace Identity inside a notebook can’t be passed directly to a pipeline output. The secure and supported approach is to use the secret within the same notebook where it’s retrieved, or have each pipeline activity that needs it access Key Vault directly through a managed identity or linked service. That way, the secret never leaves a protected environment and your pipeline remains fully compliant with Fabric’s security model.

Best Regards,
Tejaswi.
Community Support

[Maverikk]

thankyou @v-tejrama for your response and your input.  Can I ask what would be the appropriate way inside Fabric for me to retrieve a Bearer Token for an API call (from Keyvault), and then use that Bearer Token for a Copy Activity [in a pipeline]?  You have mentioned: "pipeline activity that needs it access Key Vault directly through a managed identity", I am not clear how this can be achieved with a copy activity. Ideally I would NOT be using a Notebook to retrieve the values as I am trying to reduce cu's, where not needed. Perhaps I am missing something?  Thankyou for your time. 

[v-tejrama]

Hi @Maverikk ,

That’s an excellent follow-up question, and your focus on secure handling and minimizing CU usage is well placed. In Fabric, it is essential to ensure that secrets and tokens remain within secure contexts and are not written to logs. To avoid using a notebook, I recommend configuring the pipeline to manage both Key Vault retrieval and API authorization directly. You can achieve this by adding a Web activity before the Copy Activity to obtain the token from your authentication endpoint or Key Vault using a managed identity.

The token output from the Web activity can then be referenced in the Copy Activity’s Authorization header, for example, using activity GetToken output.access_token. This method ensures the token is securely used at runtime and not exposed or stored in plain text. If your environment allows, grant the pipeline’s managed identity access to Key Vault secrets, enabling direct retrieval without a notebook. This approach maintains security, reduces CU costs, and aligns with Fabric’s best practices for secret management.

Thank you.

[Maverikk]

Thankyou for your lengthy response @v-tejrama , however, my "Cloud & AI - Data Solution Engineer" at microsoft has advised that "Pipelines don’t support Workspace Identity for Web Activity calls whereas Dataflow Gen2 support." So whilst I agree in principal with your pipeline flow, I still do not see how I can use Managed Identity to retrieve a secret from my Keyvault.  Perhaps if you try it you will understand my frustration.  Thankyou so much so far.

[v-tejrama]

Hi @Maverikk ,

You are correct, and your Microsoft engineer’s assessment is accurate. Currently, Fabric pipelines do not support workspace identity or direct integration with Azure Key Vault, which prevents secure retrieval of secrets for use in a Copy Activity.

 This is a known limitation of the platform, as documented by Microsoft, and is not related to your setup. The recommended and secure solution is to perform token retrieval within a notebook, as notebooks in Fabric can utilize workspace identity to access Key Vault securely. 

While I understand your concerns regarding CU usage, this remains the only supported secure method until Fabric enhances managed identity and Key Vault integration for pipelines. If reducing CU consumption is a priority, you might use a lightweight notebook to retrieve the token and execute the API call within the same session, keeping the secret secure. Microsoft is working on expanding these capabilities, which will eventually allow pipeline activities to access Key Vault directly.

Best Regards,
Tejaswi.
Community Support

[Maverikk]

Thankyou @v-tejrama for your response.  Your suggestion would take me back to the root of the issue, in that I can retrieve the token inside a Notebook, but am unable to pass the value OUT of the notebook and into the pipeline for consumption.  The notebook passes [REDACTED] out as the value.  I am able to deconstruct it inside the notebook and pass that out, but that is not secure, and no better than just storing the value in the pipeline in the first instance.  It does seem that this is a fundamental failure in Fabric and its security model, so any suggestions are appreciated. 

[v-tejrama]

Hi @Maverikk ,

 I completely understand the frustration here. What you are seeing with the notebook returning the value as REDACTED is the expected behavior in Fabric because the platform will not allow a secret retrieved inside a notebook to be passed back out to the pipeline. Anything Fabric detects as sensitive is automatically masked so that it never leaves the secure execution boundary. This means that even though the notebook can authenticate with Workspace Identity and obtain the token, there is no supported way to expose that token for a downstream activity to use. At the moment, Fabric does not provide an end to end pattern where a notebook retrieves a secret and then hands it back to a pipeline activity such as Copy. The only supported approach today is to have the pipeline itself retrieve the token through a connection that uses a service principal, because that is the identity type pipelines are currently able to authenticate with. I know that is not the workflow you were hoping for, but it is the secure and supported path with the capabilities that exist right now.

Thank you.

[v-tejrama]

Hi @Maverikk ,

I wanted to follow up and see if you had a chance to review the information shared. If you have any further questions or need additional assistance, feel free to reach out.

Thank you.

[v-tejrama]

Hi @Maverikk ,

I wanted to follow up and see if you had a chance to review the information shared. If you have any further questions or need additional assistance, feel free to reach out.

Thank you.