Forum Discussion
pass KeyVault values OUT OF Notebook to pipeline to consume - securely
I have a fabric pipeline that initiates a Notebook that calls to KeyVault to return a bearer_token, this is [REDACTED], but when I use mssparkutils.notebook.exit in my Notebook it passes out the string [REDACTED], NOT the value. Using a Web activity isnt acceptable as it is using Service Principal, whereas the Notebook uses Workspace Identity, which is more secure.
What is the recommended approach to securely use the keyvault in a pipeline? I dont have to use a Notebook if there is a better methodology.
Hi Maverikk ,
I completely understand the frustration here. What you are seeing with the notebook returning the value as REDACTED is the expected behavior in Fabric because the platform will not allow a secret retrieved inside a notebook to be passed back out to the pipeline. Anything Fabric detects as sensitive is automatically masked so that it never leaves the secure execution boundary. This means that even though the notebook can authenticate with Workspace Identity and obtain the token, there is no supported way to expose that token for a downstream activity to use. At the moment, Fabric does not provide an end to end pattern where a notebook retrieves a secret and then hands it back to a pipeline activity such as Copy. The only supported approach today is to have the pipeline itself retrieve the token through a connection that uses a service principal, because that is the identity type pipelines are currently able to authenticate with. I know that is not the workflow you were hoping for, but it is the secure and supported path with the capabilities that exist right now.
Thank you.
17 Replies
Hi Maverikk
You should think about Notebooks because everybody in the world uses Linux System. Use Spark and bit of study..
Python --> Data Lake ( Aache Spark ) --> Delta Lake. It is just using Python, and it is open source.
Thanks BhaveshPatel , I am not sure you have answered my question tbh. I am saying that I AM using a Notebook and Python, but am struggling to pass the [REDACTED] OUT of the Notebook and back into a Fabric pipeline.
Hi Maverikk ,
You’re absolutely right to point out that the issue isn’t about using notebooks or Spark in general, but about how to securely pass a value retrieved from Key Vault out of a Fabric notebook so the pipeline can use it. The behavior you’re seeing where the value is replaced with [REDACTED] is expected, as Fabric intentionally masks sensitive information to prevent secrets from being exposed in pipeline logs or variables.
This means values obtained using Workspace Identity inside a notebook can’t be passed directly to a pipeline output. The secure and supported approach is to use the secret within the same notebook where it’s retrieved, or have each pipeline activity that needs it access Key Vault directly through a managed identity or linked service. That way, the secret never leaves a protected environment and your pipeline remains fully compliant with Fabric’s security model.
Best Regards,
Tejaswi.
Community Support
Hi Maverikk Microsoft Fabric specifically prevent any secrets obtained in a Notebook to be leaked outside of the Notebook. You can read more about that technique here https://learn.microsoft.com/en-us/fabric/data-engineering/author-execute-notebook#secret-redaction.
This answer does not solve your questions, unfortunately, but this is not a problem with your solution but a current technical limitation in Fabric. Fabric allows you to create a Key Vault reference but it's not supported in data pipelines and it also does not support yet a private networking for Azure Key Vaults.
So, there is no generic recommended approach yet for your question. We maight be able to find a solution for your specific scenario, but you need to provide an explanation on why do you need to use a secret in a pipeline.
I personally always start from trying to avoid using secrets in the data pipeline in Fabric.
Hopefully that explains the situation even though does not answer your question. Feel free to continue this discussion but please explain your scenario.Thankyou apturlov for taking the time to reply. I am trying to securely store my API Bearer Token inside KeyVault, and then access and use it inside a CopyActivity for an API call. The API call holds commercially sensitive data, so naturally I want to keep the "secret" under wraps. Perhaps you can suggest a way that you would approach this, maybe there is a better methodology than I am using? Thankyou for your time
Maverikk from your explanation you are using a data pipeline (Copy activity, etc.) to retrieve data from a commercial Web API. I assume that the Web API requires authentication and because you mention a Bearer token I also assume it is a JWT token. Another assumption is you need an API key for authentication which is a secret value. It absolutely makes sense to want to store the API key in Azure KeyVault. You can use Web Activity in a pipeline to connect to AKV and retrieve a secret during a pipeline execution, which is currently a recommended approach in a data pipeline in Microsoft Fabric.
So, your data pipeline logic will look like this:
1. Web activity: retrieve a secret value for API key from Azure Key Vault using a service principla authentication. Service principal credentials for such connection will be stored in Fabric in a secure way. Use the secret value for activity output as a SecureString to avoid exposure of the sensitive value in the pipeline logs.
2. Web activity: acquire an authentication token from your API authentication point using the API key from the previous activity output. Because the outpu is secure the API key won't be visible or exposed. Return the token as activity output as SecureString. Use this activity output in an Authorization header in the next Copy activity.
3. Copy activity: retrieve data from your API endpoint using a Bearer token from the previous activity.
This approach is well known and well tested pattern in data pipeline in both Azure Data Factory and Microsoft Fabric. Feel free to ask more questions.
