Forum Discussion
pass KeyVault values OUT OF Notebook to pipeline to consume - securely
Hi Maverikk ,
I completely understand the frustration here. What you are seeing with the notebook returning the value as REDACTED is the expected behavior in Fabric because the platform will not allow a secret retrieved inside a notebook to be passed back out to the pipeline. Anything Fabric detects as sensitive is automatically masked so that it never leaves the secure execution boundary. This means that even though the notebook can authenticate with Workspace Identity and obtain the token, there is no supported way to expose that token for a downstream activity to use. At the moment, Fabric does not provide an end to end pattern where a notebook retrieves a secret and then hands it back to a pipeline activity such as Copy. The only supported approach today is to have the pipeline itself retrieve the token through a connection that uses a service principal, because that is the identity type pipelines are currently able to authenticate with. I know that is not the workflow you were hoping for, but it is the secure and supported path with the capabilities that exist right now.
Thank you.
Super UserMaverikk from your explanation you are using a data pipeline (Copy activity, etc.) to retrieve data from a commercial Web API. I assume that the Web API requires authentication and because you mention a Bearer token I also assume it is a JWT token. Another assumption is you need an API key for authentication which is a secret value. It absolutely makes sense to want to store the API key in Azure KeyVault. You can use Web Activity in a pipeline to connect to AKV and retrieve a secret during a pipeline execution, which is currently a recommended approach in a data pipeline in Microsoft Fabric.
So, your data pipeline logic will look like this:
1. Web activity: retrieve a secret value for API key from Azure Key Vault using a service principla authentication. Service principal credentials for such connection will be stored in Fabric in a secure way. Use the secret value for activity output as a SecureString to avoid exposure of the sensitive value in the pipeline logs.
2. Web activity: acquire an authentication token from your API authentication point using the API key from the previous activity output. Because the outpu is secure the API key won't be visible or exposed. Return the token as activity output as SecureString. Use this activity output in an Authorization header in the next Copy activity.
3. Copy activity: retrieve data from your API endpoint using a Bearer token from the previous activity.
This approach is well known and well tested pattern in data pipeline in both Azure Data Factory and Microsoft Fabric. Feel free to ask more questions.