Forum Discussion
pass KeyVault values OUT OF Notebook to pipeline to consume - securely
Hi Maverikk ,
I completely understand the frustration here. What you are seeing with the notebook returning the value as REDACTED is the expected behavior in Fabric because the platform will not allow a secret retrieved inside a notebook to be passed back out to the pipeline. Anything Fabric detects as sensitive is automatically masked so that it never leaves the secure execution boundary. This means that even though the notebook can authenticate with Workspace Identity and obtain the token, there is no supported way to expose that token for a downstream activity to use. At the moment, Fabric does not provide an end to end pattern where a notebook retrieves a secret and then hands it back to a pipeline activity such as Copy. The only supported approach today is to have the pipeline itself retrieve the token through a connection that uses a service principal, because that is the identity type pipelines are currently able to authenticate with. I know that is not the workflow you were hoping for, but it is the secure and supported path with the capabilities that exist right now.
Thank you.
Hi Maverikk ,
That’s an excellent follow-up question, and your focus on secure handling and minimizing CU usage is well placed. In Fabric, it is essential to ensure that secrets and tokens remain within secure contexts and are not written to logs. To avoid using a notebook, I recommend configuring the pipeline to manage both Key Vault retrieval and API authorization directly. You can achieve this by adding a Web activity before the Copy Activity to obtain the token from your authentication endpoint or Key Vault using a managed identity.
The token output from the Web activity can then be referenced in the Copy Activity’s Authorization header, for example, using activity GetToken output.access_token. This method ensures the token is securely used at runtime and not exposed or stored in plain text. If your environment allows, grant the pipeline’s managed identity access to Key Vault secrets, enabling direct retrieval without a notebook. This approach maintains security, reduces CU costs, and aligns with Fabric’s best practices for secret management.
Thank you.
Thankyou for your lengthy response Anonymous , however, my "Cloud & AI - Data Solution Engineer" at microsoft has advised that "Pipelines don’t support Workspace Identity for Web Activity calls whereas Dataflow Gen2 support." So whilst I agree in principal with your pipeline flow, I still do not see how I can use Managed Identity to retrieve a secret from my Keyvault. Perhaps if you try it you will understand my frustration. Thankyou so much so far.
Hi Maverikk ,
You are correct, and your Microsoft engineer’s assessment is accurate. Currently, Fabric pipelines do not support workspace identity or direct integration with Azure Key Vault, which prevents secure retrieval of secrets for use in a Copy Activity.
This is a known limitation of the platform, as documented by Microsoft, and is not related to your setup. The recommended and secure solution is to perform token retrieval within a notebook, as notebooks in Fabric can utilize workspace identity to access Key Vault securely.
While I understand your concerns regarding CU usage, this remains the only supported secure method until Fabric enhances managed identity and Key Vault integration for pipelines. If reducing CU consumption is a priority, you might use a lightweight notebook to retrieve the token and execute the API call within the same session, keeping the secret secure. Microsoft is working on expanding these capabilities, which will eventually allow pipeline activities to access Key Vault directly.
Best Regards,
Tejaswi.
Community SupportThankyou Anonymous for your response. Your suggestion would take me back to the root of the issue, in that I can retrieve the token inside a Notebook, but am unable to pass the value OUT of the notebook and into the pipeline for consumption. The notebook passes [REDACTED] out as the value. I am able to deconstruct it inside the notebook and pass that out, but that is not secure, and no better than just storing the value in the pipeline in the first instance. It does seem that this is a fundamental failure in Fabric and its security model, so any suggestions are appreciated.
Hi Maverikk ,
I completely understand the frustration here. What you are seeing with the notebook returning the value as REDACTED is the expected behavior in Fabric because the platform will not allow a secret retrieved inside a notebook to be passed back out to the pipeline. Anything Fabric detects as sensitive is automatically masked so that it never leaves the secure execution boundary. This means that even though the notebook can authenticate with Workspace Identity and obtain the token, there is no supported way to expose that token for a downstream activity to use. At the moment, Fabric does not provide an end to end pattern where a notebook retrieves a secret and then hands it back to a pipeline activity such as Copy. The only supported approach today is to have the pipeline itself retrieve the token through a connection that uses a service principal, because that is the identity type pipelines are currently able to authenticate with. I know that is not the workflow you were hoping for, but it is the secure and supported path with the capabilities that exist right now.
Thank you.