Forum Discussion
Redaction issue (need a workaround)
Hi dbeavon3,
Thank you for bringing this issue. It seems to be a technical limitation or bug with the notebookutils.notebook.exit functionality in ADF pipelines.
To address this effectively, we recommend raising a support ticket with Microsoft Azure Support. This will allow the engineering team to investigate the issue in detail and provide a resolution or guidance
To raise a support ticket for Fabric and Power BI, kindly follow the steps outlined in the following guide:
How to create a Fabric and Power BI Support ticket - Power BI | Microsoft LearnThank you for being a part of the Microsoft Fabric Community.
Hello dbeavon3 ,
Hope you are doing well
Thanks for connecting with the Microsoft Fabric Community Forum.
Here is the one more workaround which helps in solving your concerns.
- When you pass secrets using notebookutils.notebook.exit, they get redacted so, Instead of sending the token directly as part of the exit value, it’s better to store it in a secure location and have the pipeline activity fetch it from there.
- After that store the access token in a Key Vault or a secure storage then in your pipeline, instead of relying on the notebook to pass the token, use the Copy Activity to fetch the token from the Key Vault
- Instead of trying to pass the access token directly via the notebookutils.notebook.exit, you can have the notebook that store’s the token in the Key Vault, and then the pipeline can retrieve it.
- Instead of using notebookutils.notebook.exit to pass secrets, you can use secure parameters in your pipeline activity to securely pass sensitive data and create a secure parameter in the pipeline and assign the access token to it.
- When you use the Copy Activity in the pipeline, reference the secure parameter directly for the Authorization header.
Hope this method avoids passing sensitive data between the notebook and the pipeline directly, ensuring the token is securely retrieved when needed.
Please let us know if you have further queries or concerns we are here to assist you.
Thank you have a great day a head.
v-sgandrathi
Thanks for the tips/feedback.>> store it in a secure location and have the pipeline activity fetch it from there.
Yes I found this workaround. See my other post from yesterday. It seems unfortunate that the "redaction" functionality, which is intended to promote security, will actually cause users to pursue workarounds that might be even less secure . It is an unintended consequence and seems counterproductive, in that it directly conflicts with the original goal. Storing access tokens out in files seems less secure than if we were to simply pass them (in memory) between a notebook and a pipeline.
>> After that store the access token in a Key Vault
I didn't think pipelines (in Fabric) had access to key vault via MSI credentials. I heard that the only resource that pipelines could access via MSI were storage-blob containers. I will review that assumption, which was only based on my reading and not my first-hand experience. In any case, it seems excessive to introduce a keyvault dependency just to pass a value from a notebook to a pipeline in Fabric.
>> you can use secure parameters in your pipeline activity
You may recall that passing control back and forth between the notebook and pipeline activity was an unfortunate workaround. As I mentioned, we are only using pipelines because of a limitation in Fabric that prevents notebooks from calling REST api's. Ideally we reduce the complexity of ADF pipelines over time (or better yet eliminate them). Anything we can do in notebooks is preferrable to adding complexity to pipelines.I believe you have confirmed that sending the temporary access token to another storage location is a reasonable workaround for a Fabric developer. As a "Fabric data engineer", I find myself taking shortcuts that a standard software engineer would never use. I don't spend much time in here... but I wanted to check and see if my approach to using the product is not that much different than what others are doing.
Hello dbeavon3 ,
I hope you are doing well.
You are True and we have confirmed with our internal team that key vault via MSI credentials is not supported at this time. However, this option may be available from Microsoft in the future. If you wish to secure your pipeline, please refer to the following document:
Securing Your Data Pipelines: Best Practices for Fabric Data Factory | Microsoft Community Hub
Thank you for your patience and understanding.Hi dbeavon3,
As we have not received a response from you yet, I would like to confirm whether you have successfully resolved the issue or if you require further assistance.
If the issue has been resolved, please mark the helpful reply as a "solution" to indicate that the question has been answered and to assist others in the community.
Thank you for your cooperation. Have a great day.
