Forum Discussion
Redaction issue (need a workaround)
Hi dbeavon3,
Thank you for bringing this issue. It seems to be a technical limitation or bug with the notebookutils.notebook.exit functionality in ADF pipelines.
To address this effectively, we recommend raising a support ticket with Microsoft Azure Support. This will allow the engineering team to investigate the issue in detail and provide a resolution or guidance
To raise a support ticket for Fabric and Power BI, kindly follow the steps outlined in the following guide:
How to create a Fabric and Power BI Support ticket - Power BI | Microsoft LearnThank you for being a part of the Microsoft Fabric Community.
v-sgandrathi
Thanks for the tips/feedback.
>> store it in a secure location and have the pipeline activity fetch it from there.
Yes I found this workaround. See my other post from yesterday. It seems unfortunate that the "redaction" functionality, which is intended to promote security, will actually cause users to pursue workarounds that might be even less secure . It is an unintended consequence and seems counterproductive, in that it directly conflicts with the original goal. Storing access tokens out in files seems less secure than if we were to simply pass them (in memory) between a notebook and a pipeline.
>> After that store the access token in a Key Vault
I didn't think pipelines (in Fabric) had access to key vault via MSI credentials. I heard that the only resource that pipelines could access via MSI were storage-blob containers. I will review that assumption, which was only based on my reading and not my first-hand experience. In any case, it seems excessive to introduce a keyvault dependency just to pass a value from a notebook to a pipeline in Fabric.
>> you can use secure parameters in your pipeline activity
You may recall that passing control back and forth between the notebook and pipeline activity was an unfortunate workaround. As I mentioned, we are only using pipelines because of a limitation in Fabric that prevents notebooks from calling REST api's. Ideally we reduce the complexity of ADF pipelines over time (or better yet eliminate them). Anything we can do in notebooks is preferrable to adding complexity to pipelines.
I believe you have confirmed that sending the temporary access token to another storage location is a reasonable workaround for a Fabric developer. As a "Fabric data engineer", I find myself taking shortcuts that a standard software engineer would never use. I don't spend much time in here... but I wanted to check and see if my approach to using the product is not that much different than what others are doing.
Hello dbeavon3 ,
I hope you are doing well.
You are True and we have confirmed with our internal team that key vault via MSI credentials is not supported at this time. However, this option may be available from Microsoft in the future. If you wish to secure your pipeline, please refer to the following document:
Securing Your Data Pipelines: Best Practices for Fabric Data Factory | Microsoft Community Hub
Thank you for your patience and understanding.
Hi dbeavon3,
As we have not received a response from you yet, I would like to confirm whether you have successfully resolved the issue or if you require further assistance.
If the issue has been resolved, please mark the helpful reply as a "solution" to indicate that the question has been answered and to assist others in the community.
Thank you for your cooperation. Have a great day.Hi v-sgandrathi
I think this is a bug in "notebookutils.notebook.exit". Do you have any guidance on getting a fix?
Given that the adf pipeline activities are intended to allow passing "secure" parameters, and given that notebooks are one of the most common types of activities to run in an adf pipeline, then a customer would reasonably expect to be able to send a "secure" output parameter between activities via "notebookutils.notebook.exit"
Another reason I think this is a bug is because of the underlying goals for redaction. That is intended to prevent users of notebooks from unwittingly displaying secrets in the notebook outputs (... the cruft that appears under the cells). However, the redaction is NOT intended to prevent us from using secrets in a deliberate/intentional way for the sake of our solutions. Send an oauth token out of "notebookutils.notebook.exit" is a deliberate decision, and the redaction functionality should not interfere with that.
Obviously the workaround is available ("store it in a secure location"). But the end result of this workaround is even less secure than if the token was transmitted by way of notebookutils.notebook.exit. It makes the oauth token avaiable for a longer period of time, and potentially to a wider audience (based on the security of the "secure location").
Can you please give guidance on how to get the bug fixed, and a guestimate on how long it would take the PG to get engaged on something like this?
Hi dbeavon3,
Thank you for bringing this issue. It seems to be a technical limitation or bug with the notebookutils.notebook.exit functionality in ADF pipelines.
To address this effectively, we recommend raising a support ticket with Microsoft Azure Support. This will allow the engineering team to investigate the issue in detail and provide a resolution or guidance
To raise a support ticket for Fabric and Power BI, kindly follow the steps outlined in the following guide:
How to create a Fabric and Power BI Support ticket - Power BI | Microsoft LearnThank you for being a part of the Microsoft Fabric Community.