Redaction issue (need a workaround)

[dbeavon3]

We are migrating workloads from Synapse to Fabric.

In Synapse we were able to connect to make round-trips to remote services via an MPE for "private link service" .  We could do this directly from our Synapse notebooks, and it allowed us to reach some custom on-premise web services (REST api)

However in Fabric that is not possible, so we must rely on a pipeline copy activity to do a comparable thing. 

Part way thru a notebook, we must pass the control to a "pipeline copy-activity" and then return control to a different notebook again.  The arguments to the copy-activity are sent via the output from a notebook ("notebookutils.notebook.exit").  This exit value is intended to include an access token (eg. for the "Authorization" header to the on-premise web service).

But we encounter a problem related to redaction.  When my outputs from the notebook are transmitted by way of "notebookutils.notebook.exit", the access token which is destined for the "copy activity" will become obfuscated.  Within that exit value, it is always producing a redacted literal ("[REDACTED]").   I haven't found an easy way to send the token without the obfuscation.  It seems to happen because the notebook recalls the prior use of "notebookutils", when we had retrieved the secret in the past. 

Here are more details about secret redaction in fabric.
https://learn.microsoft.com/en-us/fabric/data-engineering/author-execute-notebook#secret-redaction

I am able to avoid the problem by doing silly things like cutting the secret in half or converting to a base64 string:

However I'd rather not do these things, since the entire purpose of the "pipeline copy activity" is to perform a simple transient REST api call.  There is a limited risk in sending the access token between the notebook and the pipeline activity, in order to perform the REST api call.  Sending the access token as an exit value is certainly less risky than removing the validation of the "Authorization" header from this remote REST api.

Please let me know if there is a way to get the notebookutils to stop redacting my secrets as they are transmitted between the notebook and pipeline.

In my copy activity, I intend to retrieve the access token via json like so:

@Json(activity('Notebook001').output.result.exitValue).authorizationHeader

... any tips about a workaround would be greatly appreciated.

[v-sgandrathi]

Hello @dbeavon3 ,
Hope you are doing well
Thanks for connecting with the Microsoft Fabric Community Forum.

Here is the one more workaround which helps in solving your concerns.

• When you pass secrets using notebookutils.notebook.exit, they get redacted so, Instead of sending the token directly as part of the exit value, it’s better to store it in a secure location and have the pipeline activity fetch it from there.
• After that store the access token in a Key Vault or a secure storage then in your pipeline, instead of relying on the notebook to pass the token, use the Copy Activity to fetch the token from the Key Vault
• Instead of trying to pass the access token directly via the notebookutils.notebook.exit, you can have the notebook that store’s the token in the Key Vault, and then the pipeline can retrieve it.
• Instead of using notebookutils.notebook.exit to pass secrets, you can use secure parameters in your pipeline activity to securely pass sensitive data and create a secure parameter in the pipeline and assign the access token to it.
• When you use the Copy Activity in the pipeline, reference the secure parameter directly for the Authorization header.

Hope this method avoids passing sensitive data between the notebook and the pipeline directly, ensuring the token is securely retrieved when needed.

Please let us know if you have further queries or concerns we are here to assist you.
Thank you have a great day a head.

[dbeavon3]

@v-sgandrathi 
Thanks for the tips/feedback.

>> store it in a secure location and have the pipeline activity fetch it from there.

Yes I found this workaround.   See my other post from yesterday.  It seems unfortunate that the "redaction" functionality, which is intended to promote security,  will actually cause users to pursue workarounds that might be even less secure .  It is an unintended consequence and seems counterproductive, in that it directly conflicts with the original goal.  Storing access tokens out in files seems less secure than if we were to simply pass them (in memory) between a notebook and a pipeline. 

>> After that store the access token in a Key Vault 
I didn't think pipelines (in Fabric) had access to key vault via MSI credentials.  I heard that the only resource that pipelines could access via MSI were storage-blob containers.  I will review that assumption, which was only based on my reading and not my first-hand experience.  In any case, it seems excessive to introduce a keyvault dependency just to pass a value from a notebook to a pipeline in Fabric.

>> you can use secure parameters in your pipeline activity

You may recall that passing control back and forth between the notebook and pipeline activity was an unfortunate workaround.  As I mentioned, we are only using pipelines because of a limitation in Fabric that prevents notebooks from calling REST api's.  Ideally we reduce the complexity of ADF pipelines over time (or better yet eliminate them).    Anything we can do in notebooks is preferrable to adding complexity to pipelines.

I believe you have confirmed that sending the temporary access token to another storage location is a reasonable workaround for a Fabric developer.  As a "Fabric data engineer", I find myself taking shortcuts that a standard software engineer would never use.  I don't spend much time in here... but I wanted to check and see if my approach to using the product is not that much different than what others are doing.

[v-sgandrathi]

Hello @dbeavon3 ,
I hope you are doing well.

You are True and we have confirmed with our internal team that key vault via MSI credentials is not supported at this time. However, this option may be available from Microsoft in the future. If you wish to secure your pipeline, please refer to the following document:
Securing Your Data Pipelines: Best Practices for Fabric Data Factory | Microsoft Community Hub

Thank you for your patience and understanding.

[v-sgandrathi]

Hi @dbeavon3,

As we have not received a response from you yet, I would like to confirm whether you have successfully resolved the issue or if you require further assistance.

If the issue has been resolved, please mark the helpful reply as a "solution" to indicate that the question has been answered and to assist others in the community.

Thank you for your cooperation. Have a great day.

[dbeavon3]

Hi @v-sgandrathi 

I think this is a bug in "notebookutils.notebook.exit".  Do you have any guidance on getting a fix?

Given that the adf pipeline activities are intended to allow passing "secure" parameters, and given that notebooks are one of the most common types of activities to run in an adf pipeline, then a customer would reasonably expect to be able to send a "secure" output parameter between activities via "notebookutils.notebook.exit"

Another reason I think this is a bug is because of the underlying goals for redaction.  That is intended to prevent users of notebooks from unwittingly displaying secrets in the notebook outputs (... the cruft that appears under the cells).  However, the redaction is NOT intended to prevent us from using secrets in a deliberate/intentional way for the sake of our solutions.  Send an oauth token out of "notebookutils.notebook.exit" is a deliberate decision, and the redaction functionality should not interfere with that.

Obviously the workaround is available ("store it in a secure location").  But the end result of this workaround is even less secure than if the token was transmitted by way of notebookutils.notebook.exit.  It makes the oauth token avaiable for a longer period of time, and potentially to a wider audience (based on the security of the "secure location").

Can you please give guidance on how to get the bug fixed, and a guestimate on how long it would take the PG to get engaged on something like this?

[v-sgandrathi · Accepted solution]

Hi @dbeavon3,

Thank you for bringing this issue. It seems to be a technical limitation or bug with the notebookutils.notebook.exit functionality in ADF pipelines.

To address this effectively, we recommend raising a support ticket with Microsoft Azure Support. This will allow the engineering team to investigate the issue in detail and provide a resolution or guidance

To raise a support ticket for Fabric and Power BI, kindly follow the steps outlined in the following guide:
How to create a Fabric and Power BI Support ticket - Power BI | Microsoft Learn

Thank you for being a part of the Microsoft Fabric Community.

[dbeavon3]

I think it is a bug, since using a token for programmatic purposes should be allowable, as should the transfer of secrets between one step of a pipeline and another.

I will probably leave this for the next customer that encounters it.  Based on my experiences with Fabric notebooks this past week, the team has much bigger fish to fry.  Even in the optimistic chance that they agreed this was a bug, I cannot see them fixing it for the next year or two.

[dbeavon3]

In addition to the workarounds above, another one I found is to use an entirely separate notebook to generate access token and save it to a file.  Once it is in a file, we can read it back in a different notebook (sequentially executed via pipeline).   After jumping thru these hoops, the Fabric notebook environment has no idea that it is a secret anymore, and doesn't try redacting anything. 

This workaround seems very unfortunate, and quite a bit less secure than chopping up the secret into parts and sending it out of notebookutils.notebook.exit().