Write secret to Key Vault from Fabric Notebook

[x_mark_x]

Hello there Fabricators!

I am using a Fabric Notebook to access an Azure Key Vault and get a secret. This secret is a refresh token which I am than using in the Notebook to call an OAuth2 API to get a new access token and a new refresh token.

My challenge starts when I am trying to write the new access and refresh token back to the Key Vault.

Right now I am using notebookutils.mssparkutils.credentials.getSecret to get the secret, but the putSecret method fails with this error:
Py4JJavaError: An error occurred while calling z:mssparkutils.credentials.putSecret. : com.microsoft.azure.synapse.tokenlibrary.TokenServiceClientResponseStatusException: Token Service returned 'Client Error' (400), with message: {"result":"UserError","errorId":"BadRequest","errorMessage":"Invalid session token - cannot be read..

There are VERY limited information about alternative workarounds, but I am aware of this article: 
https://iterationinsights.com/article/microsoft-fabric-notebooks-and-azure-key-vault/

Is this really the only way right now to write secrets from a Fabric Notebook to a Key Vault?

Does this method work at all if I call the notebook from within a Pipeline?
Is there a way to use the Workspace Identity to be able to write to the Key Vault?

[v-shex-msft]

HI @x_mark_x,

Can you please share some more detail information about these operations? They should help us clarify your scenario and test to troubleshoot.

>>Does this method work at all if I call the notebook from within a Pipeline?

Invoke notebook in pipeline should similar as use notebook to processing.

Regards,

Xiaoxin Sheng

[x_mark_x]

Hi @v-shex-msft 

As far as I know, the putSecret method is not yet supported within the Fabric Synapse Data Engineering experience Notebooks. Could you confirm this? If this is true, is there an estimate when this functionality is implemented within Fabric? (as it exists in Azure Synapse Analytics)

"Can you please share some more detail information about these operations? They should help us clarify your scenario and test to troubleshoot."
Are you referring to the TraceId and client-request-id for the putSecret method I used?
Please confirm and I will provide this information.

Based on this link (https://learn.microsoft.com/en-us/python/api/overview/azure/keyvault-secrets-readme?view=azure-pytho...) I even tried to use the azure-keyvault-secrets and azure-identity libraries in the notebook with no success. This tutorial says: "We recommend using a managed identity for authentication in production environments."
Following the example in the link and running this code

gave me the following error:


Being a simple Data Analyst, the world of authentication is way over my head so any insights, tips and help is apprechiated.
I did not yet managed to try the method described in the original post, where a Client Secret is created for the Application.

My question is the same.
How can I enable my Notebook in my Bronze Workspace to write a secret into the Key Vault?
Is there other ways to go about writing a secret to the Key Vault from a Fabric Notebook other than creating a Client Secret?

To add to the context:
I have created a Workspace identity for my Bronze layer, where the data ingestion is happening with the API call.


Our Azure platform team has added both me and the Bronze workspace to the Key Vault as Key Vault Secret Officer.

If I go to Azure → Microsoft Entra ID → Manage → App registrations → All applications I can see the Bronze Workspace registered there.


Within the Bronze App the overview looks like this.


So far we didn't add any Client Secret for the App



Please let me know if you need any further information.

[v-shex-msft]

HI @x_mark_x,

If both putSecret and set_secret all failed to processed on fabri side, it may mean these type of opeaiton current not support.
For this scenario, you can try to submit an idea for add support with these type of operations.(BTW, I also check the current release plan list but not found similar requirements)

Microsoft Fabric Ideas

Regards,

Xiaoxin Sheng

[x_mark_x · Accepted solution]

Hi @v-shex-msft 

Thank you for your reply. Yes, I got a confirmation from a Cloud Solution Architect @ Microsoft that the putSecret method is not working now, but is under developement at the moment.
I guess we just have to wait it out.

There are plenty of ideas regarding the Key Vault integration already, so I just voted for them and avoided creating more noise 🙂

Meanwhile my solution was to store the tokens in a table instead. In case of OAuth2 authentication, the access token is only valid for 1 hour (in my case) and the refresh token itself do not have any significance without the client-id and client-secret combination, and I could easily and safely get it using the getSecret method.
The token is read from the table, and when a token refresh is necessary, I read the client-id and secret from the Key Vault and send these two with my refresh token as a request.

When the putSecret method is implemented, I will change this solution, so that even the tokens are saved in the Key Vault, for an extra layer of protection.

Thank you for your time and response Xiaoxin.

[yfarinas]

Hey there, I'm facing the same issue with the putSecret method. Is there any way we can get visibility into the development timeline for this feature on Microsoft's side?

[v-shex-msft]

HI @x_mark_x,

Thanks for sharing response information from MS support and the workaround about your requirement here, I think they will help other users who faced similar scenarios.

Regards,

Xiaoxin Sheng