Key vault authorization error: Caller is not authorized to perform action on resource.

[PawelO]

Hello,

I have been facing issue with key vault authorization in notebooks while calling them in pielines. Issue is not permament, most of the time it works, however thare are cases when I receive error :

Notebook execution failed at Notebook service with http status code - '200', please check the Run logs on Notebook, additional details - 'Error name - Py4JJavaError, Error value - An error occurred while calling z:mssparkutils.credentials.getSecret.
: java.io.IOException: 403 {"error":{"code":"Forbidden","message":"Caller is not authorized to perform action on resource.\r\nIf role assignments, deny assignments or role definitions were changed recently, please observe propagation time.\r\nCaller: appid=943da971-3786-412e-a184-ec3deda79a6f;oid=fd31060f-e7c1-410e-8ff3-c906f53f2692;iss=https://sts.windows.net/e95bb028-ffe2-4bfb-aea0-5e837e6e4fdb/\r\nAction: 'Microsoft.KeyVault/vaults/secrets/getSecret/action'\r\nResource: '/subscriptions/cfb6daad-4c01-43ce-ab2e-b4c72a88f198/resourcegroups/rg-frame-dev-01/providers/microsoft.keyvault/vaults/kv-frame-dev-001/secrets/aw-sample-db-username'\r\nAssignment: (not found)\r\nDenyAssignmentId: null\r\nDecisionReason: null \r\nVault: kv-frame-dev-001;location=northeurope\r\n","innererror":{"code":"ForbiddenByRbac"}}}
at com.microsoft.azure.trident.tokenlibrary.TokenLibrary.getSecretWithToken(TokenLibrary.scala:804)
at com.microsoft.azure.trident.tokenlibrary.TokenLibrary$.getSecretWithToken(TokenLibrary.scala:1347)
at mssparkutils.credentials$.getSecret(credentials.scala:166)
at mssparkutils.credentials.getSecret(credentials.scala)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:374)
at py4j.Gateway.invoke(Gateway.java:282)
at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
at py4j.commands.CallCommand.execute(CallCommand.java:79)
at py4j.GatewayConnection.run(GatewayConnection.java:238)
at java.base/java.lang.Thread.run(Thread.java:829)
' :

I have found the post (https://community.fabric.microsoft.com/t5/General-Discussion/Not-able-to-connect-keyvault-in-noteboo...) where following appid = 943da971-3786-412e-a184-ec3deda79a6f is also present, which leads my mind to the conclusion that is maybe some kind of fabric internal app. 

Did anyone have common issues ? 

[billyswitzerprx]

Had a similar issue. I was manually testing a newly-deployed notebook in production and getting this same error. I discovered that the call to get_secret_with_token() was being made under my account and not the Service Principal for our Fabric workspace. Once I gave my account permissions on the associated key vault it worked.

[Zall]

Hi @PawelO 

We encountered similiar issues so I am sharing my insight if you did not figure it out yet.

What I think is happening that it is using the "submitters" (red in the image) account for the access control. However determining who is going to be the submitter is kind of tricky. In our case the submitter who did not truly have access and was seen in the notebook snapshot was:
Not the person who run the pipeline
Not owner of the workspace
Not owner of the pipeline
Not owner of the notebook
Not person who made last change to the notebook

Adding him to our key vault fixed the issue.

[PawelO]

Hello @Anonymous ,

I did not make any changes at Key Vault, permossions are granted as they should be. 

In the notebook editor connection to key vault works fine, only runing in pipeline was an issue.

Today I was testing some scenarios and when I have added session tag to the notebook it started working. I remamber that last time when I faced the same issue it was after adding a session tag ( then it stop working). You can try with chaning session tag to replicate the error.

Best regards,

Paweł

[Anonymous]

Hi @PawelO 

I tried adding a session tag but still failed to reproduce it. Can you share the piece of codes you are using in the notebook (please remove sensitive information)? 

In addition, the thread you linked mentions the networking configuration in Azure Key Vault. What's your configuration there? 

Best Regards,
Jing

[Anonymous]

Hi @PawelO 

I couldn't reproduce this error. Did you make any change at the Azure Key Vault side recently? Ensure that you still have sufficient permissions to access that Key Vault and that the pipeline is running under your account. On Recent runs page for the pipeline, you will see who submitted a data pipeline run. 

In addition, when you run the notebook in the Fabric notebook editor manually, does this error occur? Or does it only fail intermittently when runnning in a pipeline? 

Best Regards,
Jing
If this post helps, please Accept it as Solution to help other members find it. Appreciate your Kudos!