Some methods in notebooks do not work when executed from Data Pipelines deployed via SPN

[mmilosanovic]

Hi,

I already created the ticket with Microsoft (2504161420001430) and also opened the issue here and also seems to be related to this issue here.

`notebookutils.runtime.context.get("currentWorkspaceName")` works when executed directly from the Notebook, but does not work when executed from the DataPipeline which was deployed in another workspace using Service Principal via Fabric Core APIs (Create Item - https://learn.microsoft.com/en-us/rest/api/fabric/core/items/create-item?tabs=HTTP).

If I reopen the notebook with my personal account and rerun it again from the DataPipeline, then it works. This implies that the issue is coming from the service principal somehow losing permissions/token and is not able to read the current workspace name where it is running in from the notebookutils runtime context.

What I noticed is that this issue is also present for other built-in methods in Fabric. For example: `notebookutils.lakehouse.get()` and `synapsesql()`. Details can be found here: https://github.com/microsoft/fabric-cicd/issues/202#issuecomment-2797384465. Same approach is being followed in the fabric-cicd library and it is using SPN auth + Fabric Core Create Item APIs (as already mentioned and referenced above).

Did anyone else experience the same/similar issues?

Best regards,

Milos

[v-tsaipranay]

Hi @mmilosanovic ,

Could you please confirm if the issue has been resolved after raising a support case? If a solution has been found, it would be greatly appreciated if you could share your insights with the community. This would be helpful for other members who may encounter similar issues.

Thank you for your understanding and assistance.

[mmilosanovic]

hi @v-tsaipranay , not yet resolved. Still in the back and forth with MS support. Last thing I got is this: "I have consulted with the notebook team, and they have confirmed that it is a known issue." 

I have asked if there is a workaround and got the following feedback:

• "It has been noted that there are issues with executing notebooks via Service Principal (SPN) authentication, particularly with certain functions such as notebookutils. The internal team is actively investigating this matter to identify any limitations or configuration issues that may be causing these failures, and they will share any updates they have over the link.
• Also, I've informed Santhiya that by 20th May, Pipeline product team are releasing connection experience where users can create SPN connection via it."

So I guess we all monitor the progress here: https://github.com/microsoft/fabric-cicd/issues/248, and also keep an eye for the connection experience updates based on the seconds bullet above.

[annhwallinger]

We have also raised a support request for the same issue

[g3kuser]

we have the same issue with workspace identity as executing user. There are errors from cluster just with import statements in notebook and on applying run magic command. Even notebookutils.lakehouse.getWithProperties method also fails with 403 error. We were able to execute all of our code artefacts successfully with executing user as SPN (we generated our own SPN and added a secret to it) whereas when deploying same using workspace identity and running through it failed miserably. We even created a secret for the workspace identity and tried to use it by initializing credential class still no success. 

[annhwallinger]

Thank you for the detail in your response. You said you were able to run as SPN.... do you know if you were able to run these simple lines? 

import json

import sempy.fabric as fabric

from notebookutils import mssparkutils

#Instantiate the client

client = fabric.FabricRestClient()

# Get Workspace

workspaceId = fabric.resolve_workspace_id([workspace name here])

[gaya3krishnan86]

It works by explicitly creating an Service principal token implementation and passing that to fabric rest client. Here is a blog that can helped me with this implementation.

https://fabric.guru/using-service-principal-authentication-with-fabricrestclient

[v-tsaipranay]

Hi @mmilosanovic ,

Thanks for sharing the details and for raising a support ticket .

The issue likely stems from the notebook’s runtime context not being fully initialized when triggered via a Service Principal (SPN) using the Fabric Core APIs. This affects methods like notebookutils.runtime.context.get() and others that rely on workspace-level context.

In the meantime, please try the following steps:

• Ensure the SPN has Contributor or higher role at the workspace level.

• Verify that the workspace is explicitly included in the SPN’s access scope.

• Test execution using a user-assigned managed identity if supported.

• As a workaround, consider passing the required context (e.g., workspace name) as parameters to the notebook via the pipeline.

Please continue monitoring your support case with microsoft and share any updates or guidance as it becomes available, so that other community members who have similar problems to solve it faster.

Thank you.

[deepakagarwal]

Having same issue.

[annhwallinger]

we are also having the same issue

[SparkedMy401]

Yes, I'm having the same issue.. I'm using the fabric-cicd library