Notebook activity in the Fabric pipeline, using Service Principal resulting in error

Hi bkhan2,

Thank you Lodha_Jaydeep, for your insights.

The failure happens during notebook execution when Fabric attempts to create a Spark (Livy) session. Microsoft documentation states that the Livy API used for Spark execution currently does not support Service Principals, which may cause failures if a Service Principal is used. Even with the correct permissions, this is a platform limitation, not a configuration problem. As a workaround, you can run the notebook activity with a user identity or Workspace Identity, or use a hybrid method where only the notebook operates under a user identity.

Submit Spark batch jobs using the Livy API - Microsoft Fabric | Microsoft Learn

Notebook activity - Microsoft Fabric | Microsoft Learn

Thank you.

Hi bkhan2,

Since you’ve followed the documented setup for Microsoft Fabric and the issue still continues with both Service Principal and Workspace Identity, it seems unlikely that this is a configuration issue.
The error during Spark (Livy) session creation suggests there may be a backend or token flow problem that needs further investigation. I recommend raising a Microsoft support ticket with the Pipeline Run ID and RootActivityId for further analysis.

Create a Fabric and Power BI Support Ticket - Power BI | Microsoft Learn

Thank you.

Hi bkhan2,

Thanks for reaching fabric community. will happy to assist.

Can you please test first the service principal is working as accepted or not by follow the steps below.
SPNCheck

After sucessully test of SPN, still you are facing any issues while run the notebook/pipeline as service principal (mentioned in your post), below are the some steps you can try.

1. Re-bind the Activity:
   Community reports indicate a "sticky" metadata issue where manually adding the notebook from the pipeline doesn't always trigger the correct token flow.
   • Delete the existing Notebook activity from your pipeline.
   • Open the Notebook itself.
   • Go to the Run tab and select Add to pipeline -> Existing pipeline.
   • Reconnect your dependencies and try running it again. 
     
     
2. Enable "Code-First Artifacts" on the Connection if used:
   If you created the connection in the "Manage Gateways and Connections" portal, it may lack the permission to be used by notebooks.
   • Verify if the connection has the checkbox "Allow Code-First Artifacts like Notebooks to access this connection (Preview)" enabled.
   • Crucial: This setting can often only be set during the initial creation of the connection. If it is unchecked, you must delete and recreate the connection.
     
     

Please accept as an answer if it helps, kudos are really appriciated.

Hi bkhan2 ,

In the Notebook activity's settings, you would see the Connection field as optional. In case, if your Notebook and the Data Pipeline are in the same workspace, just remove the Connection and the Notebook activity will succeed.

You can go through my blog here for more info - https://praveenkumarsreeram.com/2026/04/27/11644/ 

Thanks,

Praveen Sreeram

https://praveenkumarsreeram.com