Executing Azure Databricks Notebook using Fabric Data Factory with Service Principal authentication

[BhaskarMishra22]

Hello there,

I am trying to run a Databricks notebook via an activity within my Fabric pipeline.

While using a Privileged Access Token it works fine. When i try using a Service Principal i’m getting a vague error message.

I've added the Service Principal to both the workspace and cluster permissions, but I'm still getting that vague error message when trying to browse notebooks. The fact that it works perfectly with a PAT token but fails with Service Principal is particularly puzzling.

I'm also wondering if there's some API permission we're missing, as the error message isn't giving us much to work with.

[v-pbandela-msft]

Hi @BhaskarMishra22,

I wanted to check if you had the opportunity to review the information provided. Please feel free to contact us if you have any further questions. If my response has addressed your query, please "Accept  as  Solution" and give a 'Kudos' so other members can easily find it.

Thank you,
Pavan.

[v-pbandela-msft]

Hi @BhaskarMishra22 

I wanted to check if you had the opportunity to review the information provided. Please feel free to contact us if you have any further questions. If my response has addressed your query, please "Accept  as  Solution" and give a 'Kudos' so other members can easily find it.

Thank you,
Pavan.

[v-pbandela-msft]

Hi @BhaskarMishra22,

I wanted to follow up since we haven't heard back from you regarding our last response. We hope your issue has been resolved.
If the community member's answer your query, please mark it as "Accept as Solution" and select "Yes" if it was helpful.
If you need any further assistance, feel free to reach out.

Please continue using Microsoft community forum.

Thank you,
Pavan.

[v-pbandela-msft]

Hi @BhaskarMishra22,

Thank you for reaching out in Microsoft Community Forum.

Please follow below steps to acheive the error;

1. Make sure Service Principal has the correct roles assigned in the Databricks workspace, cluster, and necessary API permissions in Azure AD.

2. make sure the Service Principal has the right scope and compare the authentication flow between PAT and Service Principal.

Please continue using Microsoft community forum.

If you found this post helpful, please consider marking it as "Accept as Solution" and select "Yes" if it was helpful. help other members find it more easily.

Regards,
Pavan.

[BhaskarMishra22]

Hi @v-pbandela-msft 

I have assigned my service principal all the necessary roles required , below is the attached SS for that

Can you please suggest something .

[BhaskarMishra22]

When using service principal authentication with Unity Catalog mirroring, the mirroring process works successfully.
However, there seems to be an issue when trying to execute Databricks notebooks using the same service principal.
Could you help identify why this is happening and what permissions or configurations might be missing? (What should be the correct scope for Service Principal?)

[v-pbandela-msft · Accepted solution]

Hi @BhaskarMishra22,

Thank you for providing more context. Based on the issue you're facing with Service Principal authentication when running Databricks notebooks. please follow below steps to resolve your issue;

1. please verify the permissions on notebooks and cluster. and make sure all the below roles are assigned to the Service Principal
    1. Databricks Workspace User
     2.Databricks Job User
     3.Cluster User
2. for seamless interaction between the Service Principal and Databricks resources, make sure the Service Principal has the correct scope in Azure Active Directory to access the Databricks REST API, including permissions to execute notebooks, interact with clusters, and manage jobs

3. if the still issue persists enable verbose logging to identify the exact permission failure

Please continue using Microsoft community forum.

If you found this post helpful, please consider marking it as "Accept as Solution" and select "Yes" and give it a 'Kudos'. if it was helpful. help other members find it more easily.

Regards,
Pavan.