Trigger fabric data factory pipeline using python code

[PunnamSrinivas]

I am trying to run a Microsoft Fabric Data Factory pipeline using Python code. To achieve this, I have created an Azure Active Directory App Registration and assigned all necessary permissions. Additionally, I have provided subscription-level permissions and assigned the roles of Data Factory Contributor and Contributor to the Azure AD app.

While I am successfully generating an access token, I am encountering a 401 Unauthorized error when trying to access Fabric pipelines. Specifically, I am testing the following URL in Postman, as outlined in the Fabric documentation:

https://api.fabric.microsoft.com/v1/workspaces/<your WS Id>/items/<pipeline id>/jobs/instances?jobType=Pipeline

However, I receive the following error message:

{
"requestId": "4ffd9d63-b77f-4fd0-8a71-d98ffcf0e5d6",
"errorCode": "InvalidToken",
"message": "Access token is invalid"
}

Additionally, I have tried using the following URL:

https://management.azure.com/subscriptions/{subscription_id}/resourceGroups/default-resource-group/p...

But I am receiving this error:

{
"error": {
"code": "InvalidResourceType",
"message": "The resource type could not be found in the namespace 'Microsoft.Fabric' for api version '2023-04-01-preview\"'."
}
}

Despite assigning all required permissions and roles, I am unable to authenticate and run the pipeline. Could anyone provide guidance on how to resolve these issues and properly authenticate to execute the pipeline?

[rsuresh_cit]

Hi @PunnamSrinivas, Any update on your issue? I am also facing the same issue. Please share it will be helpful to us. 

[PunnamSrinivas]

Hi Fabian,

Thank you for the detailed guidance. Following your instructions, I created a new security group in Azure and added the service principal as a member of this security group. After that, I was able to successfully access the workspace items using the following endpoints:

• GET: https://api.fabric.microsoft.com/v1/workspaces/
• GET: https://api.fabric.microsoft.com/v1/workspaces/<your WS Id>/items/<pipeline id>

However, when trying to access the pipeline run URL using the same token, I encountered the following error:

Pipeline run URL:

https://api.fabric.microsoft.com/v1/workspaces/37de13f9-7411-4c41-9a8b-6658a9bb2310/items/1273d239-c...

Error Message: {
"requestId": "f0bd2298-42f9-43d4-8c4b-1ffcf536c703",
"errorCode": "PrincipalTypeNotSupported",
"message": "The operation is not supported for the principal type"
}

I haven't yet added the Tenant.ReadWrite.All permission to the service principal, and my organization hasn't assigned me as a Fabric Administrator. However, I did attempt the process using a Fabric Administrator role account, and the issue persists. Could you please advise if there are any additional permissions or roles that need to be assigned to resolve this error?

Thank you in advance for your assistance.

Kind regards,
Punnam Srinivas

[mandania]

hi Punnam Srinivas, did you manage to solve this? i am having a same issue as well.

[FabianSchut]

Hi @PunnamSrinivas,

This particular API call is indeed not supported for a service principal: https://learn.microsoft.com/en-us/rest/api/fabric/core/job-scheduler/get-item-job-instance?tabs=HTTP....

You could try to run this in the Azure Cloud Shell: https://learn.microsoft.com/en-us/azure/cloud-shell/get-started/classic?tabs=azurecli and login with your account and see if that works.

[charlyS]

Hello, 

Looks like you are trying to use Fabric API that does not support SPN authentication for the moment as mentioned here > https://learn.microsoft.com/en-us/fabric/data-factory/pipeline-rest-api#known-limitations isn't it ? 

Regards

[PunnamSrinivas]

Hi Fabian,

Thank you for the information. Could you please provide step-by-step guidance on how to add the service principal to the security group? I attempted to generate a token in Postman using my personal account, but I was unsuccessful. I’m sharing a screenshot of the error message below.

Additionally, could you advise if I am missing any configurations regarding the service principal permissions or Fabric workspace permissions? Is there any specific URL or alternative method to generate the access token for accessing Fabric pipelines? Your help is greatly appreciated.

[FabianSchut]

Hi @PunnamSrinivas,

You can find a detailed step-by-step guide to add a service principal to a security group in Microsoft Entra: https://prodata.ie/2023/11/15/service-principal-fabric/. Scroll down to the section of Creation of Security Group; all the other steps you already implemented correctly. Do note that the group you need is of group type Security, otherwise, a service principal cannot be selected as a member. If the group you currently have is not of Security type, please create a new one and add this group to the developer settings Service principals can use Fabric APIs as you already screenshotted below. Once your service principal is added to the security group, try to authenticate again with the service princapal (and not your user account) using the grant_type client_credentials.

All the other steps seem good so far.

[PunnamSrinivas]

Hi Fabian,

I wanted to provide an update regarding our Fabric API access. We have enabled the "Service Principal can use Fabric APIs" option under the Fabric tenant settings for a specific security group, rather than for the entire organization. My account is included in this security group.

Please find the attached screenshot for your reference.

[FabianSchut]

@PunnamSrinivas, I think you are almost there. Could you also add the service principal to the security group? In Postman, you are not connecting with your personal account, but with the service principal. That one needs access to the Fabric APIs too.

[PunnamSrinivas]

Thank you for sharing the information. I followed the steps, and while the token is being generated successfully, I am still encountering unauthorized access errors when trying to interact with Fabric items such as workspaces or pipelines. Despite having the token, I am unable to access these resources.

Could you please provide further guidance on resolving this issue? Specifically, could you confirm what permissions are required for the service principal app, and what roles or permissions are needed within the workspace to allow proper access? I suspect there might be a missing permission or role assignment.

I can provide screenshots of the steps I’ve taken so far if that would help diagnose the problem.

[mandania]

@PunnamSrinivas  I hope you dont mind if i piggy back on your question. I also have similar problem , token is generated on but continue to her "The caller is not authenticated..." i wonder if you managed to solve this problem and if so , would you kind enough to share your solution please?

[PunnamSrinivas]

Hi Fabian,

Thank you for your response. I have added the service principal (app registration) to the Fabric workspace with the Contributor role, as per your suggestion. However, I am still encountering the "Unauthorized" error related to the access token.

Please find the attached screenshot for your reference, showing where the app registration has been added within the Fabric workspace.

I would appreciate any further guidance you can provide to resolve this issue.

Thank you again for your assistance.

Thanks & Regards,
Punnam Srinivas

[FabianSchut]

Hi @PunnamSrinivas,

Check, that looks good. What code do you use to retrieve the bearer token? Can you follow these steps:
https://stackoverflow.com/questions/78226072/generating-token-for-fabric-rest-api-using-client-secre...

[PunnamSrinivas]

Hi Fabian,

Thank you for your response.

Yes, I have already added the app registration to the workspace where the Data Factory is created. The app has been granted the necessary permissions at the workspace level, but I am still encountering the issue.

Please let me know if there are any additional steps I should take.

Thanks & Regards,

Punnam Srinivas

[FabianSchut]

Hi @PunnamSrinivas, could you also send a screenshot of the Fabric Workspace where the app registration is added? This is not in Azure, but in Fabric itself. That is in the place where you normally add users.

[FabianSchut]

Hi @PunnamSrinivas,

Did you also add the app registration to the Workspace where the Data Factory is created in? Otherwise, it still does not have access to that specific Data Factory Pipeline.

Kind regards,

Fabian