Invoke pipeline using Workspace Identity fails

[CCal]

Hello, 

I want to use the Workspace Identity to invoke another pipeline, because I do not want to run the pipeline under my own identity.

I am using the new Invoke Pipeline activity.

I am testing it in a very simple case, yet it still fails. (The legacy component does seem to work)

I have a outer_pipeline that invokes an inner_pipeline. The connection is configured to use a service principal, so I use the Workspace Identity. For this test case the set up is simple. The inner_pipeline just have a Set Variable activity and the outer_pipeline just invokes the inner_pipeline.

When I make a new Connection, i see the option to use Service Principal, but on the "Manage connections and gateways" menu, once the OAuth 2 is checked and saved, I cannot revert back to Service Principal. The dropdown option only shows OAuth 2.

Is Invoking Pipelines using Service Principal not supported or is it a bug?

[v-pnaroju-msft]

Hi CCal,

Thank you for the followup.

As per my understanding, the new Invoke Pipeline activity in Microsoft Fabric supports Workspace Identity; however, it does not currently facilitate Service Principal authentication through the GUI. Since you are invoking a pipeline within the same or another workspace, the simplest and most secure approach is to utilise Workspace Managed Identity (WMI), which requires no secrets or client IDs.

Kindly follow the steps outlined below to resolve the issue:

1. In Microsoft Fabric, navigate to your workspace settings and confirm that Workspace Managed Identity is enabled.
2. In the Azure Portal, go to the Target Workspace's Resource Group and access the Access Control (IAM) settings. Assign the Contributor role (or higher) to the Workspace Managed Identity of the outer pipeline. You can identify this Managed Identity by its name format: Microsoft Fabric - <Workspace Name>.
3. In the outer pipeline, open the Invoke Pipeline activity and create a new connection. For the Authentication type, select Workspace Managed Identity. Please do not select OAuth 2.0. If the connection was previously saved with OAuth 2.0 and cannot be changed, delete the existing connection and create a new one from scratch using Workspace Identity.

Additionally, please refer to the following links for further information:
Workspace identity - Microsoft Fabric | Microsoft Learn
Authenticate with Microsoft Fabric workspace identity - Microsoft Fabric | Microsoft Learn

We hope that the above information will assist in resolving the issue. Should you have any further queries, please feel free to reach out to the Microsoft Fabric community.

Thank you.

[v-pnaroju-msft]

Hi CCal,

Thank you for your question on the Microsoft Fabric Community Forum.

As I understand, the new Invoke Pipeline activity does not support Service Principal (Workspace Identity) authentication yet. It uses OAuth 2.0, which is based on user authentication. Because of this, when you try to invoke pipelines using workspace identity, it fails with an authentication error. This is not a bug but a limitation in the current version.

Please try the following workaround to use a non-user identity like Service Principal or Workspace Identity to invoke pipelines:

1. Use a Web Activity instead of Invoke Pipeline.
2. Call the Fabric REST API to start the pipeline:
   POST https://api.fabric.microsoft.com/jobs/instances?jobType=Pipeline
3. In the Web Activity, set the authentication type to Organizational. Use a Service Principal or Managed Identity that has the correct permissions on the workspace.

Also, please check this link for more details:
Invoke pipeline activity - Microsoft Fabric | Microsoft Learn

We hope this helps you resolve the issue. If you have any more questions, please feel free to ask the Microsoft Fabric community.

Thank you.

[CCal]

Hello @v-pnaroju-msft , 

Thank you for your reply.
However, I am not able to get this to work. When I select Auth type to Organizationl, I would not be able to enter the Service Principal credentials (TenantID, Client ID and secret).
Also, even If I select the Service Principal as Auth Type, I get "Unexpected error" and cannot save the connection.

The link unfortunately does not delve deep into the connection part.