Deployment Pipeline fails when deploying a data pipeline

[krisjones]

I cannot deploy a Fabric Data Pipeline resource from one workspace to another using Fabric Deployment Pipelines trigger by REST API calls.

When I do it manually it works fine. 

When I deploy other items like Notebooks it also works fine.

Data Pipelines are supported (no in Preview) state: Overview of Fabric deployment pipelines - Microsoft Fabric | Microsoft Learn

I check and there does not seem to be any open issues about this: Fabric known issues - Microsoft Fabric | Microsoft Learn

Why can I not deploy a Data Pipeline in a Fabric Deployment Pipelines?

Here is my setup: 

I have WorkspaceA and WorkspaceB.

I have a Fabric deployment pipeline defined with only 2 stages - deploy from WorkspaceA to WorkspaceB.

I only have a Data Pipeline resource in WorkspaceA and nothing else. 

I have a Service Principal with the Contributor role assigned to both workspaces and Admin role assigned to the deployment pipeline. 

Here are the series of REST calls to trigger the deployment pipeline and get the status afterwards: 

# Get deployment pipelines service principal has access to
GET https://api.fabric.microsoft.com/v1/deploymentPipelines
content-type: application/json
Authorization: Bearer

# Get deployment stages for pipeline I want
GET https://api.fabric.microsoft.com/v1/deploymentPipelines/<guid>/stages
content-type: application/json
Authorization: Bearer

# Deploy from WorkspaceA to WorkspaceB
POST https://api.fabric.microsoft.com/v1/deploymentPipelines/<guid>/deploy
content-type: application/json
Authorization: Bearer

{
"sourceStageId": "<guid>",
"targetStageId": "<guid>"
}

# Get status
GET https://api.fabric.microsoft.com/v1/operations/<x-ms-operation-id retrieved from header of previous call>
content-type: application/json
Authorization: Bearer

Repose I receive:
HTTP/1.1 200 OK
Cache-Control: no-store, must-revalidate, no-cache
Pragma: no-cache
Content-Length: 303
Content-Type: application/json; charset=utf-8
Content-Encoding: gzip
Strict-Transport-Security: max-age=31536000; includeSubDomains
X-Frame-Options: deny
X-Content-Type-Options: nosniff
RequestId: <guid>
Access-Control-Expose-Headers: RequestId
request-redirected: true
home-cluster-uri: https://wabi-us-east-a-primary-redirect.analysis.windows.net/
Date: Mon, 02 Jun 2025 06:13:25 GMT
Connection: close

{
"status": "Failed",
"createdTimeUtc": "2025-06-02T05:59:52.5077701",
"lastUpdatedTimeUtc": "2025-06-02T05:59:54.0234048",
"percentComplete": null,
"error": {
"errorCode": "DeploymentOperationFailed",
"moreDetails": [
{
"errorCode": "UnknownError",
"message": "An error occurred while processing the operation",
"relatedResource": {
"resourceId": "<guid>",
"resourceType": "DataPipeline"
}
}
],
"message": "Failed to deploy content to target stage"
}
}

[garddolau]

Hi @kris MS support should now have now resolved this issue.  I've confirmed its fixed in these regions UK South and West Europe

[krisjones]

@garddolau  This now works for me and I didn't not change anything in my code so I can confirm that it works in North Europe Azure region.

[burakkaragoz]

Hi @krisjones ,

I’ve reviewed all the recent updates and the discussion so far. As mentioned previously, this issue with deploying Data Pipelines between workspaces using the Fabric Deployment Pipelines REST API is a known limitation. Even when permissions and workspace bindings are correct, the REST API doesn’t always support Data Pipeline artifact deployments as expected, especially compared to what you can do via the UI.

Here’s a summary of the main points:

• Deployment of Data Pipelines between workspaces through the UI works reliably, but the REST API is not fully supported for this scenario yet.
• Service Principal permissions, admin rights, and matching workspace capacities are all required, but even with these set up perfectly, the API can still return errors like the one you’re seeing.
• Keeping your pipelines and dependencies as simple as possible can sometimes help, but it’s not a guaranteed fix.
• The REST API is still evolving. For now, the safest way to move Data Pipelines between workspaces is using the UI or exporting/importing definitions manually.
• If you need to automate, the fabric-cicd Python library and PowerShell scripts are worth a try, but they may hit the same limitations as direct REST API calls.
• If you keep running into the same error, raising a Microsoft support ticket and providing full details is a good step. Sometimes there are undocumented limitations or workarounds available from the product team.

If you (or anyone else following this thread) find a reliable workaround, please share it here so the community can benefit. I’ll also keep an eye out for any updates or fixes and will tag you if I see something new.

Thanks again for your detailed troubleshooting and for helping to surface these real-world issues. If you need help with logs or want to try any step-by-step checks, just let me know!

[garddolau]

Can anyone for MS confirm there is currently a known limitation/issue for this combination please?

(a) Deployment Pipelines REST API = https://api.fabric.microsoft.com/v1/deploymentPipelines/{{deployment_pipeline_id}}/deploy

(b) itemType = DataPipeline

(c) via Service Principal

(d) Service Principal has admin access on all 3 workpsaces inlcude in the deployment pipeline

(e) Service Principal has admin access on the deployment pipeline

(f) Method being used to post to the API works fine other item types eg(Lakehouse, VariableLibrary, SemanticModel, Notebook)

(g) When completing the test I am only trying 1 DataPipeline

(h) When completing the test the Data Pipeline being used to test is empty so has no dependencies

[v-veshwara-msft]

Hi @garddolau ,
Thanks for checking on this.
At the moment, there’s no official documentation confirming this as a known issue, but the behavior you’re seeing is consistent with other recent reports.

As noted, the recommendation from Microsoft Support in this case was to avoid using Deployment Pipelines for Data Pipelines and instead implement automation using REST APIs to export and recreate pipelines across workspaces.

This REST-based approach appears to be working reliably and may be the best workaround until more formal guidance or fixes are provided.

Since a support ticket has already been raised by @krisjones and the matter is under discussion with Microsoft Support, it would be best to wait for the outcome of that investigation.

However, if this is blocking a critical scenario on your end, raising a separate support ticket would help ensure it receives the appropriate attention for your specific context.

If any updates or confirmations become available, it would be helpful to share them here for the benefit of others facing similar challenges.

Hope this helps. Please reach out for further assistance.

Thank you.

[KevinChant]

Have you tested with the fabric-cicd Python library instead of the pure API calls to see if that works? 

https://microsoft.github.io/fabric-cicd/0.1.19/ 

If so, the underlying cause may be with your API call.

[krisjones]

Thanks for the suggestion, 
I have not used that library but what I have done is use Microsoft's own PowerShell script here: 
fabric-samples/features-samples/fabric-apis/DeploymentPipelines-AssignToNewDeploymentPipelineAndDepl...
And I get the exact same error.

[burakkaragoz]

Hi @krisjones ,

I’ve reviewed your detailed description and the follow-up responses, and I can confirm this is a known friction point in Fabric—especially when it comes to handling Data Pipelines in deployment pipelines via the REST API.

Key points and steps to consider:

1. Workspace Permissions:
   Double-check that both WorkspaceA and WorkspaceB have the same Fabric capacities, and that your service principal (or user) has at least Contributor rights on both, and Admin on the deployment pipeline itself. Even a small mismatch here can cause silent failures.

2. Pipeline Binding:
   After assigning the deployment pipeline to WorkspaceB, try using the UI to manually bind WorkspaceB as the deployment stage, if possible. Sometimes the deployment pipeline might not recognize the target workspace for the artifact if the binding isn’t clear.

3. Data Pipeline Dependencies:
   Data Pipelines can be more sensitive than datasets or reports because their dependencies (linked services, datasets, parameters, etc.) might not exist or be identical in the target workspace. If anything referenced in the pipeline is missing or different, the deployment may fail—even if it works fine manually in the UI.

4. Try Exporting/Importing:
   As a workaround, try exporting the pipeline definition from WorkspaceA and importing it into WorkspaceB directly. This sometimes surfaces dependency or compatibility errors that aren’t obvious from the REST API response.

5. Check Pipeline Type:
   Are you using the new Fabric-native Data Pipelines or legacy Dataflows Gen2? Some behaviors and limitations can differ, especially around workspace-to-workspace deployments and dependency resolution.

6. REST API Limitations:
   The Deployment Pipelines REST API is still maturing and may not fully support all artifact types with workspace-to-workspace deployments, as confirmed in the official documentation and by community feedback.

Summary of what you can try next:

• Re-check permissions and bindings as above.
• Export/import the pipeline definition as a test.
• Confirm all dependencies exist in the target workspace.
• If possible, try with a simplified pipeline (no parameters, minimal dependencies) to isolate the problem.
• If the issue persists, using the Data Pipeline REST API directly (instead of the Deployment Pipeline API) for automation can sometimes be more reliable.

If you continue to hit the same error, you might want to:

• Raise a support ticket with Microsoft, including the full API request and response details.
• Follow the official Fabric documentation and release notes for any updates on deployment pipeline support for Data Pipelines.

Let me know if you’d like to troubleshoot specific error messages or want a step-by-step on exporting/importing pipelines. I’m happy to help further!

If this helps, please consider marking my response as Accepted Solution so other users can find it easily.

[garddolau]

Hi, I have the same issue and created a post before I found yours, https://community.fabric.microsoft.com/t5/Fabric-platform/Deployment-Pipelines-API-Error-with-itemTy...

If you find a fix let me know

[v-veshwara-msft · Accepted solution]

Hi @krisjones ,

Thanks for posting in Microsoft Fabric Community and for sharing the details.

Based on the available information and testing, it appears that while Data Pipelines are supported in Fabric Deployment Pipelines through the UI, there may be limitations when using the Deployment Pipelines REST API to deploy Data Pipelines between workspaces.

This observation is aligned with the current state of the APIs--while the Data Pipeline REST API supports managing Data Pipelines for CI/CD scenarios, integrating this with the Deployment Pipelines REST API for workspace-to-workspace promotion might not be fully supported yet.

Additional Suggestions:

1. Check the following resources to ensure all configurations align with best practices and required permissions:

   • Required permissions for popular actions

   • Pipeline automation in Fabric

   • Best practices for CI/CD in Fabric

2. If the issue persists, consider using the Data Pipeline REST API directly for deployment automation.

If needed, raising a support ticket with Microsoft, providing the full API response and request details, might help clarify the current limitations and available workarounds.

Hope this helps. Please reach out for further assistance.
If this post helps, then please consider to give a kudos and Accept as the solution to help the other members find it more quickly.

Thank you.

[v-veshwara-msft]

Hi @krisjones ,

Just checking in to see if your query has been resolved and whether any of the responses were helpful. If so, kindly consider marking the relevant reply as the Accepted Solution to assist others who may have a similar question.

Otherwise, feel free to reach out for further assistance.

Additionally, could you please confirm whether the issue has been addressed through the support ticket with Microsoft?

If the issue is now resolved, we’d greatly appreciate it if you could share any key insights or the resolution here for the benefit of the wider community. We may consider closing this thread, if we dont hear back.

Should you need further assistance in the future, please feel free to create a new post in the Microsoft Fabric Community Forum. We’ll be glad to support you.

Thank you for your time and participation.

[krisjones]

The query has not been resolved. I had a Microsoft Support engineer contact me - the matter is still being discussed. 

For my specific scenario, I'm looking at Option 4 from this Microsoft link: 

CI/CD workflow options in Fabric - Microsoft Fabric | Microsoft Learn
The recommendation is to not use Fabric deployment pipelines but build pipelines that use REST calls. And this is what I will implement. So instead of relying on Fabric's Deployment Pipelines to deploy data pipelines, I implemented REST calls to copy the data pipeline from one workspace to another: 

List pipelines for source workspace: 

GET https://api.fabric.microsoft.com/v1/workspaces/{{<source workspace id>}}/dataPipelines

Get the definition for a particular data pipeline for the GET 

POST https://api.fabric.microsoft.com/v1/workspaces/{{<source workspace id>}}/items/{{pipelineId}}/getDefinition

Create the data pipeline in the target workspace: 

POST https://api.fabric.microsoft.com/v1/workspaces/{{<taget workspace id>}}/dataPipelines

[v-veshwara-msft]

Thanks for the update and for explaining the approach you're taking.

It's helpful to hear that you're moving ahead with an API-based deployment process, using REST calls to manage content across multiple workspaces. This method is especially useful when you're working in scenarios where each customer or environment has its own workspace.

Thanks also for outlining the key steps you followed.

This is a practical approach and will likely help others working in similar multi-workspace or multi-customer setups.

Since your case is still active with Microsoft Support, once there's a confirmed resolution or update, it would be great if you could share the outcome here for the benefit of others.

In the meantime, please consider marking any relevent responses as the accepted solution so it’s easier for others to find relevant guidance when facing similar challenges.

Regards,
Vinay.

[v-veshwara-msft]

Hi @krisjones ,

As we haven’t heard from you in a while, just checking in to see if your query has been resolved and whether there has been any update from Microsoft Support. If the issue has been addressed, please consider sharing any insights or the resolution here to help others facing similar challenges.

If you require further assistance in the future, feel free to create a new post in the Microsoft Fabric Community Forum. We’ll be glad to support you.

Thank you for your time and participation.

[krisjones]

Hi, the query has not been resolved. As stated, I decided to take a differen approach where I don't utilize Fabric Deployment Pipelines but use Fabric's REST API and possibly this package: fabric-cicd
Microsoft has not replied to my questions.

[v-veshwara-msft]

Hi @krisjones ,

Thanks for the update and for sharing the approach you've taken.

Hopefully, Microsoft Support will get back to you soon with a clear response. In the meantime, your decision to move forward using Fabric's REST APIs, along with the fabric-cicd package, sounds like a practical workaround, especially considering the current limitations with Deployment Pipelines and Data Pipelines.

If you do receive any further information or resolution from Support, it would be appreciated if you could share it here for the benefit of others following similar paths.

Should you need any assistance going forward, please feel free to create a new post in the Microsoft Fabric Community Forum. We’ll be glad to support you.

Thank you for being part of Microsoft Fabric Community.

[krisjones]

Hi, to answer you suggested check:

Workspace permissions: both workspaces have the same capacity and as written I have a service principal that is assigned the Contributor role to both workspaces and the Admin role to the Fabric Deployment pipeline. 

Pipeline binding: As written, when I run the pipeline manually using the UI - the data pipeline deployes fine. Also when I have other items other than a data pipeline like a notebook or a lakehouse - they deploy fine manually or using the REST API.

Data Pipeline dependencies: My data pipeline is vanilla i.e., it's empty ( I also tried with a simple Lookup activity)

Try exporting manually: I tried and it imports fine since the pipeline is vanilla. 

Are you using the new Fabric-native Data Pipelines or legacy Dataflows Gen2? I'm using the native-Fabric data pipeline where you define activities.

[burakkaragoz]

Hi @krisjones ,

Yeah, I’ve seen this behavior before and it usually comes down to how Fabric handles Data Pipelines in deployment pipelines — they’re still a bit finicky compared to datasets or reports.

A couple of things to check:

• Workspace permissions: Make sure both Workspace1 and Workspace2 have the same Fabric capacities and that the user triggering the deployment has at least Contributor access on both. Sometimes the pipeline fails silently if it can’t write to the target workspace.

• Pipeline binding: After assigning the deployment pipeline to Workspace1, did you also bind Workspace2 to the Test stage explicitly? If not, the deployment might not know where to push the artifact.

• Data Pipeline dependencies: If your Data Pipeline references linked services, datasets, or parameters that are not present or not identical in Workspace2, the deployment might fail. Unlike reports, Data Pipelines don’t always bring their dependencies cleanly.

• Try exporting manually: As a sanity check, try exporting the Data Pipeline JSON from Workspace1 and importing it manually into Workspace2. If that fails too, the issue is likely with the pipeline definition itself.

Also, just to confirm — are you using the new Fabric-native Data Pipelines or legacy Dataflows Gen2? The behavior can differ slightly depending on that.

Let me know what you find, happy to dig deeper if needed.

If my response resolved your query, kindly mark it as the Accepted Solution to assist others. Additionally, I would be grateful for a 'Kudos' if you found my response helpful.