Data pipeline deployment using Deployment pipeline , issue with connection objects

[chetanhiwale]

While deploying data pipelines using Deployment pipelines, if a pipeline A has connection id in workspace A , after deploying it to another workspace B, the pipeline has the same connection id. Is there any way to change connection id too , like we do for semantic model using deployment rules. Currently for data pipeline, deployment rules are not supported, thus we need to manually change the connections in each pipeline.
Also what are the bestpracties with respect to handling connection objects with different envs, in git and deployment pipeline

My expected output is , after deploying pipeline A with connection id , we should be able change connection objects in other workspace, similarly we do using deployment rules.

[v-ssriganesh]

Hi @chetanhiwale 

May I ask if you have resolved this issue? If so, please mark the helpful reply and Accept it as the solution. This will be helpful for other community members who have similar problems to solve it faster.

Thank you.

[v-ssriganesh]

Hi @chetanhiwale 

Thanks for contacting the Microsoft fabric community forum.

You've identified correctly the limitation of deployment rules for data pipelines in Microsoft Fabric. There is currently no direct way to automatically change connection IDs as part of the deploy task - this means this has to be done manually, updating connection references in each pipeline for different environments.
While there isn't a direct solution in Fabric, we can use PowerShell scripting to automate the process of updating connection IDs within pipeline definitions.

If this helps then please Accept it as a solution and dropping a "Kudos" so other members can find it more easily.

Hope this works for you!

Thanks.

[chetanhiwale]

Thanks spencer_sa, for explaining the workaround. Currently we have also written scripts to handle the connection objects as you explained. But again, it would be very easily if Fabric has this functionality. Lets wait for Microsoft updates !!

[spencer_sa · Accepted solution]

As you've said, currently pipeline settings (and specifically connection settings are not changed between stages of a deployment pipeline - there's no rule you can use.
As far as changing the connection id, I suspect you're going to be in the same boat as us regarding this in so far as the option you might find does the trick is altering the pipeline using the API.
Couple of ways of orchestrating this - you could write your pipeline alteration code using PowerShell and run it from the many places you can execute PowerShell.
Another alternative is to use a Fabric notebook and semantic_link_labs (sempy) with the class FabricRESTClient to get the pipeline JSON, search/replace the connection string, and then rewrite it back to Fabric.
We will be using a Fabric notebook to rewrite our lakehouse/workspace ids.
I await the arrival of workspace parameters which will make some of this *so* much easier.

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.