Best practices por CI/CD for Data Pipelines

[amaaiia]

It is recommended as CI/CD best practice in Fabric to create working branches on top of the main branch to implement changes, working with a temporary parallel workspace for that temporary branch, and once the changes are complete, integrate them into the main branch through Git.

I guess this is as a good practice for notebooks. I wonder if it is a good practice for Data Pipelines. There are pipeline components that create connections to data, such as a Lookup activity, Script, etc., and when creating a temporary Git branch that is associated with a different workspace in Fabric, the connections to those Warehouses, Lakehouses, etc. no longer work because you can only make data connections on the same workspace. This is what happens when I open a Data Pipeline from a temporary branch created from main branch integrated in a new workspace:

There are also other problems with other kind of activities, such as Outlook activities, that get misconfigured.

I would like to know what is the best CI/CD practice to make modifications on a Data Pipeline. I can think of making a copy and edit the copy on the same workspace, but it doesn't seem to be a good solution because then those changes that are made, once validated, would have to be implemented again in the main Pipeline, it is not enough to delete the main one and rename the new one, because Fabric indicates that there is already a Pipeline with that name, and besides, all the configuration associated with the main Pipeline would be lost.

Which is the best practice for Data Pipeline CI/CD?

[avishkarthorat]

Hi, I wanted to know Can we trigger or scedule the deployment pipeline of fabric from Azure devops and how ?

[v-nikhilan-msft]

Hi @amaaiia 
Thanks for using Fabric Community.
Just sharing this link if this might help you.
Best practices for lifecycle management in Fabric - Microsoft Fabric | Microsoft Learn

Please let me know if you have any further questions.

[amaaiia]

Hi @v-nikhilan-msft 

I had it in mind. Not useful because it doen't say anything specific about CI/CD for Data Pielines management.

[v-nikhilan-msft]

Hi @amaaiia 
The internal team replied as follows:
You can find how to work with branches here : https://learn.microsoft.com/en-us/fabric/cicd/git-integration/manage-branches

The idea is that the workspace (at least, all supported items) is copied through the new branch to the new workspace, and connections between items in the same workspace should be automatically switched to be get the same structure and relations between items in the new workspace. 
For data pipelines, if you identify that it is not the case, we need specific details to be able to investigate and understand if some is not supported yet, or if it is a bug.

Hope this helps. Please let me know if you have any further questions.

[amaaiia]

Hi,

Warehouse is not supported yet, so every LookUp activity or Script activitie that reads/writes from/to warehouse tables, is getting an error.

So, for now, until warehouse is integrated with Git, which is the best approeach to work with Data Pipeline CI/CD? If I do as recommended (https://learn.microsoft.com/en-us/fabric/cicd/git-integration/manage-branches), I'm having issues with pointers to warehouse tables. Otherwise, if I copy the Data Pipeline as a new one in the same workspace as temporary way to work with Data Pipelines, is very difficult to integrate the copy with the main one, because I hace to repeat all changes into the main.

[v-nikhilan-msft · Accepted solution]

Hi @amaaiia 
Warehouse should be supported soon in git. You can either manually create the non-Git supported workspace items (i.e. warehouse) or edit the pipeline JSON definition to reference the warehouse from the other workspace.

Hope this helps.

[v-nikhilan-msft]

Hi @amaaiia 
We haven’t heard from you on the last response and was just checking back to see if your query got resolved. Otherwise, will respond back with the more details and we will try to help.
Thanks