Forum Discussion

Gpop13's avatar
Gpop13
Icon for Advocate IV rankAdvocate IV
1 year ago
Solved

Dataflow gen2 in deployment pipelines

The dataflow gen2 has been listed as preview feature in the deployment pipeline. https://learn.microsoft.com/en-us/fabric/cicd/deployment-pipelines/intro-to-deployment-pipelines?tabs=new-ui   howe...
CICD
  • v-prasare's avatar
    v-prasare
    1 year ago

    Your feedback is extremely valuable and highlights a key area for improvement. as Microsoft's official support for for pipeline orchestration support with CI/CD-enabled Dataflow Gen2 is limited we recommend to submit your ideas here to Fabric team: https://ideas.fabric.microsoft.com/ideas/search-ideas/

    as feedback given here will closely monitored by PG team. We are committed to ensuring our solutions meet your business needs and workflows effectively. Thank you for your patience and understanding as we work to enhance this functionality.

Stinkys's avatar
Stinkys
Icon for Advocate II rankAdvocate II
1 year ago

I am hitting the same issue. How are people actually deploying DFG2 resources to multi-stage environments via pipelines currently? Or is it just that nobody is doing it because it's not possible?

Gpop13's avatar
Gpop13
Icon for Advocate IV rankAdvocate IV
1 year ago

Hi Stinkys - The DFG2 are still not supported in depl pipelines. But DFG2 CICD versions are.

I did try that approach but the biggest issue I faced was even if the DFG2 is deployed to next stage, all the destination settings point to the source stage. Example, if in a DFG2, Query1 writes to a lakehouse table in dev env, after its deployed to Test, this setting will still be the same and NOT pointing to corressponding Test lakehouse. 

This was a very manually work for me. 

In the first go, I recreated all the DFG2 in all stages, updated the target connections accordingly. Now whener there is a change to existing query, I manually update (advanced query) in each env (I know this is not the best solution but closest to an easier approach I could have)

  • Stinkys's avatar
    Stinkys
    Icon for Advocate II rankAdvocate II
    1 year ago

    Thanks for the response Gpop13 . I am facing the exact same issue as you. I guess manually updating the default data destination in the next stage, as you are, is the only answer. Looks like the functionality we want will be introduced sometime in Q3 this year.

    • Gpop13's avatar
      Gpop13
      Icon for Advocate IV rankAdvocate IV
      1 year ago

      Yeah, I even checked with Microsoft by opening a ticket but they mentioned this is a limitation for now. Unless someone has any other better solution, I guess that is what we have to live with as of now. Where did you see Q3 for this feature, can you share please?