DataflowGen2 CICD -> Deployment pipeline rule support

[anon97242]

Greetings!

We've been working on getting a project working with DataflowGen2, we moved to using DataflowGen2 CICD projects to make use of source control and deployment pipelines.

 However, we are unable to assign rules to different environments in the pipeline process. Which seems to negate the purpose of moving to CICD in the firstplace. In doing so we also lose support for datapipelines, which would also enable some form of specifying different destinations. 

Do we know when support for rules in deployment pipelines will be allowed for DataflowGen2 CICD Items will be happening, as it is not apparent in the roadmap? Or looking the other way when DataflowGen2 will get CICD support, and we no longer have this functional misalignment on the same tool ( DataflowGen2 vs DataflowGen2 CI/CD ) ?

Thanks! 

[anon97242]

Thanks for the confirmation @girishthimmaiah . This confirms our understanding of the limitations of DataflowGen2 and Dataflowgen2 CICD.

@v-veshwara-msft  we'll be voting on the the fabric ideas. We're unlikely to proceed with DataflowGen2 or DataflowGen2 CICD for larger projects and will be looking at other options as both pieces of functionality have rather large limitations and little functional overlap. At the moment they are almost entirely different products, so responding with what is possible with DataflowGen2 is not helpful unfortunately.

[v-veshwara-msft]

Hi @anon97242 ,

Thank you for reaching out regarding deployment pipeline rules for Dataflow Gen2 CI/CD projects.
Thanks @girishthimmaiah  for your insights,

As of now, Dataflow Gen2 supports deployment rules, allowing different environments to have distinct configurations. You can define data source rules and parameter rules in deployment pipelines. However, there are still some limitations, such as the inability to create rules at the development stage.

For details on creating deployment rules, please refer to the official Microsoft documentation:
Create deployment rules in Fabric ALM

Additionally, community discussions on related topics can be found here:
Dataflow-gen2-in-deployment-pipelines 
If you're looking for additional improvements, I’d encourage you to upvote and share feedback on Microsoft's ideas platform. This helps the engineering team prioritize feature updates.
Microsoft Idea
We will notify you if there are any updates regarding deployment rules for Dataflow Gen2.

Additionally, I recommend following the Microsoft Fabric roadmap and community updates for upcoming enhancements. If you need further clarification, feel free to ask!

If you find this post helpful, please mark it as 'Accept as Solution' to assist others with similar queries.

Best regards,
Vinay.

[Mauro89]

Another point on the intermediate state. If there are already Dataflow Gen2 items, I haven't found an option to enhance an "old" Dataflow Gen2 into the new version.

Would be nice to have it on the roadmap soon to get Dataflow Gen2 "production ready".

[girishthimmaiah · Accepted solution]

I see your trouble with the current limitations in DataflowGen2 CICD, especially around deployment pipeline rules and the lack of support for data pipelines. It does seem counterintuitive to adopt CICD if you can’t assign rules to different environments properly.

As of now, there hasn’t been an official announcement on when deployment pipeline rules will be supported for DataflowGen2 CICD. I completely understand how important this is for a smooth CICD workflow, and unfortunately, the roadmap doesn’t provide any clear indications either.

Regarding full CICD support for DataflowGen2 itself, there also hasn’t been a confirmed timeline. It’s definitely a pain point, as it creates this misalignment where using DataflowGen2 directly provides different functionality than the CICD version.

That being said, I’d recommend keeping an eye on Microsoft’s roadmap updates and community forums, as changes like these often get rolled out in preview before being widely available. If I come across any updates, I’ll be sure to share them.

"If this answer resolves your issue, kindly mark it as a solution. It helps others with similar queries. Thanks!"