Gen2 dataflows and deployment pipelines

[mikesmall]

Hi

We are looking to move from legacy dataflows to Gen2, but in my testing I can't see any Gen2 dataflows in the deployment pipeline. 

I wonder if it is because lakehouses are not suported yet and they require staging in order to be published, so need the system generated staging lakehouse.  I didn't set a destination in the dataflow to keep it simple for testing.  I also tried recreating the dataflows in Dev and Test, hoping that if the same dataflow existed in both it might then deploy (as the staging lakehouses had been created) but will can't see any Gen2 artifacts in the pipeline.

Has anyone managed to use deployment pipelines with Gen2 dataflows?

Thanks.

[Joelon]

Is there any alternative to overcome this shortcoming?

[abpgupta]

I am wondering then what will be approach for separating Development to Production for all related objects. Some managed with deployment pipeline and some with git integration?

[cyibbs]

Did you ever figure this out? I'm in the same boat. Trying to transition to Gen2 DF's and just realized I can't see them in my existing deployment pipeline. 😕

[raschaoot]

As of June 2024, Gen2 data flows are still not supported in deployment pipelines. I also cannot find a mention on any roadmap.

Can anyone confirm when this feature is expected? We cannot take a system into production that uses Gen2 flows without deployment pipeline support.

Thanks!

[DanielAmbler]

Second this - could not recommend moving to production until there is full support for deployment (one way or another) for DFG2.

[cwilce]

Just adding this from reddit where the same question was asked https://www.reddit.com/r/MicrosoftFabric/comments/1dogga0/dataflows_gen2_with_deployment_pipelines/

TLDR: Gen2 Dataflows with Deployment Pipelines isn't in the current roadmap which goes up until October. Check back then to get the next 6 month roadmap (which will hopefully have this feature included!).

[mikesmall]

I was in some Microsoft Azure training on Tuesday and asked the question, the confirmed it was not possible and there was no timeline.  Seems to be a massive oversight and a real barrier in moving to Fabric.

[alfBI]

Exactly in the same situation.  Instead of the Dataflow Gen2 

the depoyment pipeline shows the lakehouse used by the DF Gen2 for staging purposes......

What is more when we recreated the deployment pipeline following error message appeared

So, seems clear that DF Gen2 are not yet supported by now (no idea if the will available after end of preview phase)

[mikesmall]

No, my guess is this is something they are working on for GA.

There is obviously an issue with the auto generated staging lakehouses for Gen2 dataflows (eg DataflowsStagingLakehouse) as they shouldn't be visible, so I suspect that could be the reason the artifacts can't be deployed with a pipeline yet.

[abpgupta]

This is still not available after GA.  

[mikesmall]

I have had a look at the roadmap and in Q2 2024 they are going to provide Git integration for dataflows, I wonder if this is the route they will go down for deploying content.  Would be useful to know if including Gen 2 DFs in deployment pipelines was even a consideration though.