Gen2 dataflows and deployment pipelines

[mikesmall]

Hi

We are looking to move from legacy dataflows to Gen2, but in my testing I can't see any Gen2 dataflows in the deployment pipeline. 

I wonder if it is because lakehouses are not suported yet and they require staging in order to be published, so need the system generated staging lakehouse.  I didn't set a destination in the dataflow to keep it simple for testing.  I also tried recreating the dataflows in Dev and Test, hoping that if the same dataflow existed in both it might then deploy (as the staging lakehouses had been created) but will can't see any Gen2 artifacts in the pipeline.

Has anyone managed to use deployment pipelines with Gen2 dataflows?

Thanks.

[abpgupta]

I am wondering then what will be approach for separating Development to Production for all related objects. Some managed with deployment pipeline and some with git integration?

[cyibbs]

Did you ever figure this out? I'm in the same boat. Trying to transition to Gen2 DF's and just realized I can't see them in my existing deployment pipeline. 😕

[alfBI]

Exactly in the same situation.  Instead of the Dataflow Gen2 

the depoyment pipeline shows the lakehouse used by the DF Gen2 for staging purposes......

What is more when we recreated the deployment pipeline following error message appeared

So, seems clear that DF Gen2 are not yet supported by now (no idea if the will available after end of preview phase)

[mikesmall]

No, my guess is this is something they are working on for GA.

There is obviously an issue with the auto generated staging lakehouses for Gen2 dataflows (eg DataflowsStagingLakehouse) as they shouldn't be visible, so I suspect that could be the reason the artifacts can't be deployed with a pipeline yet.

[abpgupta]

This is still not available after GA.  

[mikesmall]

I have had a look at the roadmap and in Q2 2024 they are going to provide Git integration for dataflows, I wonder if this is the route they will go down for deploying content.  Would be useful to know if including Gen 2 DFs in deployment pipelines was even a consideration though.