topic Re: Dataflows vs Data Pipelines vs Deployment Pipelines in Data Engineering

Dataflows vs Data Pipelines vs Deployment Pipelines

Anonymous — Fri, 07 Feb 2025 16:47:33 GMT

I have 4 Gen2 dataflows in a data pipeline, in a Dev workspace.

Initially the dataflows are the standard Gen2 dataflows, which were in a data pipeline, but won't work in a Deployment Pipeline.

So I began creating the new CI/CD Gen2 dataflows, which I've read do work in a deployment pipeline, but I can't add them to a data pipeline (when I add the dataflow step the CI/CD dataflow isn't there, but the other ones are).

What can I do to have a series of dataflows, that are in a pipeline, and work with deployment pipelines?

Re: Dataflows vs Data Pipelines vs Deployment Pipelines

nilendraFabric — Fri, 07 Feb 2025 16:58:46 GMT

Hello Anonymous

I understand this is not ideal

CI/CD-enabled Dataflows Gen2 cannot be directly added to a Data Pipeline for orchestration ( As you mentioned). Only standard Dataflows Gen2 are supported in Data Pipelines.

One option is to split or maintain 2 copied - ( Not Ideal)

Use standard Dataflows Gen2 within your Data Pipeline for orchestration.
Separately manage CI/CD-enabled Dataflows for deployment purposes until full integration support is available.

Once CI/CD-enabled Dataflows are available in Data pipelines , delete the Non CICD one

Hope this helps

Thanks

Re: Dataflows vs Data Pipelines vs Deployment Pipelines

v-menakakota — Mon, 10 Feb 2025 07:02:32 GMT

Hi Anonymous ,

Thank you for helpful respose nilendraFabric ,Standard Gen2 Dataflows can be added to a data pipeline for orchestration but won't work in a deployment pipeline.I request you to follow the suggestions provided by Nilendra and let us know if it was helpful.If you still need any assistance feel free to reachout.

Regards,
Menaka.

Re: Dataflows vs Data Pipelines vs Deployment Pipelines

v-menakakota — Thu, 13 Feb 2025 10:18:31 GMT

Hi Anonymous ,

May I ask if you have resolved this issue? If so, please mark the helpful reply and accept it as the solution. This will be helpful for other community members who have similar problems to solve it faster.

Thank you.