Forum Discussion

Anonymous's avatar
Anonymous
Not applicable
1 year ago
Solved

Dataflows vs Data Pipelines vs Deployment Pipelines

I have 4 Gen2 dataflows in a data pipeline, in a Dev workspace.   Initially the dataflows are the standard Gen2 dataflows, which were in a data pipeline, but won't work in a Deployment Pipeline.  ...
data engineering
  • nilendraFabric's avatar
    nilendraFabric
    1 year ago

    Hello Anonymous 

    I understand this is not ideal

    CI/CD-enabled Dataflows Gen2 cannot be directly added to a Data Pipeline for orchestration ( As you mentioned). Only standard Dataflows Gen2 are supported in Data Pipelines.

    One option is to split or maintain 2 copied - ( Not Ideal) 

    • Use standard Dataflows Gen2 within your Data Pipeline for orchestration.
    • Separately manage CI/CD-enabled Dataflows for deployment purposes until full integration support is available.

    Once CI/CD-enabled Dataflows are available in Data pipelines , delete the Non CICD one 

    Hope this helps 

    Thanks

     

     

v-menakakota's avatar
v-menakakota
Community Support
1 year ago

Hi Anonymous ,

May I ask if you have resolved this issue? If so, please mark the helpful reply and accept it as the solution. This will be helpful for other community members who have similar problems to solve it faster.

Thank you.