Forum Discussion

Anonymous's avatar
Anonymous
Not applicable
1 year ago
Solved

Dataflows vs Data Pipelines vs Deployment Pipelines

I have 4 Gen2 dataflows in a data pipeline, in a Dev workspace.   Initially the dataflows are the standard Gen2 dataflows, which were in a data pipeline, but won't work in a Deployment Pipeline.  ...
data engineering
  • nilendraFabric's avatar
    nilendraFabric
    1 year ago

    Hello Anonymous 

    I understand this is not ideal

    CI/CD-enabled Dataflows Gen2 cannot be directly added to a Data Pipeline for orchestration ( As you mentioned). Only standard Dataflows Gen2 are supported in Data Pipelines.

    One option is to split or maintain 2 copied - ( Not Ideal) 

    • Use standard Dataflows Gen2 within your Data Pipeline for orchestration.
    • Separately manage CI/CD-enabled Dataflows for deployment purposes until full integration support is available.

    Once CI/CD-enabled Dataflows are available in Data pipelines , delete the Non CICD one 

    Hope this helps 

    Thanks

     

     

nilendraFabric's avatar
nilendraFabric
Super User
1 year ago

Hello Anonymous 

I understand this is not ideal

CI/CD-enabled Dataflows Gen2 cannot be directly added to a Data Pipeline for orchestration ( As you mentioned). Only standard Dataflows Gen2 are supported in Data Pipelines.

One option is to split or maintain 2 copied - ( Not Ideal) 

  • Use standard Dataflows Gen2 within your Data Pipeline for orchestration.
  • Separately manage CI/CD-enabled Dataflows for deployment purposes until full integration support is available.

Once CI/CD-enabled Dataflows are available in Data pipelines , delete the Non CICD one 

Hope this helps 

Thanks