Forum Discussion

Anonymous's avatar
Anonymous
Not applicable
1 year ago
Solved

Dataflows vs Data Pipelines vs Deployment Pipelines

I have 4 Gen2 dataflows in a data pipeline, in a Dev workspace.

 

Initially the dataflows are the standard Gen2 dataflows, which were in a data pipeline, but won't work in a Deployment Pipeline.

 

So I began creating the new CI/CD Gen2 dataflows, which I've read do work in a deployment pipeline, but I can't add them to a data pipeline (when I add the dataflow step the CI/CD dataflow isn't there, but the other ones are).

 

What can I do to have a series of dataflows, that are in a pipeline, and work with deployment pipelines?

data engineering
  • nilendraFabric's avatar
    nilendraFabric
    1 year ago

    Hello Anonymous 

    I understand this is not ideal

    CI/CD-enabled Dataflows Gen2 cannot be directly added to a Data Pipeline for orchestration ( As you mentioned). Only standard Dataflows Gen2 are supported in Data Pipelines.

    One option is to split or maintain 2 copied - ( Not Ideal) 

    • Use standard Dataflows Gen2 within your Data Pipeline for orchestration.
    • Separately manage CI/CD-enabled Dataflows for deployment purposes until full integration support is available.

    Once CI/CD-enabled Dataflows are available in Data pipelines , delete the Non CICD one 

    Hope this helps 

    Thanks

     

     

3 Replies

  • nilendraFabric's avatar
    nilendraFabric
    Super User
    1 year ago

    Hello Anonymous 

    I understand this is not ideal

    CI/CD-enabled Dataflows Gen2 cannot be directly added to a Data Pipeline for orchestration ( As you mentioned). Only standard Dataflows Gen2 are supported in Data Pipelines.

    One option is to split or maintain 2 copied - ( Not Ideal) 

    • Use standard Dataflows Gen2 within your Data Pipeline for orchestration.
    • Separately manage CI/CD-enabled Dataflows for deployment purposes until full integration support is available.

    Once CI/CD-enabled Dataflows are available in Data pipelines , delete the Non CICD one 

    Hope this helps 

    Thanks

     

     

  • v-menakakota's avatar
    v-menakakota
    Community Support
    1 year ago

    Hi Anonymous ,

    Thank you for helpful respose  nilendraFabric ,Standard Gen2 Dataflows  can be added to a data pipeline for orchestration but won't work in a deployment pipeline.I request you to follow the suggestions provided by Nilendra and let us know if it was helpful.If you still need any assistance feel free to reachout.

    Regards,
    Menaka.

  • v-menakakota's avatar
    v-menakakota
    Community Support
    1 year ago

    Hi Anonymous ,

    May I ask if you have resolved this issue? If so, please mark the helpful reply and accept it as the solution. This will be helpful for other community members who have similar problems to solve it faster.

    Thank you.