Forum Discussion
Can create multiple data flow gen2 inside one pipeline?
Ideally the best way would have been to have a single pipeline and a single dataflow Gen 2 with a meta data driven framework but as of today, dataflow Gen 2 cannot be parameterized.
So owning to that, you would have to create sepearte dataflows per table.
Now the question is should those be triggered or integrated within a signle pipeline or different pipelines?
For that, it depends on your requirement :
1) Is the load for all tables supposed to be scheudled at the same time?
2) Is there any flow dependency between the tables? meaning should those be some sequentail / dependant flow or all tables are independant of each other?
If those tables are to be loaded atthe same time, then the best way would be for you to craete a signle pipeline and integrate all the dataflows within the same pipeline.
Okay. got it. 👍
Thank you once again for the prompt responses.
I really appreciate your prompt response. Thank you for that.
Lets keep the dependency aside for a minute.
Will it be still ok to have multiple dataflows each for a table inside a single pipeline if those tables have millions of records?
- NandanHegde
Super User
The dataflows gen 2 would use the same capacity tied to the workspace.
So whether you run them within the same pipeline or run them parallely via different pipelines, the capacity being utilized is the same. So it wont matter but in case if the capacity is getting throttled with some many runs, atleast you can make them sequential or depeandt in case if all are within the same pipeline and manage the capacity utilization
