Forum Discussion
Can create multiple data flow gen2 inside one pipeline?
Ideally the best way would have been to have a single pipeline and a single dataflow Gen 2 with a meta data driven framework but as of today, dataflow Gen 2 cannot be parameterized.
So owning to that, you would have to create sepearte dataflows per table.
Now the question is should those be triggered or integrated within a signle pipeline or different pipelines?
For that, it depends on your requirement :
1) Is the load for all tables supposed to be scheudled at the same time?
2) Is there any flow dependency between the tables? meaning should those be some sequentail / dependant flow or all tables are independant of each other?
If those tables are to be loaded atthe same time, then the best way would be for you to craete a signle pipeline and integrate all the dataflows within the same pipeline.
Okay. got it. 👍
Thank you once again for the prompt responses.
Super UserAll the dataflows are using the same workspace capacity is the major thing to take into consideration.
In case if the DFs are independant, then executing them parallely would the best approach but executing all DFs in parallel might choke up the workspace capacity; so you might have to execute DFs in parallel in batches and not all at the same time
Yes they are within the same workspace. Can you please share the link to see how to implement them in parallel in batches?