Multiple dataflows in pipeline

Hi ManasiL ,

Thanks for using Fabric Community.

For ingesting data from multiple tables with millions of records each, creating one pipeline per domain with parallel dataflows is generally the recommended approach. Here's why:

• Efficiency: Parallel execution within a single pipeline allows tables to be ingested concurrently, significantly reducing overall processing time compared to sequential execution in separate pipelines.
• Scalability: A single domain pipeline simplifies scaling if needed in the future. You can adjust parallelism within the pipeline to handle increased data volume.
• Resource Optimization: Fabric can leverage its distributed processing capabilities more effectively with parallel dataflows.

Benefits of this Approach:

• Faster Ingestion: Parallel execution significantly improves processing speed.
• Simplified Management: Easier to monitor and manage a single domain pipeline compared to numerous individual table pipelines.
• Scalability: Flexibility to scale the pipeline by adjusting dataflow parallelism or adding compute resources.

When Might Individual Pipelines or a Sequential Approach can be considered?

• Complex Dependencies: If your tables have intricate dependencies that require specific execution order, individual pipelines or a sequential approach might be necessary to maintain data integrity. However, double-check if these dependencies can be addressed within a single pipeline using conditional logic within dataflows.

• Very Small Tables: If some tables within your domain are extremely small and ingesting them sequentially won't significantly impact overall processing time, a sequential approach within the domain pipeline could be considered.

At last it is completely depends on your scenario, data volume and your pipeline management, as discussed in previous thread: Solved: Can create multiple data flow gen2 inside one pipe... - Microsoft Fabric Community you can also considered the points mentioned in it.

Hope this is helpful. Please do let me know incase of further queries.

Thank you for your reply.

So if going with parallel execution of dataflows what if one of the dataflow fails? Even if we run the other dataflows, later on can we just run that one failed dataflow?