topic Re: Multiple dataflows in pipeline in Dataflow

Multiple dataflows in pipeline

ManasiL — Wed, 19 Jun 2024 02:15:21 GMT

Hi,

I am working on Data Ingestion project on fabric for first time and need some guidance:

I have following architecture

Domain A

Table1..... Table 10

Domain B

Table 1.....Table 7

with each table having millions of records

I have to ingest data from all these tables from SAP HANA DB as source to fabric datawarehouse.

Should i create one pipeline for each domain containing dataflows for all the tables under that domain or should i create pipeline for each table. The tables are not dependent on each other.

Which approach is best considering the data size?

And if we go with one pipeline with mutiple dataflows should they be arranged sequentially or paraller?

Re: Multiple dataflows in pipeline

Anonymous — Wed, 19 Jun 2024 06:55:44 GMT

Hi ManasiL ,

Thanks for using Fabric Community.

For ingesting data from multiple tables with millions of records each, creating one pipeline per domain with parallel dataflows is generally the recommended approach. Here's why:

Efficiency: Parallel execution within a single pipeline allows tables to be ingested concurrently, significantly reducing overall processing time compared to sequential execution in separate pipelines.
Scalability: A single domain pipeline simplifies scaling if needed in the future. You can adjust parallelism within the pipeline to handle increased data volume.
Resource Optimization: Fabric can leverage its distributed processing capabilities more effectively with parallel dataflows.

Benefits of this Approach:

Faster Ingestion: Parallel execution significantly improves processing speed.
Simplified Management: Easier to monitor and manage a single domain pipeline compared to numerous individual table pipelines.
Scalability: Flexibility to scale the pipeline by adjusting dataflow parallelism or adding compute resources.

When Might Individual Pipelines or a Sequential Approach can be considered?

Complex Dependencies: If your tables have intricate dependencies that require specific execution order, individual pipelines or a sequential approach might be necessary to maintain data integrity. However, double-check if these dependencies can be addressed within a single pipeline using conditional logic within dataflows.

Very Small Tables: If some tables within your domain are extremely small and ingesting them sequentially won't significantly impact overall processing time, a sequential approach within the domain pipeline could be considered.

At last it is completely depends on your scenario, data volume and your pipeline management, as discussed in previous thread: Solved: Can create multiple data flow gen2 inside one pipe... - Microsoft Fabric Community you can also considered the points mentioned in it.

Hope this is helpful. Please do let me know incase of further queries.

Re: Multiple dataflows in pipeline

Anonymous — Thu, 20 Jun 2024 09:58:29 GMT

Hi ManasiL ,

We haven’t heard from you on the last response and was just checking back to see if your query was answered.
Otherwise, will respond back with the more details and we will try to help .

Thanks

Re: Multiple dataflows in pipeline

ManasiL — Thu, 20 Jun 2024 10:54:05 GMT

Thank you for your reply.

So if going with parallel execution of dataflows what if one of the dataflow fails? Even if we run the other dataflows, later on can we just run that one failed dataflow?

Re: Multiple dataflows in pipeline

Anonymous — Thu, 20 Jun 2024 11:02:39 GMT

Hi ManasiL ,

Yes you can re-run from the failed activity.

Docs to refer -
How to monitor pipeline runs - Microsoft Fabric | Microsoft Learn

Hope this is helpful. Please do let me know incase of further queries.

Re: Multiple dataflows in pipeline

Anonymous — Fri, 21 Jun 2024 10:03:13 GMT