Power BI is turning 10, and we’re marking the occasion with a special community challenge. Use your creativity to tell a story, uncover trends, or highlight something unexpected.
Get startedJoin us at FabCon Vienna from September 15-18, 2025, for the ultimate Fabric, Power BI, SQL, and AI community-led learning event. Save €200 with code FABCOMM. Get registered
Hi,
I am working on Data Ingestion project on fabric for first time and need some guidance:
I have following architecture
Domain A
Table1..... Table 10
Domain B
Table 1.....Table 7
with each table having millions of records
I have to ingest data from all these tables from SAP HANA DB as source to fabric datawarehouse.
Should i create one pipeline for each domain containing dataflows for all the tables under that domain or should i create pipeline for each table. The tables are not dependent on each other.
Which approach is best considering the data size?
And if we go with one pipeline with mutiple dataflows should they be arranged sequentially or paraller?
Thank you for your reply.
So if going with parallel execution of dataflows what if one of the dataflow fails? Even if we run the other dataflows, later on can we just run that one failed dataflow?
Hi @ManasiL ,
Yes you can re-run from the failed activity.
Docs to refer -
How to monitor pipeline runs - Microsoft Fabric | Microsoft Learn
Hope this is helpful. Please do let me know incase of further queries.
Hi @ManasiL ,
We haven’t heard from you on the last response and was just checking back to see if your query was answered.
Otherwise, will respond back with the more details and we will try to help .
Thanks
Hi @ManasiL ,
Thanks for using Fabric Community.
For ingesting data from multiple tables with millions of records each, creating one pipeline per domain with parallel dataflows is generally the recommended approach. Here's why:
Benefits of this Approach:
When Might Individual Pipelines or a Sequential Approach can be considered?
At last it is completely depends on your scenario, data volume and your pipeline management, as discussed in previous thread: Solved: Can create multiple data flow gen2 inside one pipe... - Microsoft Fabric Community you can also considered the points mentioned in it.
Hope this is helpful. Please do let me know incase of further queries.
Hi @ManasiL ,
We haven’t heard from you on the last response and was just checking back to see if your query was answered.
Otherwise, will respond back with the more details and we will try to help .
Thanks
This is your chance to engage directly with the engineering team behind Fabric and Power BI. Share your experiences and shape the future.
Check out the June 2025 Fabric update to learn about new features.
User | Count |
---|---|
2 | |
1 | |
1 | |
1 | |
1 |
User | Count |
---|---|
4 | |
3 | |
1 | |
1 | |
1 |