Power BI is turning 10! Tune in for a special live episode on July 24 with behind-the-scenes stories, product evolution highlights, and a sneak peek at what’s in store for the future.
Save the dateEnhance your career with this limited time 50% discount on Fabric and Power BI exams. Ends August 31st. Request your voucher.
Hi,
I am working on Data Ingestion project on fabric for first time and need some guidance:
I have following architecture
Domain A
Table1..... Table 10
Domain B
Table 1.....Table 7
with each table having millions of records
I have to ingest data from all these tables from SAP HANA DB as source to fabric datawarehouse.
Should i create one pipeline for each domain containing dataflows for all the tables under that domain or should i create pipeline for each table. The tables are not dependent on each other.
Which approach is best considering the data size?
And if we go with one pipeline with mutiple dataflows should they be arranged sequentially or paraller?
Thank you for your reply.
So if going with parallel execution of dataflows what if one of the dataflow fails? Even if we run the other dataflows, later on can we just run that one failed dataflow?
Hi @ManasiL ,
Yes you can re-run from the failed activity.
Docs to refer -
How to monitor pipeline runs - Microsoft Fabric | Microsoft Learn
Hope this is helpful. Please do let me know incase of further queries.
Hi @ManasiL ,
We haven’t heard from you on the last response and was just checking back to see if your query was answered.
Otherwise, will respond back with the more details and we will try to help .
Thanks
Hi @ManasiL ,
Thanks for using Fabric Community.
For ingesting data from multiple tables with millions of records each, creating one pipeline per domain with parallel dataflows is generally the recommended approach. Here's why:
Benefits of this Approach:
When Might Individual Pipelines or a Sequential Approach can be considered?
At last it is completely depends on your scenario, data volume and your pipeline management, as discussed in previous thread: Solved: Can create multiple data flow gen2 inside one pipe... - Microsoft Fabric Community you can also considered the points mentioned in it.
Hope this is helpful. Please do let me know incase of further queries.
Hi @ManasiL ,
We haven’t heard from you on the last response and was just checking back to see if your query was answered.
Otherwise, will respond back with the more details and we will try to help .
Thanks
This is your chance to engage directly with the engineering team behind Fabric and Power BI. Share your experiences and shape the future.