Forum Discussion
Dataflow - timeout/performance issue while merging/appending 2 linked tables
I have dimension data soure GBQ with 70m rows. I want to load it as is to PBI dataflow.
I have to use odbc driver. Due to import performace I am trying to test approach with 2 dataflow imports & 1 dataflow merge:
1. full load periodically - weekly
(enhanced engine on)
2. last 10 days updated - daily
(enhanced engine on)
3. merging both together in 3rd dataflow
In the 3rd dataflow I take 2 linked tables. I am using linked tables with disabled data load.
In 3rd query:
- first I merge 2 linked tables, 1. full load table ref. excluding delta records from 2nd linked table.
- then append 2. table to previous step
- last i remove table column from merge, as i do not need it
The issue is that it takes forever to run, each time (after 1-4h) failing with below error and its really week performance.
Error: PipelineException: Internal error DirectQueryUnexpectedEndOfStream. . RootActivityId = 739a9240-a082-4ba1-98cc-25503e91ad5a.Param1 = PipelineException: Internal error DirectQueryUnexpectedEndOfStream. Request ID: 93044d33-5676-41bf-bb23-f79937f0800b.
Please help.
1 Reply
- lbendlinSuper User
Avoid doing merges. Ingest both dataflows as is and then let the data model (or DAX) do the work for you.