Dataflow - timeout/performance issue while merging/appending 2 linked tables

[deermajk]

I have dimension data soure GBQ with 70m rows. I want to load it as is to PBI dataflow.

I have to use odbc driver. Due to import performace I am trying to test approach with 2 dataflow imports & 1 dataflow merge:

1. full load periodically - weekly

(enhanced engine on)

2. last 10 days updated - daily

(enhanced engine on)

3. merging both together in 3rd dataflow

In the 3rd dataflow I take 2 linked tables. I am using linked tables with disabled data load. 

In 3rd query:

- first I merge 2 linked tables, 1. full load table ref. excluding delta records from 2nd linked table.

- then append 2. table to previous step

- last i remove table column from merge, as i do not need it

The issue is that it takes forever to run, each time (after 1-4h) failing with below error and its really week performance.

Error: PipelineException: Internal error DirectQueryUnexpectedEndOfStream. . RootActivityId = 739a9240-a082-4ba1-98cc-25503e91ad5a.Param1 = PipelineException: Internal error DirectQueryUnexpectedEndOfStream.  Request ID: 93044d33-5676-41bf-bb23-f79937f0800b.

Please help.

[lbendlin]

Avoid doing merges.  Ingest both dataflows as is and then let the data model (or DAX) do the work for you.