Forum Discussion

Braxx99's avatar
Braxx99
Frequent Visitor
2 years ago

dataflow incremental refresh slower then full refresh

I have a dataflow with 3 tables. 2 of them have the incremental refresh applied.  

- the dataflow takes about 1:57min to complete with incremental mode

- when incremental refresh is turn off it takes about 48min

That's a problem cos I expected incremental would be faster.

 

Let's take one of these incrementaly refreshed table to examin closely.

- it is configured to refreshed last 45 days, although 48 partitions are created

- the refresh starts at 13:08

- 32 out of 48 partitions are refreshed in parallel starting at 13:08

- rest of te partitions waiting in queue. The last two ones begin refresh at 14:09

 

here is a screenshot from dataflow refresh logs. (not all the columns and partition visible)

 

questions:

1) why not all of the partitions are refreshed in parallel? 

2) what can I do to be able to refresh all in parallel? I guess this is a problem why the incremental refresh takes so long. The partition which is refreshed at 14:09 need 37min to complete. 

3 Replies

    • Braxx99's avatar
      Braxx99
      Frequent Visitor

      Thanks for the link.

      Unfortunatelly, I could not find answer to my questions there. 

      What I am particulary interested in is what are the possible reasons some of the partitions are queued?

       

      The source of the table mentioned earier is SQL Server on-premises. It has about 12mln rows and cover daily data from 2019-01-02