Skip to main content
cancel
Showing results for 
Search instead for 
Did you mean: 

Find everything you need to get certified on Fabric—skills challenges, live sessions, exam prep, role guidance, and more. Get started

Reply
Braxx99
Frequent Visitor

dataflow incremental refresh slower then full refresh

I have a dataflow with 3 tables. 2 of them have the incremental refresh applied.  

- the dataflow takes about 1:57min to complete with incremental mode

- when incremental refresh is turn off it takes about 48min

That's a problem cos I expected incremental would be faster.

 

Let's take one of these incrementaly refreshed table to examin closely.

- it is configured to refreshed last 45 days, although 48 partitions are created

- the refresh starts at 13:08

- 32 out of 48 partitions are refreshed in parallel starting at 13:08

- rest of te partitions waiting in queue. The last two ones begin refresh at 14:09

 

here is a screenshot from dataflow refresh logs. (not all the columns and partition visible)

Braxx99_0-1707302223340.png

 

questions:

1) why not all of the partitions are refreshed in parallel? 

2) what can I do to be able to refresh all in parallel? I guess this is a problem why the incremental refresh takes so long. The partition which is refreshed at 14:09 need 37min to complete. 

3 REPLIES 3
Braxx99
Frequent Visitor

@v-kongfanf-msft any updates? Thanks in advance

v-kongfanf-msft
Community Support
Community Support

Hi @Braxx99 ,

 

Detailed documentation on incremental refresh and real-time data troubleshooting is officially available. Refer to below document:

Troubleshoot incremental refresh and real-time data in Power BI - Power BI | Microsoft Learn

 

Best Regards,
Adamk Kong

 

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

Thanks for the link.

Unfortunatelly, I could not find answer to my questions there. 

What I am particulary interested in is what are the possible reasons some of the partitions are queued?

 

The source of the table mentioned earier is SQL Server on-premises. It has about 12mln rows and cover daily data from 2019-01-02

Helpful resources

Announcements
July 2024 Power BI Update

Power BI Monthly Update - July 2024

Check out the July 2024 Power BI update to learn about new features.

July Newsletter

Fabric Community Update - July 2024

Find out what's new and trending in the Fabric Community.