Forum Discussion
Incremental refresh - Table keeps loading in PBI desktop
Hello,
I defined new parameters on a date/time column in one of the tables (RangeStart and RangeEnd), and filtered the column to a custom date range, but when loading into PBI desktop the table keeps loading forever, even I filtered the date range to 1-day only, but again it keeps loading.
And even when I want to cancel the loading it stuck and I have to force close PBI through task manager.
I think query is folding (because the view native query is active and not grayed out after defining parameters and filtering date range).
This is only happening in our largest table (with 10 Million records) and other tables are loaded fine.
13 Replies
Every time you make structural changes Power BI Desktop throws away all meta data and recollects it. Set your RangeStart and RangeEnd to two adjacent dates, then let Power Query do its thing and do not interfere. Messing with it will make it worse.
Do not attempt anything fancy in the Power Query code - no merges or sorting!
amir_mm when you say 10M rows, that probably means it is coming from some sort database. What if you write native db queries instead of fetching the data from db using PQ ? In my case, I regularly query large data tables but always use native db query (e.g. SQL). I find no advantage in trying to oblige query folding; it is rather a performance killer, Use the SQL and bake IR on top of that. It is a killer combo.
Aren't these a little too small for the effort you put into Incremental Refresh? How long does a regular load take?
After I defined the RangeStart and RangeEnd on both tables, it takes 2-3 min to load.
The main issue is auto refreshing, as we face memory capacity limit (we are on premium A2), so I thought maybe I can do some partitioning and refresh each partition separately.
fair enough. Usually you start thinking of incremental refresh when you are north of 200 Million rows, or getting close to the 5 hr hard limit.
