Update backfill in chunks for incremental refresh

[tcloran]

I'm using a large table as a dataset in Power BI. I applied an incremental refresh to this dataset, but after publishing with the initial scope defined by the RangeStart and RangeEnd, it requires a one-time back fill of archived data. I am running into connection and timeout issues with this back fill. Is there a way to run this back fill in smaller chunks? 

[lbendlin]

Use the XMLA endpoint with tools like SSMS and then refresh individual partitions selectively.

Advanced incremental refresh and real-time data with the XMLA endpoint in Power BI - Power BI | Micr...

[tcloran]

This is a premium per user workspace but we do not have premium capacity. From what I've read, XMLA requires premium capacity, is this true? If so, is there a way to partition the data load outside of premium capacity?

[lbendlin · Accepted solution]

Use CSV or Parquet files as fake partitions and then append them in Power Query.   Yes, you will reload all the data each time but both CSV and Parquet are ingesting very fast.

[tcloran]

Sorry if this is a novice question but how would I create separate csv files out of a sql table/view? This needs to be an automated, scheduled refresh. 

[lbendlin]

There is another option - using bootstrapping to create empty partitions and then fill them individually via XMLA.

https://learn.microsoft.com/en-us/power-bi/connect-data/incremental-refresh-xmla

Troubleshoot incremental refresh and real-time data - Power BI | Microsoft Learn