incremental load

Rsteinke · ‎06-02-2020

I am pulling 67 million records from an AS400/DB2 flat file. I break it up into 5 tables in my PBI. I would like to use incremental refresh, the question is if doing it by date, do I need to have the same date field in all five tables or will PBI refresh all the tables based on the date in the one table?

Anonymous · ‎06-02-2020

Hi @Rsteinke,

I agree with mwegener's point of view, incremental load features may not fully works on data source that not support query folding. Perhaps you can add a T-SQL query(with query parameters) to your connector to limit the loading data range instead of configuring incremental refresh.

All You Need to Know About the Incremental Refresh in Power BI: Load Changes Only

Regards,

Xiaoxin Sheng

mwegener · ‎06-02-2020

Hi @Rsteinke ,

Flat files do not support query folding.
So all data must always be loaded through Power BI.
No real incremental refresh.

Did I answer your question?
Please mark my post as solution, this will also help others.
Please give Kudos for support.

Marcus Wegener works as Full Stack Power BI Engineer at BI or DIE.
His mission is clear: "Get the most out of data, with Power BI."
twitter - LinkedIn - YouTube - website - podcast - Power BI Tutorials