Dataflow Incremental Refresh

[DataSundowner]

Hi, I have a question on dataflow incremental refresh. If I'm pulling the data from SQL Server using a query, does it still run the full SQL query first before implementing incremental refresh policy? Or does it implement incremental refresh policy in the SQL query? For example, if I have a query such as "SELECT * FROM sales WHERE saledate >='2023-01-01'" and a incremental refresh policy such as "store rows from past 1 year, and refresh rows from past 5 days". How would this execute behind the scence? Does it run the query with everything >= 2023-01-01 first and then apply the refresh policy? If that's the case, what the point of incremental refresh? I must be missing something. Any insights would be appreciated. 

[Anonymous · Accepted solution]

Hi @DataSundowner 

According to my understanding, Power BI will load the data that satisfies the SELECT * FROM sales WHERE saledate >='2023-01-01 statement into Power BI. When using incremental refresh, the service will dynamically partition the data and create incremental refresh partitions and historical partitions. In each subsequent refresh, the rows whose date/time is within the refresh cycle will be refreshed. The rows whose date/time is no longer within the refresh cycle will become part of the historical cycle and will not be refreshed again.

Best Regards,
Yulia Xu

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

[DataSundowner]

Hi Yulia - Thank you for the information. That's very helpful to know.