Forum Discussion
Incremental taking longer time than normal refresh
Hi yogita_2025
The Power BI report is connected to Databricks, and incremental refresh has been configured using native SQL queries with RangeStart and RangeEnd parameters. However, during the refresh process, it is observed that the incremental refresh is taking significantly longer than a full refresh, which defeats the purpose of implementing incremental loading.The performance issue arises because Databricks views do not support native query folding, and manually constructing SQL queries in Power BI breaks folding as well. As a result, Power BI cannot push partition filters directly to the source, causing the entire dataset to load first before filters are applied. This leads to inefficient refresh behavior and significantly increased processing time.
To enable effective incremental refresh, it is recommended to implement a parameterized table function in Databricks that accepts date filters (startDate, endDate) as arguments. This approach allows Power BI to maintain query folding and push filters directly at the source, enabling efficient partition pruning and faster refresh performance. Alternatively, if Microsoft Fabric is being used, leveraging Direct Lake mode or composite models with pre-partitioned data can further optimize refresh strategies and report performance.
If this post helps, kindly mark it as Accepted Solution.
Thank You!
Hi, Thanks for the clarification. I have some more issues
Power Bi report is connected with Databricks.
Databricks does not support view native query which does not ensure query folding.
If I setup SQL query myself in advanced editor that is also not working as required as it is first loading the whole data and then applying incremental which leads to increased refresh time.
Can I apply Incremental refresh on Power BI in this case.
Hi yogita_2025
The Power BI report is connected to Databricks, and incremental refresh has been configured using native SQL queries with RangeStart and RangeEnd parameters. However, during the refresh process, it is observed that the incremental refresh is taking significantly longer than a full refresh, which defeats the purpose of implementing incremental loading.
The performance issue arises because Databricks views do not support native query folding, and manually constructing SQL queries in Power BI breaks folding as well. As a result, Power BI cannot push partition filters directly to the source, causing the entire dataset to load first before filters are applied. This leads to inefficient refresh behavior and significantly increased processing time.
To enable effective incremental refresh, it is recommended to implement a parameterized table function in Databricks that accepts date filters (startDate, endDate) as arguments. This approach allows Power BI to maintain query folding and push filters directly at the source, enabling efficient partition pruning and faster refresh performance. Alternatively, if Microsoft Fabric is being used, leveraging Direct Lake mode or composite models with pre-partitioned data can further optimize refresh strategies and report performance.
If this post helps, kindly mark it as Accepted Solution.
Thank You!