Forum Discussion
Incremental Refresh with Power BI + Databricks – Query Folding Issue
Best Approach – Databricks.Query() with SQL filter
Use Databricks.Query() instead of Catalogs.
Write SQL like:
SELECT * FROM schema.table
WHERE LastChanged >= timestamp '...RangeStart...'
AND LastChanged < timestamp '...RangeEnd...'
This pushes filters to Databricks → query folding works (in Desktop).⚠️ Dataflows Gen2 may incorrectly flag folding as unsupported (MS limitation).
Delta Tables + Partitioning (if using Lakehouse/ADLS)
Use DeltaLake.Table().
Partition data on the datetime column.
Even without folding, partition pruning makes refresh very fast.
Default Catalog Path
No folding → pulls full dataset each time.
Works but slow for large data.
Hi yedu,
Just following up to see if the Response provided was helpful in resolving your issue. Please feel free to let us know if you need any further assistance.
Best regards,
Prasanna Kumar