Forum Discussion
Incremental Refresh with Power BI + Databricks – Query Folding Issue
Best Approach – Databricks.Query() with SQL filter
Use Databricks.Query() instead of Catalogs.
Write SQL like:
SELECT * FROM schema.table
WHERE LastChanged >= timestamp '...RangeStart...'
AND LastChanged < timestamp '...RangeEnd...'
This pushes filters to Databricks → query folding works (in Desktop).⚠️ Dataflows Gen2 may incorrectly flag folding as unsupported (MS limitation).
Delta Tables + Partitioning (if using Lakehouse/ADLS)
Use DeltaLake.Table().
Partition data on the datetime column.
Even without folding, partition pruning makes refresh very fast.
Default Catalog Path
No folding → pulls full dataset each time.
Works but slow for large data.
Best Approach – Databricks.Query() with SQL filter
Use Databricks.Query() instead of Catalogs.
Write SQL like:
SELECT * FROM schema.table
|
This pushes filters to Databricks → query folding works (in Desktop).
⚠️ Dataflows Gen2 may incorrectly flag folding as unsupported (MS limitation).
Delta Tables + Partitioning (if using Lakehouse/ADLS)
Use DeltaLake.Table().
Partition data on the datetime column.
Even without folding, partition pruning makes refresh very fast.
Default Catalog Path
No folding → pulls full dataset each time.
Works but slow for large data.