Incremental Refresh with Power BI + Databricks – Query Folding Issue

[yedu]

Hi Community,

I’m working on setting up Incremental Refresh in Power BI using Databricks as the source. I’ve created the RangeStart and RangeEnd parameters and applied the filters in Power Query. The refresh policy itself applies, but I’m running into issues:

• Using the default connection path (Databricks.Catalogs → Database → Schema → Table). the refresh works but query folding does not occur. This means the refresh acts like a full refresh each time, with no performance benefit.

• I also tried the Databricks.Query approach, embedding RangeStart and RangeEnd directly in the SQL. In my report this also doesn’t fold down to Databricks, so again it behaves like a full refresh.

• Using Value.NativeQuery is not supported with Databricks, so that path is blocked too.

At this point, Incremental Refresh doesn’t seem to work end-to-end in my setup.

Has anyone here successfully implemented Incremental Refresh with Databricks and confirmed that folding actually works?
If yes, could you please share the exact approach (connection type, query pattern, or even a working example)?

I’d like to document the right way for my team before closing out this effort. Any help or confirmation would be hugely appreciated.

Thanks!

I want to document the right steps for my team, so any guidance or working examples would really help.

Thanks in advance!

[v-pgoloju]

Hi @yedu,

Thank you for reaching out to the Microsoft Fabric Forum Community, and special thanks to @Shahid12523  for prompt and helpful response.

Just following up to see if the Response provided by community member were helpful in addressing the issue.

If one of the responses helped resolve your query, please consider marking it as the Accepted Solution. Feel free to reach out if you need any further clarification or assistance.

Best regards,
Prasanna Kumar

[Shahid12523]

Best Approach – Databricks.Query() with SQL filter

Use Databricks.Query() instead of Catalogs.

Write SQL like:

SELECT * FROM schema.table
WHERE LastChanged >= timestamp '...RangeStart...'
AND LastChanged < timestamp '...RangeEnd...'

This pushes filters to Databricks → query folding works (in Desktop).

⚠️ Dataflows Gen2 may incorrectly flag folding as unsupported (MS limitation).

Delta Tables + Partitioning (if using Lakehouse/ADLS)

Use DeltaLake.Table().

Partition data on the datetime column.

Even without folding, partition pruning makes refresh very fast.

Default Catalog Path

No folding → pulls full dataset each time.

Works but slow for large data.