Forum Discussion

yedu's avatar
yedu
Frequent Visitor
11 months ago
Solved

Incremental Refresh with Power BI + Databricks – Query Folding Issue

Hi Community, I’m working on setting up Incremental Refresh in Power BI using Databricks as the source. I’ve created the RangeStart and RangeEnd parameters and applied the filters in Power Query. Th...
Need Help
  • Shahid12523's avatar
    Shahid12523
    11 months ago

    Best Approach – Databricks.Query() with SQL filter

    Use Databricks.Query() instead of Catalogs.

    Write SQL like:

    SELECT * FROM schema.table
    WHERE LastChanged >= timestamp '...RangeStart...'
    AND LastChanged < timestamp '...RangeEnd...'

     


    This pushes filters to Databricks → query folding works (in Desktop).

    ⚠️ Dataflows Gen2 may incorrectly flag folding as unsupported (MS limitation).

     

    Delta Tables + Partitioning (if using Lakehouse/ADLS)

    Use DeltaLake.Table().

    Partition data on the datetime column.

    Even without folding, partition pruning makes refresh very fast.

     

    Default Catalog Path

    No folding → pulls full dataset each time.

    Works but slow for large data.

Anonymous's avatar
Anonymous
Not applicable
11 months ago

Hi yedu,

 

Thank you for reaching out to the Microsoft Fabric Forum Community, and special thanks to Shahid12523  for prompt and helpful response.

Just following up to see if the Response provided by community member were helpful in addressing the issue.

If one of the responses helped resolve your query, please consider marking it as the Accepted Solution. Feel free to reach out if you need any further clarification or assistance.

 

Best regards,
Prasanna Kumar