Query folding with incremental refresh in dataflows gen2 using Databricks.Query() connector

[mmarcoux]

Hello All,

We are testing incremental refresh in Dataflows Gen2 using the Databricks connector and are running into a query folding validation error. The validation fails with the message that the query is not folding, even though in the Power Query editor the applied steps show the green folding indicator.

The query is written using the Databricks.Query() syntax, which has been suggested as the correct approach for enabling folding with Databricks catalogs. When attempting to validate incremental refresh, Dataflows Gen2 does not appear to recognize the query as foldable.

The query destination is a fabric warehouse.

Documentation on native query folding indicates that Databricks is not currently listed among the supported sources, and the use of Value.NativeQuery() for folding is also not supported with Databricks. However, there is no mention of the Databricks.Query() connector in the folding documentation.

Based on this, it seems incremental refresh with query folding is not currently supported for Databricks in Dataflows Gen2. Can Microsoft confirm whether this is the case, and if support for Databricks query folding with incremental refresh is on the roadmap?

Example M Code using Databricks.Query():

let
  Result = Databricks.Query(
    "your-databricks-workspace-url",
    "/sql/1.0/warehouses/your-warehouse-id",
    [Catalog = null, Database = null, EnableAutomaticProxyDiscovery = "enabled"]
  )(
    "SELECT * FROM your_catalog.your_schema.your_table"
  )
in
  Result

References:

• Power Query native database query
• Native query folding
• Community thread on Databricks connector and query folding

[v-agajavelly · Accepted solution]

Hi @mmarcoux ,

Thanks for clarifying, I see what you mean now. You’ve already confirmed query folding is working correctly (green lightning icon and supported syntax), so the issue isn’t with your M code or transformations.

What you’re running into is more about how incremental refresh validation in Dataflows Gen2 checks for foldability. At the moment, that validation logic doesn’t seem to recognize the Databricks connector the same way it does for SQL-based sources (like SQL Server, Synapse, etc.). That’s why you still get the “query is not foldable” error even though folding indicators show it is.

From what’s documented today, incremental refresh in Gen2 officially supports only a limited set of connectors, and Databricks isn’t yet on that list. So this looks less like a mistake on your side and more like either:

• a limitation/unsupported feature (incremental refresh not yet enabled for Databricks in Gen2), or
• a bug in how the validation step is detecting foldability.

If you haven’t already, I’d recommend logging this on the Fabric Ideas - Microsoft Fabric Community or raising a support ticket through the How to create a Fabric and Power BI Support ticket - Power BI | Microsoft Learn. That’s the best way to get confirmation from the product team on whether this is an unsupported scenario or a bug, and to track if/when Databricks query folding support for incremental refresh in Dataflows Gen2 is on the roadmap.

Regards,
Akhil.

[AnkitKukreja]

Hi! @mmarcoux 

Did you try checking https://learn.microsoft.com/en-us/power-query/step-folding-indicators. 
Generally when we try to setup incremental refresh, we need to ensure that the steps used are supported as a native query and it can work as a SQL query. Also, as you see in the solution of the community thread that you shared should help you understand that how simple the code should be.

[mmarcoux]

Hello @AnkitKukreja,

Thanks for the reply,

I have validated that the steps used are supported as a native query, as indicated in the thread you linked:

You can see that from the screenshot in my initial post that this is indeed the case.

The example M code I provided uses the same syntax as the solution thread, and trying variants of the same syntax, I.E. using Value.NativeQuery() wrapped around Databricks.Catalog(), which also shows that the query is indeed folding.

The key point here is not that the query is not folding (by all accounts it is indeed folding), it is that the incremental refresh in dataflows Gen2 gives the error that the query is not foldable, which seems to be either a bug, or an unsupported feature with the connector I am using.

Appreciate the response.

Matt