Forum Discussion
Query folding with incremental refresh in dataflows gen2 using Databricks.Query() connector
Hi mmarcoux ,
Thanks for clarifying, I see what you mean now. You’ve already confirmed query folding is working correctly (green lightning icon and supported syntax), so the issue isn’t with your M code or transformations.
What you’re running into is more about how incremental refresh validation in Dataflows Gen2 checks for foldability. At the moment, that validation logic doesn’t seem to recognize the Databricks connector the same way it does for SQL-based sources (like SQL Server, Synapse, etc.). That’s why you still get the “query is not foldable” error even though folding indicators show it is.
From what’s documented today, incremental refresh in Gen2 officially supports only a limited set of connectors, and Databricks isn’t yet on that list. So this looks less like a mistake on your side and more like either:
- a limitation/unsupported feature (incremental refresh not yet enabled for Databricks in Gen2), or
- a bug in how the validation step is detecting foldability.
If you haven’t already, I’d recommend logging this on the Fabric Ideas - Microsoft Fabric Community or raising a support ticket through the How to create a Fabric and Power BI Support ticket - Power BI | Microsoft Learn. That’s the best way to get confirmation from the product team on whether this is an unsupported scenario or a bug, and to track if/when Databricks query folding support for incremental refresh in Dataflows Gen2 is on the roadmap.
Regards,
Akhil.
Hi! mmarcoux
Did you try checking https://learn.microsoft.com/en-us/power-query/step-folding-indicators.
Generally when we try to setup incremental refresh, we need to ensure that the steps used are supported as a native query and it can work as a SQL query. Also, as you see in the solution of the community thread that you shared should help you understand that how simple the code should be.
Hello AnkitKukreja,
Thanks for the reply,
I have validated that the steps used are supported as a native query, as indicated in the thread you linked:
You can see that from the screenshot in my initial post that this is indeed the case.
The example M code I provided uses the same syntax as the solution thread, and trying variants of the same syntax, I.E. using Value.NativeQuery() wrapped around Databricks.Catalog(), which also shows that the query is indeed folding.
The key point here is not that the query is not folding (by all accounts it is indeed folding), it is that the incremental refresh in dataflows Gen2 gives the error that the query is not foldable, which seems to be either a bug, or an unsupported feature with the connector I am using.
Appreciate the response.
Matt
