Forum Discussion
Dataflow Gen2 execution of chained queries and caching
- 2 years ago
It depends. For the staged queries, once the query is staged the subsequent queries will simply reference the staged data as mentioned in the articles shared.
For other queries, it relies on how they're being evaluated as well as the privacy / firewall partitions set by the privacy levels. The best articles that I can share to answer the question on how things are being evaluated by the Power Query engine are the ones below:
- Understanding query evaluation and query folding in Power Query - Power Query | Microsoft Learn
- Query folding examples in Power Query - Power Query | Microsoft Learn
- Query folding indicators in Power Query - Power Query | Microsoft Learn
- Query plan - Power Query | Microsoft Learn
Particularly the query plan is the one that will help you the most to understand how your query will be evaluated. It could be that you have 10 queries, but all of them result into a single native query to your data source due to the query folding of the data source and the connector taking place, so looking at each of them separately would not be the best way to look at things. Its because of this that the answer really is "it depends" and I hope that the articles mentioned above help you understand better how your queries are being evaluated.
If you want to force a query to be evaluated once and subsequent queries to simply reference it and take advantage of the query that was computed, the most explicit way to do so is to take advantage of the "staging" mechanism described in previous articles shared. There are other ways to do so, but the one recommended by the Product Group is to leverage the Staging mechanism.
Hi tdenbow
Thanks for using Fabric Community.
In order to understand how does queries works in Dataflow Gen2 during staging is enabled/disabled.
You can refer to this documents that might help you. Link1, Link2.
I hope this information is helpful. Please do let us know if you have any queries.