Query folding using Dataflow as datasource - Is it possible? On Roadmap?

[S_loke]

Similar to the issues reported on https://community.powerbi.com/t5/Desktop/Dataflows-don-t-support-Query-Folding/td-p/673500  Unfortunately, the tread seem to have gone into different direction, proving that query folding is possible on select datasources and how to use it but have not answered the original question of query folding using dataflows as a datasource.

We have implemented dataflows within a P1 capacity that represents a golden dataset.  As such, the size of the dataflow is rather large (6gb).  The intent is for users to apply a filter on the dataflow to retrieve only relevant results.

However, when using the on desktop, query folding doesn't seem possible as the data retrieved initially is the 6gb and the filter condition processed on desktop thereafter.  Obvious implication is the time/resources it takes to download the entire dataflow only to remove the majority when applying the filter.

The immediate questions

1.   Is query folding using dataflows as source data possible (i.e. filter records on dataflow)
2.   If query folding using dataflows is not currently possible, is this on the roadmap (MBAS suggest major performance improvements in dealing with large data volume within dataflows so hoping that query folding capability ships with it!)

[Lunar_sea]

Hi Microsoft,

I'm also looking for an offical update with respect to Query Folding for dataflows.

[samloud]

I'd really like this too. But how coudl it be done? isn't a dataflow just a text file on Azure Data Lake under the covers? I can see how you'd fold back to a relational dB, and (especially if your indexes / partitions were set up nicely in the back end) it would limit the rows at source. But how would you do that against a file on ADLS?

I wonder if they'd do something like a DirectQuery, where you'd target the Dataflow, but it would really fold the query back to the relational store behind the dataflow...

[S_loke]

Hi,

I guess this is where the new engine kicks in.  The whitepaper suggest that when using dataflows, there are instances of SSAS that are deployed in the backend so in theory you could do query folding against that instance.  The question now is how to trigger this because it is a temporary instance.  So far in my testing, i haven't been able to isolate the steps to take advantage of this but hopefully more information will come through to assist in the near future.

[nickyvv]

Matthew Roche has an excellent post about Query Folding and Dataflows:

https://ssbipolar.com/2019/10/12/power-bi-dataflows-and-query-folding/

The summary:

1. When accessing dataflow entities from Power BI Desktop, no query folding takes place, even if the enhanced compute engine is enabled.
2. When accessing dataflow entities from other entities in the Power BI service, no query folding takes place unless the enhanced compute engine enabled.
3. When accessing dataflow entities from other entities in the Power BI service, query folding will take place when the enhanced compute engine is enabled, because the linked entity’s query will be executed against the cached data in SQL, rather than the underlying CDM folder.

[S_loke]

Thanks - the more information that is becoming avaliable, the more it makes sense.

Point 3 suggests that down the line query folding from desktop could be made possible.  PBI already have some examples of similar concepts (query caching for datasets)

fingers crossed that this somehow makes it onto their roadmap!!

[Anonymous · Accepted solution]

HI @S_loke ,

Based on test, I haven't found that option on dataflow side and it means this feature still not available on dataflow.
In addition, I also checked plan list but haven't found it. You can consider to submit an idea to increase priority of this feature.

What’s new and planned for business intelligence

Regards,

Xiaoxin Sheng

[S_loke]

Thank you - not the answer i was hoping for but at least i can now plan for alternative actions

[nickyvv]

Is there an official update on Query Folding for dataflows?