Forum Discussion
Dataverse query performance question
- 10 months ago
Hi mbowler
The performance difference between the two Power Query approaches stems from how query folding and data retrieval are handled when connecting to Dataverse (Dynamics). In method A, Power Query uses the default connector path — CommonDataService.Database("mycomp.dynamics.com") — which retrieves metadata and applies transformations through the OData API layer. This process results in Power Query downloading the full dataset into memory before applying filters or transformations locally, leading to very slow performance, especially for large entities. In contrast, method B uses Value.NativeQuery, which sends a direct SQL-like query (SELECT * FROM my_entity) to the Dataverse endpoint via the TDS (Tabular Data Stream) interface. This means the filtering and data shaping happen server-side, and only the processed results are sent back to Power Query, allowing for significant speed improvements — often seconds instead of minutes. Essentially, method B leverages query folding and server execution, while method A relies on a less efficient OData fetch. The default behavior of Power Query when choosing “Dataverse” as a source still uses the OData path, which prioritizes compatibility and schema safety but sacrifices performance. For large datasets or production refreshes, using Value.NativeQuery with the TDS endpoint is therefore the preferred approach.
Interesting topic! Regarding why Query A is extremely slow, if you look at this article from Microsoft (it also explained Query B)
https://learn.microsoft.com/en-us/power-bi/guidance/powerbi-modeling-guidance-for-power-platform
"If you're using the Dataverse connector (formerly known as the Common Data Service), you can add the CreateNavigationProperties=false option to speed up the evaluation stage of a data import.
The evaluation stage of a data import iterates through the metadata of its source to determine all possible table relationships. That metadata can be extensive, especially for Dataverse. By adding this option to the query, you're letting Power Query know that you don't intend to use those relationships. The option allows Power BI Desktop to skip that stage of the refresh and move on to retrieving the data."
So Each [Data] access (that Source{[Schema="dbo",Item="my_entity"]}[Data]) doesn’t just send a single SQL query.
It first makes multiple metadata calls (schema, relationship, permissions, type info) to Dataverse. Then it requests the data in paged batches (often thousands records per call). These metadata and pagination calls happen sequentially and latency adds up.