Forum Discussion
Filtering data (Date/Time) taking forever from a large file.
- 3 years ago
Hi Darger ,
It sounds like you should be doing your initial filtering within the connection string, similar to how you would manage an API source, in order to implement the filters server-side. You may need to check your source documentation to find the correct parameter names, but you'd create a connection string something like this:
https://analytics.dev.azure.com/{organization}/{project}/_odata/v3.0-preview/WorkItems? $select=WorkItemId,Title,WorkItemType,State,CreatedDate &$filter=startswith(Area/AreaPath,'{area path}') &$orderby=CreatedDate desc &$top=10MS Learn source:
https://learn.microsoft.com/en-us/azure/devops/report/powerbi/odataquery-connect?view=azure-devops
Additionally, a lot of ODATA feeds actually support query folding, even though PQ may not display the 'View Native Query' option as you might expect. As such, you should check carefully that you haven't broken folding in a step previous to your filter step causing PQ to implement the filter locally.
MS Learn source:
https://learn.microsoft.com/en-us/power-query/power-query-folding
Pete
Hi Darger ,
It sounds like you should be doing your initial filtering within the connection string, similar to how you would manage an API source, in order to implement the filters server-side. You may need to check your source documentation to find the correct parameter names, but you'd create a connection string something like this:
https://analytics.dev.azure.com/{organization}/{project}/_odata/v3.0-preview/WorkItems?
$select=WorkItemId,Title,WorkItemType,State,CreatedDate
&$filter=startswith(Area/AreaPath,'{area path}')
&$orderby=CreatedDate desc
&$top=10
MS Learn source:
https://learn.microsoft.com/en-us/azure/devops/report/powerbi/odataquery-connect?view=azure-devops
Additionally, a lot of ODATA feeds actually support query folding, even though PQ may not display the 'View Native Query' option as you might expect. As such, you should check carefully that you haven't broken folding in a step previous to your filter step causing PQ to implement the filter locally.
MS Learn source:
https://learn.microsoft.com/en-us/power-query/power-query-folding
Pete