Forum Discussion
Incremental refresh for custom connector with call to REST API: RangeStart and RangeEnd
Finally we found the solution:
It is all up to query folding. The article helped a lot:
https://bengribaudo.com/blog/2022/01/20/6500/power-query-m-primer-part-23-query-folding-i
One does not need to define RangeStart nor RangeEnd parameters anywhere in the connector.
Instead one should implement Table.View as shown here:
https://learn.microsoft.com/en-us/power-query/samples/trippin/10-tableview1/readmeThe only missing information is that one need to implement OnSelectRows.
Further information:
https://community.fabric.microsoft.com/t5/Power-Query/Query-folding-Table-View-handler-function-to-filter-rows-add/td-p/3154149br, Michael
We want to reduce the amount of data to be transfered. We configured to load factData for the last 5 years into the archive partition, so only once. The last 3 month should load on every refresh to get the latest data. So we applied incremental refresh on a timestamp-columd.
Will you have separate semantic models for each of the customers?
yes! But we are talking about a custom connector. From other posts I learned, that custom connectors have the ability for incremental refresh. Unfortunately I can not find any samples online.
The way incremental refresh works is as follows:
You (the developer) create a Power Query script that access your data source (including via a custom connector). The Power Query code contains a filter for RangeStart (inclusive) and RangeEnd (exclusive). That filter can be in the standard M code or it can be a part of the constructed but visible URL of the custom connector.
The Power BI service is managing the partitions and will inject the appropriate values for RangeStart and RangeEnd into the query for each partition.
If the Power BI service cannot set these parameters in a way that allows your custom connector to consume them then this won't work.
Ok, can you give an example for that filter within an custom connector?