Forum Discussion
Incremental refresh from API as zip file possible?
- 7 months ago
Hi Bokazoit,
I hope the information provided above assists you in resolving the issue. If you have any additional questions or concerns, please do not hesitate to contact us. We are here to support you and will be happy to help with any further assistance you may need.
Regards,
Community Support Team.
This is possible if your data source you are pulling zips from supports date filtering. See https://learn.microsoft.com/en-us/power-bi/connect-data/incremental-refresh-overview
Here is I think the most pertinent excerpt (my emphasis added):
Incremental refresh and real-time data works best for structured, relational data sources like SQL Database and Azure Synapse, but can also work for other data sources. In any case, your data source must support the following:
Date filtering: The data source must support some mechanism to filter data by date. [irrelevant info re: relational sources]
For other data sources, the RangeStart and RangeEnd parameters must be passed to the data source in some way that enables filtering. For file-based data sources where files and folders are organized by date, the RangeStart and RangeEnd parameters can be used to filter the files and folders to select which files to load. For web-based data sources, the RangeStart and RangeEnd parameters can be integrated into the HTTP request.
This is sometimes easier said than done. For example, in SharePoint, you would do something like:
- Query your document library using SharePoint REST with Web.Contents, using RangeStart and RangeEnd in $filter (converted to correct text format of yyyy-MM-ddTHH:mm:ssZ)
- For each file in range, use SharePoint REST with Web.Contents again to get file/$value to actually download the binaries (zips) + unzip + parse binary contents + transform to homogeneous tabular shape
- Expand your table column, ensuring you retain the date column you originally filtered as part of your query in #1
- Re-apply the RangeStart and RangeEnd filters (which is redundant, but required for PBI to validate the incremental refresh - at least that was needed when I did something similar an admittedly pretty long while ago)
Note: ootb SP connectors do not support date filtering last I checked, which is why we would query REST endpoint with Web.Contents. Relevant MS doc for SP REST: https://learn.microsoft.com/en-us/sharepoint/dev/sp-add-ins/working-with-folders-and-files-with-rest