Improving data refresh efficiency

[AKNTPC]

Refreshing bulk data in Power BI is tedious, since every refresh request is re-uploading GBs when there is no change in the date/time of files uploaded. Power BI has to first verify the modified data/time of files being requested for upload, and upload ONLY those files which are either new, or have modified name/date/time file attributes as compared to the data already published. If the published data does not retain this basic information on the files uploaded earlier, provision is to be created for facilitating this improvement.

This will significantly improve work productivity, and eliminate unnecessary bandwidth usage.

[v-pnaroju-msft]

Hi AKNTPC,

Just following up to confirm whether the details we shared addressed your problem. Should you need additional assistance, please feel free to connect with the Microsoft Fabric community.

Thank you.

[v-pnaroju-msft]

Hi AKNTPC,

We would like to follow up and see whether the details we shared have resolved your problem.
If you need any more assistance, please feel free to connect with the Microsoft Fabric community.

Thank you.

[lbendlin]

You can use incremental refresh (or create your own manual equivalent), with change detection if you feel adventurous. None of this is magic - you still need to store the canary data, ideally only the high water mark per partition, but worst case (and that's how it is implemented by default) by keeping a second copy of each partition.

[v-pnaroju-msft]

Hi AKNTPC,

Thank you for contacting us and sharing your important feedback about the data refresh process in Power BI. You have mentioned a good point about saving bandwidth and using incremental refresh for large datasets. Many users face this problem when working with large amounts of data. Uploading the same unmodified data every time during refresh affects performance, bandwidth, and productivity. Power BI has built-in features like Incremental Refresh and Query Folding that help handle this efficiently.

A full refresh happens when:
a. Incremental Refresh is not set up.
b. The source system does not support checking information like Last Modified or Created Date.
c. Power BI cannot push filters back to the data source, so it reloads all records.
d. The dataset uses sources like folders, SharePoint, or blob storage where file metadata is not used properly unless modeled clearly.

To reduce unnecessary data refresh and improve performance, please follow these steps:

1. Include file information like Modified Date and Created Date in your queries when using SharePoint or folders.
2. Turn on Incremental Refresh on large tables by making RangeStart and RangeEnd parameters.
3. Make sure Query Folding is working so filters are applied at the source.
4. Use Dataflows with Incremental Refresh and store data in OneLake or Azure Data Lake for better scalability.

Also, you can find helpful information in these links:
Incremental refresh for semantic models in Power BI - Power BI | Microsoft Learn
Query folding guidance in Power BI Desktop - Power BI | Microsoft Learn
Configure incremental refresh for Power BI semantic models - Power BI | Microsoft Learn
Combine files overview - Power Query | Microsoft Learn

We hope this information helps you fix the issue. If you have more questions, please feel free to reach out to the Microsoft Fabric community.

Thank you.