Forum Discussion
Appending Data
Is there a way to Refresh the data in power query by merely appending new data and not refreshing the old data. If all the files are stored in text files what is a simple solution to avoid refreshing growing portions of historical data that will not change. Any best practices for setting an efficient data refresh? Thank you
Power BI has no memory. Either absorb the cost of having to ingest all the files (relatively fast with CSV) or consider moving to Fabric where you can use notebooks or Dataflows Gen2 to get something similar to what you want to achieve.
And if you are thinking about incremental refresh - that would technically work but will give you no performance advantage with import mode data sources.
Hey Arturo24
To efficiently refresh data in Power Query without reloading historical data, use incremental refresh, separating queries for historical and new data. Storing files in distinct folders and utilizing query folding minimizes processing time. Optimizing file handling, like partitioning by date, ensures streamlined updates. These methods improve refresh efficiency while maintaining accuracy.
Best Regards
Govind Sapkade ( Microsoft Certified Data Analyst , PL 300 Certified , MS Fabric Enthusiast)
Let's Connect
Linkdin - www.linkedin.com/in/govind-sapkade-845104225
Youtube - http://www.youtube.com/@govind_dataanalyst
6 Replies
Power BI has no memory. Either absorb the cost of having to ingest all the files (relatively fast with CSV) or consider moving to Fabric where you can use notebooks or Dataflows Gen2 to get something similar to what you want to achieve.
And if you are thinking about incremental refresh - that would technically work but will give you no performance advantage with import mode data sources.
Hey Arturo24
To efficiently refresh data in Power Query without reloading historical data, use incremental refresh, separating queries for historical and new data. Storing files in distinct folders and utilizing query folding minimizes processing time. Optimizing file handling, like partitioning by date, ensures streamlined updates. These methods improve refresh efficiency while maintaining accuracy.
Best Regards
Govind Sapkade ( Microsoft Certified Data Analyst , PL 300 Certified , MS Fabric Enthusiast)
Let's Connect
Linkdin - www.linkedin.com/in/govind-sapkade-845104225
Youtube - http://www.youtube.com/@govind_dataanalystDo you have a link to a simple example, some type of tutorial?
Thank you
Hi Arturo24,
we would like to follow up to see if the solution provided by the super user resolved your issue. Please let us know if you need any further assistance.
If our super user response resolved your issue, please mark it as "Accept as solution" and click "Yes" if you found it helpful.Regards,
Vinay PabbuHi @Arturo24,
we would like to follow up to see if the solution provided by the super user resolved your issue. Please let us know if you need any further assistance.
If our super user response resolved your issue, please mark it as "Accept as solution" and click "Yes" if you found it helpful.Regards,
Vinay PabbuHi @Arturo24,
we would like to follow up to see if the solution provided by the super user resolved your issue. Please let us know if you need any further assistance.
If our super user response resolved your issue, please mark it as "Accept as solution" and click "Yes" if you found it helpful.Regards,
Vinay Pabbu
