Forum Discussion
Refreshing large CSV
Hi mp390988
Thank you for posting in the Microsoft Fabric Community. Also thankyou Deku for your inputs.Since CSV files don’t have a query engine, Power BI’s incremental refresh won’t work as expected. Without an ETL process, one approach is to manually maintain a rolling CSV archive in SharePoint, where a “latest” file (latest_data.csv) contains only new records extracted from the full vendor file. In Power Query, filter and append only this new data to your dataset. If Power BI Dataflows are an option, they can help store historical data while appending only new records daily.
If this post helps, then please give us Kudos and consider Accept it as a solution to help the other members find it more quickly.
Thankyou.
Hi mp390988
Thank you for posting in the Microsoft Fabric Community. Also thankyou Deku for your inputs.
Since CSV files don’t have a query engine, Power BI’s incremental refresh won’t work as expected. Without an ETL process, one approach is to manually maintain a rolling CSV archive in SharePoint, where a “latest” file (latest_data.csv) contains only new records extracted from the full vendor file. In Power Query, filter and append only this new data to your dataset. If Power BI Dataflows are an option, they can help store historical data while appending only new records daily.
If this post helps, then please give us Kudos and consider Accept it as a solution to help the other members find it more quickly.
Thankyou.
Hi mp390988 ,
May I ask if you have resolved this issue? If so, please mark the helpful reply and accept it as the solution. This will be helpful for other community members who have similar problems to solve it faster.
Thank you.