Forum Discussion
Refreshing large CSV
Hi mp390988
Thank you for posting in the Microsoft Fabric Community. Also thankyou Deku for your inputs.Since CSV files don’t have a query engine, Power BI’s incremental refresh won’t work as expected. Without an ETL process, one approach is to manually maintain a rolling CSV archive in SharePoint, where a “latest” file (latest_data.csv) contains only new records extracted from the full vendor file. In Power Query, filter and append only this new data to your dataset. If Power BI Dataflows are an option, they can help store historical data while appending only new records daily.
If this post helps, then please give us Kudos and consider Accept it as a solution to help the other members find it more quickly.
Thankyou.
Hi mp390988
Thank you for posting in the Microsoft Fabric Community. Also thankyou Deku for your inputs.
Since CSV files don’t have a query engine, Power BI’s incremental refresh won’t work as expected. Without an ETL process, one approach is to manually maintain a rolling CSV archive in SharePoint, where a “latest” file (latest_data.csv) contains only new records extracted from the full vendor file. In Power Query, filter and append only this new data to your dataset. If Power BI Dataflows are an option, they can help store historical data while appending only new records daily.
If this post helps, then please give us Kudos and consider Accept it as a solution to help the other members find it more quickly.
Thankyou.
Hi mp390988 ,
I wanted to check if you had the opportunity to review the information provided. Please feel free to contact us if you have any further questions. If my response has addressed your query, please accept it as a solution and give a 'Kudos' so other members can easily find it.
Thank you.