Forum Discussion

mp390988's avatar
mp390988
Post Partisan
1 year ago
Solved

Refreshing large CSV

Hello,   A user downloads CSV file from vendor website. The CSV is huge (1.2GB) I am planning to load this file into Power BI. However, user requires daily refresh so this means refreshing every...
Need Help
  • Anonymous's avatar
    Anonymous
    1 year ago

    Hi mp390988 
    Thank you for posting in the Microsoft Fabric Community. Also thankyou Deku  for your inputs.

     

    Since CSV files don’t have a query engine, Power BI’s incremental refresh won’t work as expected. Without an ETL process, one approach is to manually maintain a rolling CSV archive in SharePoint, where a “latest” file (latest_data.csv) contains only new records extracted from the full vendor file. In Power Query, filter and append only this new data to your dataset. If Power BI Dataflows are an option, they can help store historical data while appending only new records daily.

     

    If this post helps, then please give us Kudos and consider Accept it as a solution to help the other members find it more quickly.

     

    Thankyou.

mp390988's avatar
mp390988
Post Partisan
1 year ago

In this situation, I have no option to use ETL as not funded piece of project, more of just helping to build an EUC.

So basically, there is no way for Power BI to only load new records from a CSV extract