Refreshing large CSV

[mp390988]

Hello,

A user downloads CSV file from vendor website.

The CSV is huge (1.2GB)

I am planning to load this file into Power BI.

However, user requires daily refresh so this means refreshing everyday.

The issue is that CSV file always contains historical data. So it is not like if you download the CSV file tomorrow, it will only contain today's data. It actually contains data all the way from inception. 

I am planning to move this CSV file into Sharepoint and connect to it using Sharepoint folder.

But how do I go about only loading new records after the first refresh? I don't think incremental refresh would work in this situation, right?

[v-tsaipranay · Accepted solution]

Hi @mp390988 
Thank you for posting in the Microsoft Fabric Community. Also thankyou @Deku  for your inputs.

Since CSV files don’t have a query engine, Power BI’s incremental refresh won’t work as expected. Without an ETL process, one approach is to manually maintain a rolling CSV archive in SharePoint, where a “latest” file (latest_data.csv) contains only new records extracted from the full vendor file. In Power Query, filter and append only this new data to your dataset. If Power BI Dataflows are an option, they can help store historical data while appending only new records daily.

If this post helps, then please give us Kudos and consider Accept it as a solution to help the other members find it more quickly.

Thankyou.

[v-tsaipranay]

Hi @mp390988 ,

May I ask if you have resolved this issue? If so, please mark the helpful reply and accept it as the solution. This will be helpful for other community members who have similar problems to solve it faster.

Thank you.

[v-tsaipranay]

Hi @mp390988 ,

I wanted to check if you had the opportunity to review the information provided. Please feel free to contact us if you have any further questions. If my response has addressed your query, please accept it as a solution and give a 'Kudos' so other members can easily find it.

Thank you.

[v-tsaipranay]

Hi @mp390988 ,

I wanted to check if you had the opportunity to review the information provided. Please feel free to contact us if you have any further questions. If my response has addressed your query, please accept it as a solution and give a 'Kudos' so other members can easily find it.

Thank you.

[mp390988]

In this situation, I have no option to use ETL as not funded piece of project, more of just helping to build an EUC.

So basically, there is no way for Power BI to only load new records from a CSV extract

[Deku]

Incremental refresh really requires the data source to have a query engine for the date filtering. Applying incremental refresh on a source with no query engine will result in all the data being pulled for each partition, which will be filtered in powerquery. In the second scenario the refresh will be more costly and longer than just pulling the data normally.

Really you want a ETL/ELT process to load the data in the database, warehouse or lake. Then pull the data from there