Skip to main content
cancel
Showing results for 
Search instead for 
Did you mean: 

Don't miss out! 2025 Microsoft Fabric Community Conference, March 31 - April 2, Las Vegas, Nevada. Use code MSCUST for a $150 discount. Prices go up February 11th. Register now.

Reply
Anonymous
Not applicable

Incremental data load from CSV

Hi,

 

Is there a way to do incremental loads from csv files? I have csv files delivered weekly, currently being overwritten. 

I want to know

1. Can we store the previously loaded data and not get it overwritten when I do the data load of the new file into Power BI? The file name remains the same.  I will have to add the week, month and year in Power query

2. If we cant do the above, I can get the team to append the week number and year to the file but can we then just load the new data file and not load all the previous ones?

 

Thanks a lot

 

4 REPLIES 4
PaulDBrown
Community Champion
Community Champion

@Anonymous
You might give this method a go, shown by Ruth over at Curbal:
https://youtu.be/9WLZ9jfoXwA




Did I answer your question? Mark my post as a solution!
In doing so, you are also helping me. Thank you!

Proud to be a Super User!
Paul on Linkedin.






Greg_Deckler
Super User
Super User

According to this article, no:

 

https://docs.microsoft.com/en-us/power-bi/service-premium-incremental-refresh

 

Limitations

Currently, for composite models, incremental refresh is supported for SQL Server, Azure SQL Database, SQL Data Warehouse, Oracle, and Teradata data sources only.

 

Sometimes the documentation is out-of-date though.



Follow on LinkedIn
@ me in replies or I'll lose your thread!!!
Instead of a Kudo, please vote for this idea
Become an expert!: Enterprise DNA
External Tools: MSHGQM
YouTube Channel!: Microsoft Hates Greg
Latest book!:
Power BI Cookbook Third Edition (Color)

DAX is easy, CALCULATE makes DAX hard...
Anonymous
Not applicable

Thanks @Greg_Deckler .

 

 

Anonymous
Not applicable

Yes, @Greg_Deckler is correct...

 

BUT, you can use the same M code and create a Dataflow.  Dataflows DO enable incremental refresh on sources like .csv flat files. 

 

When the dataflow refreshes, it will only pick up the newest data, perform the transformations, and load that newly shaped data into the Azure Data Lake.

 

If you do all of your transformations in the dataflow, the Power BI just has to read from the Data Lake to pull data into the model.  That read is usually much faster than reading from .csv files in SharePoint and performing transformations.

 

I'm pretty sure you can also enable incremental refresh for pulling data from the dataflow.

 

Best,

 

~ Chris

Helpful resources

Announcements
Las Vegas 2025

Join us at the Microsoft Fabric Community Conference

March 31 - April 2, 2025, in Las Vegas, Nevada. Use code MSCUST for a $150 discount! Prices go up Feb. 11th.

Jan25PBI_Carousel

Power BI Monthly Update - January 2025

Check out the January 2025 Power BI update to learn about new features in Reporting, Modeling, and Data Connectivity.

Jan NL Carousel

Fabric Community Update - January 2025

Find out what's new and trending in the Fabric community.