Forum Discussion

Anonymous's avatar
Anonymous
Not applicable
6 years ago

Incremental data load from CSV

Hi,

 

Is there a way to do incremental loads from csv files? I have csv files delivered weekly, currently being overwritten. 

I want to know

1. Can we store the previously loaded data and not get it overwritten when I do the data load of the new file into Power BI? The file name remains the same.  I will have to add the week, month and year in Power query

2. If we cant do the above, I can get the team to append the week number and year to the file but can we then just load the new data file and not load all the previous ones?

 

Thanks a lot

 

Need Help

4 Replies

      • Anonymous's avatar
        Anonymous
        Not applicable
        6 years ago

        Yes, Greg_Deckler is correct...

         

        BUT, you can use the same M code and create a Dataflow.  Dataflows DO enable incremental refresh on sources like .csv flat files. 

         

        When the dataflow refreshes, it will only pick up the newest data, perform the transformations, and load that newly shaped data into the Azure Data Lake.

         

        If you do all of your transformations in the dataflow, the Power BI just has to read from the Data Lake to pull data into the model.  That read is usually much faster than reading from .csv files in SharePoint and performing transformations.

         

        I'm pretty sure you can also enable incremental refresh for pulling data from the dataflow.

         

        Best,

         

        ~ Chris