Reply
bilingual
Helper V
Helper V
Partially syndicated - Outbound

Append large data files - best practice?

Hi, i have used Power Query in order to combine historical data in our organization. 

The problem is the only form of export from SAP webi is either excel or csv format. 

As every months has about 125.000 rows and 15 columns, i have made a workaround by creating Excel files per Quarter and then combined them with Append and then created relationships with the Append data to other tables.

However it is still very heavy to use as data in the Append has reached about 8 mio rows and i am continually adding data. 

 

With my limit in the export opportunities can anybody suggest a better solution?

7 REPLIES 7
avatar user
Anonymous
Not applicable

Syndicated - Outbound

If these csv files are all in the same folder, can't you just use the folder connector so that all of your new files are added to the dataset upon refresh?

 

--Nate

ppm1
Solution Sage
Solution Sage

Syndicated - Outbound

You can use incremental refresh with files, if you can parse a date value from the filenames. Please see this video.

https://www.youtube.com/watch?v=IVMdg16yBKE

 

Pat

Microsoft Employee
Maha1
Helper II
Helper II

Syndicated - Outbound

Hi @bilingual , @johannesDE , @v-alq-msft , @PC2790 ,

 

This was the best solution!

https://youtu.be/9WLZ9jfoXwA

johannesDE
Frequent Visitor

Syndicated - Outbound

I am also interested in finding the best practice in a case like this.

 

So far, I have split a large fact table by year, turned off refresh for prior years and appended the data in PowerQuery.

 

I am not sure if this will improve performance.

 

 

v-alq-msft
Community Support
Community Support

Syndicated - Outbound

Hi, @bilingual 

 

I'd like to suggest you try the following data reduction techniques:

 

For further information, you may refer to the following document.

Data reduction techniques for Import modeling 

 

Best Regards

Allan

 

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

PC2790
Community Champion
Community Champion

Syndicated - Outbound

Have you considered using Direct Query and incremental refresh to load the new data keeping the old data intact? 

Syndicated - Outbound

Unfortunately it does not work with csv or Excel files.

avatar user

Helpful resources

Announcements
March PBI video - carousel

Power BI Monthly Update - March 2025

Check out the March 2025 Power BI update to learn about new features.

March2025 Carousel

Fabric Community Update - March 2025

Find out what's new and trending in the Fabric community.

Top Solution Authors (Last Month)
Top Kudoed Authors (Last Month)