NEED HELP - add content of updated csv to existing csv datasource

[Anonymous]

Hi. I searched extensively for my question and I think it may be that I am unable to articulate it accurately in a search.

My setup:  I have a PowerBI report set up to pull from a csv file as my backend table. Lets say its a 3,000 row report from a particular reporting system that runs weekly. The following week a new csv file comes out and its 3,500 rows in a csv. What is the easiest way to bring in this new 3,500 row report into my PowerBI report? Assume that I have an exhaustive array of powerquery modifications built within pBI. 

Is my only option Ive found is to append the new 3,500 row report to the end of the existing 3,000 row report to retain all of my pBI internal work? in a couple months this file would get super huge super quick.

[I do know that I can build out pBI with a file called Master.csv with the 3,000 rows and then replace it with the 3,500 row csv using the same filename as Master.csv.. and then the content is just replaced... but that does not allow for the weekly trends that i want. So that option is not acceptable. ]

Any assistance would be grately appreciated.

[Anonymous]

Hi @Anonymous ,

You may refer this blog and video.

https://sqlitybi.com/loading-multiple-excel-files-from-a-folder-in-power-bi/ 

https://www.youtube.com/watch?v=nPlrQUbEn4o 

Best Regards,

Jay

[parry2k]

@Anonymous Power BI is designed to work with large datasets, it is you have to decide the data retention. For the actual solution, you should put files in a folder, read files from the folder (grab all files), and transform the content of each file based on the transformation logic you have. You can put your transformation in a function in PQ and call that function for each file so transformation will be in one function and you will get a single view of your data from all the files.

As you add and remove files from the folder, Power BI will table will get updated as soon as you refresh the data. There are tons of blog posts on how to read files from folders. From here I think it will be easier for you to get going.

✨ Follow us on LinkedIn and  to our YouTube channel

Learn about conditional formatting at Microsoft Reactor

My latest blog post The Power of Using Calculation Groups with Inactive Relationships (Part 1) (perytus.com) I would ❤ Kudos if my solution helped. 👉 If you can spend time posting the question, you can also make efforts to give Kudos to whoever helped to solve your problem. It is a token of appreciation!

⚡ Visit us at https://perytus.com, your one-stop-shop for Power BI-related projects/training/consultancy.