Forum Discussion
How to make Power Query faster?
Performance depends on a lot of specific details. If there were optimizations that were universally true, then they would be built in. If your tables have infinite rows, then loading or transforming will take an infinite amount of time regardless of the rate involved.
Assuming you're asking about real-world (finite) load, to help determine if optimizations are possible, we'd need to know what data sources (e.g. CSV, folder, SQL server, SharePoint, etc.) you're working with and the approximate number of rows from each (just the order of magnitude is fine but infinite isn't a real answer). If they're all from a single SQL database, then you're much more likely to be able to make things faster than if you're trying to merge huge tables from separate sources.
Dude, it has about 500,000 rows. I pulled the data through a folder, where there were several CSV files and merged them to have only one table. This final table has more or less 500,000 rows.
OK. Half a million rows is likely small enough to buffer the component tables so you can join them in memory. Loading CSVs from a folder will be slow no matter what but if you buffer them after loading, you should be able to do merges and transformations pretty quickly on tables that small.
I'd recommend checking out parry2k's post here for more detail on buffering:
I had the same issue in the past. Turned out my issue wasn't with rows as Power BI is more than capable of handling 500k rows. My issue was appending the csv. The more csv files I threw it at, the slower it got regardless of how many rows it was in total. That's just one downside to using a sharepoint folder and appending several files with power query. POwer Query is not that good handling it in my personal experience.
I've since moved on and started using power automate to append new csv data onto a base csv file and using that csv as source.
Imrans123 Raw text files like CSV just aren't a very efficient source to load data from but it's definitely faster to load from one big one than from lots of smaller ones. Appending incrementally with Power Automate is a nice way to do the appending once rather than every time you refresh the query. You could get even better query performance if you used Power Automate to load the data to something more like a database with a query engine rather than a text file.
So, but on that particular report you will have no updates.
