Forum Discussion
How to make Power Query faster?
Does anyone know of a way to work with Power Query in a much faster way?
I am merging 3 different tables, which have infinite rows, because they are tables that come from combinations of several other tables. But, always to do any transformation or merge, it takes a long time to load.
Does anyone know a way to make it faster, so I don't lose so much time?
6 Replies
Performance depends on a lot of specific details. If there were optimizations that were universally true, then they would be built in. If your tables have infinite rows, then loading or transforming will take an infinite amount of time regardless of the rate involved.
Assuming you're asking about real-world (finite) load, to help determine if optimizations are possible, we'd need to know what data sources (e.g. CSV, folder, SQL server, SharePoint, etc.) you're working with and the approximate number of rows from each (just the order of magnitude is fine but infinite isn't a real answer). If they're all from a single SQL database, then you're much more likely to be able to make things faster than if you're trying to merge huge tables from separate sources.
Dude, it has about 500,000 rows. I pulled the data through a folder, where there were several CSV files and merged them to have only one table. This final table has more or less 500,000 rows.
OK. Half a million rows is likely small enough to buffer the component tables so you can join them in memory. Loading CSVs from a folder will be slow no matter what but if you buffer them after loading, you should be able to do merges and transformations pretty quickly on tables that small.
I'd recommend checking out parry2k's post here for more detail on buffering:
