Forum Discussion
Referencing Power Query queries by other Power queries
- 5 years ago
Yes, just understand the size of the tables does have an impact. Table.Buffer() is an "it depends" function when it comes to performance. See this post by Ehren - he is on the MS PQ Product Team. When to use Table.Buffer (microsoft.com)
Please mark one or more of these answers as the solution so this thread can be shown to be resolved.
That video was incremental helpful - thanks! Anyone doing hardcore hand coded M code should watch that video.
Combining the 5 queries into one super query along with the use of Table.Buffer worked! It cut the run time on my small test from 285 seconds to 105 seconds. I can also tell by the way I tested it the first 65 seconds of that 105 seconds was loading the Table.Buffer. That performance gain is going to become much more significant as build out the real queries with significantly more logic.
I used the approach of Custom Functions to create the super query. It worked perfectly because I was able to directly cut and paste the M code from each of the 5 queries completely unmodified. This means I can maintain the original query and build the super query.
One factor in my environment is my workstation and the SQL server are in two physically separate locations so each time it pulls data from the server it needs to use an external network - a network that has a lot of traffic during business hours. That Table.Buffer saved pulling the data down 3 times. Even though I have 128GB workstation I read that Power BI caps memory usage so I suspect that there may be efficiencies caused by using Table.Buffer everyone warns about but they are more than offset by avoiding the network transfers.
It is going to take awhile to document this in a teaching mode - I will publish an article in this thread in the next 7-10 days. But the bottom line is there is a way to have Table.Buffer work with multiple queries to speed things up!
Yes, just understand the size of the tables does have an impact. Table.Buffer() is an "it depends" function when it comes to performance. See this post by Ehren - he is on the MS PQ Product Team. When to use Table.Buffer (microsoft.com)
Please mark one or more of these answers as the solution so this thread can be shown to be resolved.