Forum Discussion
Referencing Power Query queries by other Power queries
- 5 years ago
Yes, just understand the size of the tables does have an impact. Table.Buffer() is an "it depends" function when it comes to performance. See this post by Ehren - he is on the MS PQ Product Team. When to use Table.Buffer (microsoft.com)
Please mark one or more of these answers as the solution so this thread can be shown to be resolved.
I'll try to answer these Anonymous -
First of all, Power Query works bottom up, not top down. So if the 5 query is loaded, it starts there and works backwards to get what it needs. That is why Query 1 gets called several times. 5 calls 2, 3, and 4, and those each call 1.
Putting all of that code in one superquery will not change how it works. PQ looks at the path of the query and a query refrences isn't viewed as another query as I understand it, so having 3 queries refernces each other in a chain - Q1 -> Q2 -> Q3 will act as if it were 1 query with all of the 3 queries just being steps. The advantage of refrences is so Q1 can be split into Q2, Q3, and Q4, each with their own logic and transformations, then rejoined at the end, or merged, or whatever, without having to rewrite the Q1 logic 4 times. Change Q1's transformations later and it changes all downstream query results.
The other reason I use references is in Excel. Say I have 1 table with 1 set of transformations, but I need to load it into both an Excel table and the Excel data model for Power Pivot. I don't load Q1 at all. I do Q1, then create 2 references - Q2 and Q3. Q2 gets loaded to an Excel table, and Q3 to the data model. There is no performance impact. However, if I later decide to tweak Q2 or Q3 independently, I can, or I can choose to unload one without affecting the other. If you unload a single query in Excel that is loaded to both a table and the data model, it will unload it from both places and then reload it back to the one you want automatically. That trashes everything though because when the table is temporarily removed from either a sheet or the data model, any dependent formulas, relationships, and/or measures are wrecked.
Table.Buffer() will not help you here. It will not take Q1 and store it in RAM, then prevent further calls back to the source. Table.Buffer() will load the entire table in memory vs streaming the contents one record at a time, which is what it normally does. This can help performance in some cases, but can hurt it drastically in others. What it is intended to do is preserve the state of the final results of that query and prevent it from changing - sort order being the most common issue it solves. But be careful. If that query has 3M records and is several GB in size, your machine will slow to a crawl as it brings it in to RAM.
You should take a look at this video - it will explain a lot and I think you'll find it really useful. How Power Query Thinks: Taking the Mystery Out of Streaming and Query Folding (Video) | Ben Gribaudo