Forum Discussion
Refresh starts over
Hi all,
I have a query (or a few) that was getting progressively slower and now often fails due to timing out.
I noticed in the Desktop application that the refresh of certain of the bigger queries would work once, very rapidly, loading thousands of rows per second from a SQL source.
Then it would finish and basically start over - but this time much, much slower. From taking seconds the first time aroudn to hours.
Please see the below GIF and watch the query named History.
Historys source is CostDownAnalysis above but with the added step of merging with a fact table and expanding certain colums in the second step as below. How can I speed this up and overall improve both speed and stability of this query?
let
Source = Table.NestedJoin(CostDownAnalysis,{"IdeaId"},#"Idea Table",{"IdeaID"},"Idea Table",JoinKind.LeftOuter),
#"Expanded Idea Table" = Table.ExpandTableColumn(Source, "Idea Table", {"True Percentage", "Segment", "Division", "Product Line", "Plant"}, {"Idea Table.True Percentage", "Idea Table.Segment", "Idea Table.Division", "Idea Table.Product Line", "Idea Table.Plant"})
in
#"Expanded Idea Table"
7 Replies
Before the merge: rename your IdeaID column in the second table to Idea.2. Then replace Table.NestedJoin with Table.Join and remove the new column name parameter (in quotes before JoinKind.Outer.
This will significantly improve the performance. Essentially, we are just using Table.Join instead of Table.NestedJoin.
--Nate
Hi Anonymous ,
Firstly, +2 for the tidy, well-explained request and the gif. Really easy to help when people take the time to ask their question carefully and give clear information.
I think Anonymous has given you a smart solution that you should go with, but I just wanted to cover your implicit question about why your refresh appears to restart:
It's not actually restarting as such, but performing data loads for two different parts of your query. This happens with merges because there's the initial load of the table, then there's the load of it as it's merged with a second table. Even though the 'double-load' appears to be occurring on the same table (and in some respects it is) there's a second table to consider within the query, so you need to re-run the load for every row in the tableto match to your second table.
Anonymous knows some really cool tricks to speed up merges so I'd definitely be giving what he suggests a go. Myself, I tend to avoid merges like the plague for exactly the reasons above.
Pete
BA_Pete is definitely correct, ralationships in a data model definitely beats joins/merges.
Hello,
I tried to implement your solution. To begin with, History would now be stuck for ours on "Evaluating" as the only query.
It seems to take ~5 minutes in DAX Studio - but goes on forever in PowerBI Desktop.
This is my new querylet Source = Table.Join(CostDownAnalysis,{"IdeaIdCDA"},#"Idea Table",{"IdeaID"},JoinKind.LeftOuter) in Source
