Refresh starts over

[donaldo]

Hi all,

I have a query (or a few) that was getting progressively slower and now often fails due to timing out.

I noticed in the Desktop application that the refresh of certain of the bigger queries would work once, very rapidly, loading thousands of rows per second from a SQL source.

Then it would finish and basically start over - but this time much, much slower. From taking seconds the first time aroudn to hours.

Please see the below GIF and watch the query named History.

Historys source is CostDownAnalysis above but with the added step of merging with a fact table and expanding certain colums in the second step as below. How can I speed this up and overall improve both speed and stability of this query?

let
    Source = Table.NestedJoin(CostDownAnalysis,{"IdeaId"},#"Idea Table",{"IdeaID"},"Idea Table",JoinKind.LeftOuter),
    #"Expanded Idea Table" = Table.ExpandTableColumn(Source, "Idea Table", {"True Percentage", "Segment", "Division", "Product Line", "Plant"}, {"Idea Table.True Percentage", "Idea Table.Segment", "Idea Table.Division", "Idea Table.Product Line", "Idea Table.Plant"})
in
    #"Expanded Idea Table"

[watkinnc]

Renaming the column becomes necessary when using Table.Join (as opposed to Table.NestedJoin) because you can't have two columns with the same name, which will happen after the join (unless you use JoinKind.Inner.

--Nate

[donaldo]

Hello,

I tried to implement your solution. To begin with, History would now be stuck for ours on "Evaluating" as the only query.

 

It seems to take ~5 minutes in DAX Studio - but goes on forever in PowerBI Desktop.

This is my new query

let
    Source = Table.Join(CostDownAnalysis,{"IdeaIdCDA"},#"Idea Table",{"IdeaID"},JoinKind.LeftOuter)
in
    Source

[BA_Pete]

Hi @donaldo ,

Firstly, +2 for the tidy, well-explained request and the gif. Really easy to help when people take the time to ask their question carefully and give clear information.

I think @watkinnc has given you a smart solution that you should go with, but I just wanted to cover your implicit question about why your refresh appears to restart:

It's not actually restarting as such, but performing data loads for two different parts of your query. This happens with merges because there's the initial load of the table, then there's the load of it as it's merged with a second table. Even though the 'double-load' appears to be occurring on the same table (and in some respects it is) there's a second table to consider within the query, so you need to re-run the load for every row in the tableto match to your second table.

@watkinnc knows some really cool tricks to speed up merges so I'd definitely be giving what he suggests a go. Myself, I tend to avoid merges like the plague for exactly the reasons above.

Pete

[watkinnc]

@BA_Pete is definitely correct, ralationships in a data model definitely beats joins/merges.

[watkinnc]

Before the merge: rename your IdeaID column in the second table to Idea.2. Then replace Table.NestedJoin with Table.Join and remove the new column name parameter (in quotes before JoinKind.Outer.

This will significantly improve the performance. Essentially, we are just using Table.Join instead of Table.NestedJoin. 

--Nate

[donaldo]

Thank you for the suggestion. I will try this solution out.

1. I am unsure which table you are refering to as "the second table".
   1. I am guessing the fact table named "Idea Table"?
2. What is the reason to rename it?
3. Will this disrupt any relationships the [Idea Table] IdeaID column has?

[watkinnc]

The reason is that when you expand the nested tables, you are basically running the query for each row in the table.

--Nate