Skip to main content
cancel
Showing results for 
Search instead for 
Did you mean: 

Get certified in Microsoft Fabric—for free! For a limited time, get a free DP-600 exam voucher to use by the end of 2024. Register now

Reply
donaldo
Helper III
Helper III

Refresh starts over

Hi all,

 

I have a query (or a few) that was getting progressively slower and now often fails due to timing out.

 

I noticed in the Desktop application that the refresh of certain of the bigger queries would work once, very rapidly, loading thousands of rows per second from a SQL source.

 

Then it would finish and basically start over - but this time much, much slower. From taking seconds the first time aroudn to hours.

 

Please see the below GIF and watch the query named History.

 

Historys source is CostDownAnalysis above but with the added step of merging with a fact table and expanding certain colums in the second step as below. How can I speed this up and overall improve both speed and stability of this query?

 

let
    Source = Table.NestedJoin(CostDownAnalysis,{"IdeaId"},#"Idea Table",{"IdeaID"},"Idea Table",JoinKind.LeftOuter),
    #"Expanded Idea Table" = Table.ExpandTableColumn(Source, "Idea Table", {"True Percentage", "Segment", "Division", "Product Line", "Plant"}, {"Idea Table.True Percentage", "Idea Table.Segment", "Idea Table.Division", "Idea Table.Product Line", "Idea Table.Plant"})
in
    #"Expanded Idea Table"

 

 

powerbiload.gif

7 REPLIES 7
watkinnc
Super User
Super User

Renaming the column becomes necessary when using Table.Join (as opposed to Table.NestedJoin) because you can't have two columns with the same name, which will happen after the join (unless you use JoinKind.Inner.

 

--Nate


I’m usually answering from my phone, which means the results are visualized only in my mind. You’ll need to use my answer to know that it works—but it will work!!

Hello,

 

I tried to implement your solution. To begin with, History would now be stuck for ours on "Evaluating" as the only query.

 evaluating.gif

 

It seems to take ~5 minutes in DAX Studio - but goes on forever in PowerBI Desktop.

donaldo_1-1646407419849.png

 


This is my new query

let
    Source = Table.Join(CostDownAnalysis,{"IdeaIdCDA"},#"Idea Table",{"IdeaID"},JoinKind.LeftOuter)
in
    Source

 

BA_Pete
Super User
Super User

Hi @donaldo ,

 

Firstly, +2 for the tidy, well-explained request and the gif. Really easy to help when people take the time to ask their question carefully and give clear information.

 

I think @watkinnc has given you a smart solution that you should go with, but I just wanted to cover your implicit question about why your refresh appears to restart:

It's not actually restarting as such, but performing data loads for two different parts of your query. This happens with merges because there's the initial load of the table, then there's the load of it as it's merged with a second table. Even though the 'double-load' appears to be occurring on the same table (and in some respects it is) there's a second table to consider within the query, so you need to re-run the load for every row in the tableto match to your second table.

 

@watkinnc knows some really cool tricks to speed up merges so I'd definitely be giving what he suggests a go. Myself, I tend to avoid merges like the plague for exactly the reasons above.

 

Pete



Now accepting Kudos! If my post helped you, why not give it a thumbs-up?

Proud to be a Datanaut!




@BA_Pete is definitely correct, ralationships in a data model definitely beats joins/merges.


I’m usually answering from my phone, which means the results are visualized only in my mind. You’ll need to use my answer to know that it works—but it will work!!
watkinnc
Super User
Super User

Before the merge: rename your IdeaID column in the second table to Idea.2. Then replace Table.NestedJoin with Table.Join and remove the new column name parameter (in quotes before JoinKind.Outer.

 

This will significantly improve the performance. Essentially, we are just using Table.Join instead of Table.NestedJoin. 

--Nate


I’m usually answering from my phone, which means the results are visualized only in my mind. You’ll need to use my answer to know that it works—but it will work!!

Thank you for the suggestion. I will try this solution out.

 

  1. I am unsure which table you are refering to as "the second table".
    1. I am guessing the fact table named "Idea Table"?
  2. What is the reason to rename it?
  3. Will this disrupt any relationships the [Idea Table] IdeaID column has?

The reason is that when you expand the nested tables, you are basically running the query for each row in the table.

 

--Nate


I’m usually answering from my phone, which means the results are visualized only in my mind. You’ll need to use my answer to know that it works—but it will work!!

Helpful resources

Announcements
November Carousel

Fabric Community Update - November 2024

Find out what's new and trending in the Fabric Community.

Live Sessions with Fabric DB

Be one of the first to start using Fabric Databases

Starting December 3, join live sessions with database experts and the Fabric product team to learn just how easy it is to get started.

Las Vegas 2025

Join us at the Microsoft Fabric Community Conference

March 31 - April 2, 2025, in Las Vegas, Nevada. Use code MSCUST for a $150 discount! Early Bird pricing ends December 9th.

Nov PBI Update Carousel

Power BI Monthly Update - November 2024

Check out the November 2024 Power BI update to learn about new features.