Forum Discussion
Pivote columns refresh error
Buffering doesn't affect folding. It affects speed. It gathers all of the items in the list at once, then sends the buffered list to the SQL server in the IN clause.
If you don't buffer, it streams the data into the IN clause and takes significantly longer.
As a result, folding or not, I always use a buffered list for List.Contains. And yup, Anonymous taught that trick to me.
Hi Anonymous ,
There's a few things here:
1) Using an Excel/SharePoint source to filter an SQL source is going to break query folding, so your subsequent transformation steps are going to be performed locally, rather than on the source server.
2) Pivot/unpivot are 'whole table' operations, i.e. the whole table has to be loaded into memory before performing the operation, unless you can fold the operation back to the server to do (See Point 1). If you have broken query folding and your workstation and/or gateway do not have sufficient RAM, these types of operations can easily max out the machine and cause excessively long refresh times.
3) I can't be 100% sure, but it appears as if the error is due to server deadlocking. If your queries are taking hours to run (See Point 2), you are applying shared locks to all the tables you are querying on the server for a very long time. When the server actually wants to update one of those tables as part of a larger transaction, it can cause a 'deadlock' situation, where neither PBI nor the server can complete their transaction without the other releasing its locks. In this case, the server will just terminate one of the transactions to let the other complete.
It may not even be a full deadlock that's causing the error, it may just be the fact that the server doesn't want you applying a shared lock to any table for that amount of time.
My suggestion would be to remove the Excel/SharePoint filter in Power Query, but load it to the data model to be used as a dimension table of sorts, effectively filtering your main table via a table relationship. Once this is removed from your main query, then ensure that your query folds back to the server. I would expect this to speed up your refresh by orders of magnitude.
Pete
Wow, thanks!
I have to say, that you answer opened my eyes. I've worked for 1 year with Power BI, but never heard about somehitng as important as query folding. I've just started reading about it, but for sure it will have an impact on my work.
I have one question, as I understand filtering my main table causes problems, as it breaks query folding. Would changing order (first pivoting, then filtering based on excel) help something?
Of course I will try your solution, but I was wondering if my idea would work.