DAX Studio and SQL profiler

[PowerRon]

Hi,

I ran a refresh of a dataset  for the first time. It took 4.5 hours. 

The main query in this is a view in SQLServer, being an aggregate on a table. When I run this view in SQLServer it takes 40 minutes, going from 1.3 billion rows to 9 million.
So refreshing takes long,also the second refresh. 4 hours. Incremental refresh keeps 4 years of history, refreshes the last 13 months (based on creation-date), but only if changed (detect data change on last-change-date).

Becaise refresh takes so long I use SQL Profiler in DAX Studio for the first time. Connecting to the shared dataset in the service goes fine, but once I start SQL Profiler I get an error

It says trace with id doesn't exist in the server or the user is not authorized for having access to the object.

How to solve this?
@smpa01 

@lbendlin 
Regards
Ron

[PowerRon]

I will @lbendlin but I presume refreshing does more than only running the query in SQL ...

[lbendlin]

If you want you can show a sanitized version of your Power Query code.  Unless you are doing lots of merge operations the M transforms usually take much less time than the actual source query. In any case you can use the Power Query Diagnostics options to figure that out.

[smpa01]

@PowerRon if I were you I would start with SQL query first. Why does it take 40 minutes to run a query ? Are the SQL tables missing any index? How can the SQL query be optimized.

[lbendlin]

agree with @smpa01  - you should focus on the Power Query part, not on DAX Studio.  Also be aware that you are dangerously close to hitting the hard limit of 300 minutes - after that any dataset or dataflow refresh on the service will be killed. You want to get your duration down to not more than 2.5 hrs (to be on the safe side). Keep in mind that your refresh is blocking a renderer for the entire time, which means that renderer is not available for other users either.

Read about bootstrapping for incremental refresh partitions.

[PowerRon]

I completely agree @lbendlin and @smpa01 . Just ran the view again in SQLServer. Took 23 minutes. But we are planning to add an index. The main issue for now is that I get that SQL Profiler error. So I can't get insights in why it takes so long.

Furthermore I will dive into bootstrapping. Never heard of it 🙂

[lbendlin]

Advanced incremental refresh and real-time data with the XMLA endpoint in Power BI - Power BI | Micr...

[PowerRon]

Thnx, is on my read list. Is there something else then SQL Profiler to get insight in why refreshing takes so long?

[smpa01]

@PowerRon once the index is sorted, and the SQL query is still slow, start dissecting it in parts.

There are lots of known performance optimization techniques within SQL (one syntax is better than other performance-wise for the same output) that can be used.

If you still have issues, stack-Sql-server  will rescue out, as it did for me many many many times.

One more thing to remember, it might be possible that a native SQL query that runs on SSMS in 3 minutes, PQ might return that query result in more than what SSMS takes. (I don't know why and I struggled with that a lot when I started playing out with large SQL tables). SO in that case, I learnt that the sever-side query needs to be super-optimized in order for PQ to not time out.

Also, you might have a choice to bring the whole SQL table `select * from tbl` and use PQ syntax to manipulate the table (to preserve the PQ Query Folding) which will be even more time consuming (from my experience) and I will advise against it (speaking from my bitter experience).

Is there something else then SQL Profiler to get insight in why refreshing takes so long? - I am not sure but try Fiddler.

[lbendlin]

Use SSMS and examine the actual query execution plan. It will even suggest indexes for you.