Skip to main content
cancel
Showing results for 
Search instead for 
Did you mean: 

Enhance your career with this limited time 50% discount on Fabric and Power BI exams. Ends August 31st. Request your voucher.

Reply
BenL
New Member

Report Refresh/Mashup Engine and SQL Server Load

Hi,

 

I'm currently in the process of optimising our BI reports due to poor refresh speeds and the load they place on the SQL server. We are using an on-premise data gateway.

I've noticied that there are 3 SQL queries which are causing the high load, all 3 are being called by the Mashup Engine. Could anyone give me a bit more information about these? They are called several times per refresh over the current 20 minute period to complete a refresh and each take multiple seconds to complete, compounding to multiple minutes in total.

 

The where clause of the queries have been shortened.

Query 1

 

 

select 
	s.name TABLE_SCHEMA, 
	o.name TABLE_NAME, 
	i.name INDEX_NAME, 
	cc.name COLUMN_NAME, 
	cast ( ic.key_ordinal as bigint ) ORDINAL_POSITION, 
	i.is_primary_key PRIMARY_KEY 
from sys.objects o 
    join sys.schemas s on s.schema_id = o.schema_id 
    join sys.indexes as i on i.object_id = o.object_id 
    join sys.index_columns as ic on ic.object_id = i.object_id and ic.index_id = i.index_id 
    join sys.columns as cc on ic.column_id = cc.column_id and ic.object_id = cc.object_id 
where ( i.is_primary_key = ? or i.is_unique_constraint = ? or i.is_unique = ? ) and o.type in ( ? ) and ic.key_ordinal <> ? and ( ( s.name = N ? and o.name = N ? ) or ( s.name = N ? and o.name = N ? ) or ( s.name = N ? and o.name = N ? ) or (...) 
order by i.name, s.name, o.name

 

 

Query 2

 

select 
	convert ( nvarchar, fk.object_id ) FK_NAME, 
	cast ( f.constraint_column_id as bigint ) ORDINAL, 
	s1.name TABLE_SCHEMA_1, 
	o1.name TABLE_NAME_1, 
	c1.name PK_COLUMN_NAME_1, 
	s2.name TABLE_SCHEMA_2, 
	o2.name TABLE_NAME_2, 
	c2.name PK_COLUMN_NAME_2, 
	f.constraint_object_id, 
	f.constraint_column_id 
from sys.foreign_key_columns f 
    join sys.foreign_keys fk on f.constraint_object_id = fk.object_id 
    join sys.objects o1 on o1.object_id = f.parent_object_id 
    join sys.schemas s1 on s1.schema_id = o1.schema_id 
    join sys.objects o2 on o2.object_id = f.referenced_object_id 
    join sys.schemas s2 on s2.schema_id = o2.schema_id 
    join sys.columns c1 on c1.object_id = o1.object_id and c1.column_id = f.parent_column_id 
    join sys.columns c2 on c2.object_id = o2.object_id and c2.column_id = f.referenced_column_id 
where ( ( s1.name = N ? and o1.name = N ? ) or ( ...)) 
union select convert ( nvarchar, fk.object_id ) FK_NAME, cast ( f.constraint_column_id as bigint ) ORDINAL, s1.name TABLE_SCHEMA_1, o1.name TABLE_NAME_1, c1.name PK_COLUMN_NAME_1, s2.name TABLE_SCHEMA_2, o2.name TABLE_NAME_2, c2.name PK_COLUMN_NAME_2, f.constraint_object_id, f.constraint_column_id 
from sys.foreign_key_columns f 
    join sys.foreign_keys fk on f.constraint_object_id = fk.object_id 
    join sys.objects o1 on o1.object_id = f.parent_object_id 
    join sys.schemas s1 on s1.schema_id = o1.schema_id 
    join sys.objects o2 on o2.object_id = f.referenced_object_id 
    join sys.schemas s2 on s2.schema_id = o2.schema_id 
    join sys.columns c1 on c1.object_id = o1.object_id and c1.column_id = f.parent_column_id 
    join sys.columns c2 on c2.object_id = o2.object_id and c2.column_id = f.referenced_column_id 
where ( ( s2.name = N ? and o2.name = N ? ) or (...)) 
order by f.constraint_object_id, f.constraint_column_id

 

 

Query 3

 

select 
	s.name TABLE_SCHEMA, 
	o.name TABLE_NAME, 
	c.name COLUMN_NAME, 
	cast ( c.column_id as bigint ) ORDINAL_POSITION, 
	c.is_nullable IS_NULLABLE, 
	case when ( t.is_user_defined = ? and t.name is not ? ) then t.name when ( c.system_type_id = ? or t.name is ? ) then ? else t_system.name end DATA_TYPE, 
	case when ( c.system_type_id in ( ? ) ) then ? when ( c.system_type_id in ( ? ) ) then ? else ? end NUMERIC_PRECISION_RADIX,
	c.precision NUMERIC_PRECISION, 
	case when ( c.system_type_id in ( ? ) ) then ? else c.scale end NUMERIC_SCALE, 
	case when ( c.system_type_id in ( ? ) ) then c.scale else ? end DATETIME_PRECISION, 
	case when ( c.system_type_id in ( ? ) ) then floor ( c.max_length / ? ) when ( c.system_type_id in ( ? ) ) then c.max_length else ? end CHARACTER_MAXIMUM_LENGTH, 
	cast ( e.value as nvarchar ( max ) ) DESCRIPTION, 
	d.definition COLUMN_DEFAULT, 
	cc.definition COLUMN_EXPRESSION, 
	case when c.is_identity = ? or c.is_computed = ? or t.system_type_id = ? or c.generated_always_type > ? then ? else ? end IS_WRITABLE, ? FIELD_CAPTION 
from sys.objects o 
    join sys.schemas s on s.schema_id = o.schema_id 
    join sys.columns c on o.object_id = c.object_id 
    left join sys.types t on c.user_type_id = t.user_type_id 
    left join sys.types t_system on t.system_type_id = t_system.user_type_id 
    left join sys.default_constraints d on d.object_id = c.default_object_id 
    left join sys.computed_columns cc on c.object_id = cc.object_id and c.column_id = cc.column_id 
    left join sys.extended_properties e on o.object_id = e.major_id and c.column_id = e.minor_id and e.class = ? and e.name = ? 
where ( ( s.name = N ? and o.name = N ? ) or (...)

 

 

 

  

1 REPLY 1
Martin_D
Super User
Super User

Hi @BenL ,

 

Let's start with a few things that are useful to know about PowerQuery and querying a SQL Server from Power Query:

  1. Power Query does for each table that is loaded into the dataset first a query to get the schema and then a seperate query to get the data.
  2. Power Query tries to create SQL code from the steps that you created in Power Query, letting SQL Server do the transformations. It's not like the first Power Query step creates a SELECT... query and then the transformation is done in Power Query. Default behavior of Power Query is that it tries to push as much workload as possible to the SQL server. This behavior is called query folding.
  3. What looks like a nice reuse of queries by referencing queries in Power Query is not a reuse of data while loading. Each dataset table that refreshes queries the same referenced tables again from SQL Server.

Things you can do to improve performance:

  1. In order to get rid of the queries that query the schema you can statically provide the schema in the Power Query code as described here: https://blog.crossjoin.co.uk/2020/05/14/speed-up-data-refresh-performance-in-power-bi-desktop-using-...
  2. Although query folding usually results in the best performance, if the SQL server is slow, you can reduce the workload on the SQL Server and potentially speed up the refresh if you stop query folding and push more of the transformation workload from the SQL Server to Power Query, i.e., the Gateway. You can define in the Power Query steps what should be done on SQL Server and starting with which step Power Query should take over by calling Table.StopFolding as described here: https://powerquery.microsoft.com/en-us/blog/stop-query-folding-with-table-stopfolding/
  3. Since the Power Query is executed on the gateway, if you use a gateway, it's a good idea to install the gateway on a different machine than the SQL Server, especially if the SQL Server is slow anyway.
  4. Instead of creating transformation steps in Power Query, you can just put SQL Code into the Sql.Database step an write optimized SQL code for your needs as you would when creating a SQL view. https://learn.microsoft.com/de-de/powerquery-m/sql-database
  5. Even faster, of course, is to avoid transformations in the Power Query and at Power Query execution time at all by preprocessing the transformations on your SQL Server/DWH so that each Power Query then just loads a SQL table without transformations. You can do this e.g. using SSIS or stored procedures. Since these technologies can make use of persisted intermediate tables during transformation (other than Power Query where everything is kept in memory during transformation!) this can help to design faster and more ressource efficient transformations.
  6. Ideally step 5. helps to implement incremental refresh in Power BI which can significantly reduce the data moved during refresh. Maybe even without step 5. you can apply incremental refresh to reduce the refresh workload.

BR

Martin

Helpful resources

Announcements
July 2025 community update carousel

Fabric Community Update - July 2025

Find out what's new and trending in the Fabric community.

July PBI25 Carousel

Power BI Monthly Update - July 2025

Check out the July 2025 Power BI update to learn about new features.

Join our Fabric User Panel

Join our Fabric User Panel

This is your chance to engage directly with the engineering team behind Fabric and Power BI. Share your experiences and shape the future.