rss.livelink.threads-in-node

Fabric Warehouse statistics: do you manage them manually or rely on automatic statistics?

jubinsoni — Sun, 08 Nov 2026 00:48:11 GMT

Hi everyone,

We have a Warehouse where large fact tables are loaded in batches several times a day. A few reports that were fast last month now show slower plans on the same queries, and I suspect stale statistics after the loads.

The thing I am considering is adding a step at the end of each load, something like:

-- after the batch load UPDATE STATISTICS dbo.FactSales; -- or targeted, on columns used in joins and filters CREATE STATISTICS st_FactSales_DateKey ON dbo.FactSales (DateKey);

Questions

Do you rely on automatic statistics, or add explicit updates to your load pipeline? What made you decide?
How do you check that statistics are the cause of a slow query before changing anything?
Is there a point where updating statistics after every load costs more than it saves?

Examples of how you decide, and any numbers you saw before and after, would be very helpful. Thanks!

How Can Organizations Improve Data Warehouse Performance and Data Quality at Scale?

codeautomation — Tue, 03 Nov 2026 17:56:18 GMT

As organizations collect data from multiple applications, databases, and business systems, maintaining a reliable and efficient data warehouse becomes increasingly challenging.

Performance issues, inconsistent records, failed data loads, and delayed reporting can affect business decisions and daily operations.

I'm interested in learning how different teams address these challenges.

A few questions for the community:

What strategies do you use to optimize data warehouse query performance?
How do you automate data quality checks and detect inconsistent records?
What is your approach to handling schema changes without breaking existing pipelines?
How do you monitor scheduled data loads and recover from failed executions?
Which tools or features in Microsoft Fabric have helped improve warehouse reliability?

I'd appreciate hearing about real-world implementations, recommended practices, and lessons learned.

What approaches have delivered the best results for your organization?

Fabric DWH - RLS - Segregate/disable RLS for PBI refreshes and ETL workloads

Ayush05-gateway — Mon, 02 Nov 2026 14:33:12 GMT

I am looking at suggestions on how to avoid impact of security policies on PowerBI refreshes which are running through Service Principle part of contributor role in workspace.

After applying policies for RLS, all reports are slowing down.

There is slowdown in insert/update too by ETL workloads but it is not concerning but with PBI refresh, some models fail with timeout.

Is there a way to put some admin users in a db_owner or simlar role so policy is not checked.

just FYI, SchemaBinding OFF/ON is not helping

Fabric Warehouse blocking caused by a sleeping Spark session with an open transaction

BIstvan — Mon, 02 Nov 2026 08:34:58 GMT

Hi everyone,

I've been running a Fabric notebook that writes data to a Warehouse table for about 1.5 years without any issues.

The notebook uses this code:

sanitized_df_df_clean.write.mode("overwrite").synapsesql(target_tbl)

Recently, I had a problem where several Power BI refresh queries were blocked.

After checking the Warehouse sessions, I found that session 52 was sleeping but still had an open transaction (open_transaction_count = 1).

Looking into queryinsights.exec_requests_history, I noticed that the Spark connector created two sessions.

Session 52 executed SET IMPLICIT_TRANSACTIONS ON and some metadata queries (SELECT TOP(1) and sp_describe_first_result_set), but apparently never committed or rolled back.

Session 55 executed TRUNCATE TABLE, loaded the data using COPY INTO, and successfully committed the transaction.

Both sessions belonged to the same Spark application.

I ran KILL 52, and the blocking disappeared.

The strange part is that I reran the same pipeline afterward, without changing anything, and everything worked normally. No blocking and no open transactions.

Has anyone experienced something similar with the Fabric Spark-to-Warehouse connector?

Why would a session remain sleeping with an open transaction after the notebook finishes? Could this be related to a recent Fabric update? Is there a way to prevent this from happening again without changing the entire notebook or pipeline?

Since this has been working fine for 1.5 years, I'd really like to understand what might have changed.

Thanks in advance for any suggestions!

Fabric Warehouse blocking caused by a sleeping Spark session with an open transaction

BIstvan — Mon, 02 Nov 2026 08:34:54 GMT

Hi everyone,

I've been running a Fabric notebook that writes data to a Warehouse table for about 1.5 years without any issues.

The notebook uses this code:

sanitized_df_df_clean.write.mode("overwrite").synapsesql(target_tbl)

Recently, I had a problem where several Power BI refresh queries were blocked.

After checking the Warehouse sessions, I found that session 52 was sleeping but still had an open transaction (open_transaction_count = 1).

Looking into queryinsights.exec_requests_history, I noticed that the Spark connector created two sessions.

Session 52 executed SET IMPLICIT_TRANSACTIONS ON and some metadata queries (SELECT TOP(1) and sp_describe_first_result_set), but apparently never committed or rolled back.

Session 55 executed TRUNCATE TABLE, loaded the data using COPY INTO, and successfully committed the transaction.

Both sessions belonged to the same Spark application.

I ran KILL 52, and the blocking disappeared.

The strange part is that I reran the same pipeline afterward, without changing anything, and everything worked normally. No blocking and no open transactions.

Has anyone experienced something similar with the Fabric Spark-to-Warehouse connector?

Since this has been working fine for 1.5 years, I'd really like to understand what might have changed.

Thanks in advance for any suggestions!

Warehouse SCD Type 2 loads: MERGE or INSERT/UPDATE pattern for large dimensions?

jubinsoni — Fri, 30 Oct 2026 05:28:36 GMT

Hi everyone,

I am designing a Type 2 slowly changing dimension load in a Fabric Warehouse for a dimension with a few million rows. Two options on the table:

A single MERGE that closes changed rows and inserts new versions.
Two statements: an UPDATE that closes the current row for changed keys, followed by an INSERT of the new versions.

A sketch of the two-statement version:

UPDATE d SET d.valid_to = SYSUTCDATETIME(), d.is_current = 0 FROM dbo.dim_customer AS d JOIN stg.customer AS s ON s.customer_id = d.customer_id WHERE d.is_current = 1 AND s.row_hash <> d.row_hash; INSERT INTO dbo.dim_customer (customer_id, name, row_hash, valid_from, valid_to, is_current) SELECT s.customer_id, s.name, s.row_hash, SYSUTCDATETIME(), NULL, 1 FROM stg.customer AS s LEFT JOIN dbo.dim_customer AS d ON d.customer_id = s.customer_id AND d.is_current = 1 WHERE d.customer_id IS NULL OR d.row_hash <> s.row_hash;

Questions:

Is MERGE considered reliable enough in the Warehouse for SCD2, or do people prefer the explicit two-step approach?
How do you wrap the two statements so a failure between them does not leave the dimension half updated?
Any guidance on small file and compaction concerns when the table is updated frequently?

Thanks in advance.

Does Fabric Runtime 2.0 update affect Querying LH via Fabric Browser UI functions?

anon97242 — Thu, 29 Oct 2026 22:05:27 GMT

Hello,

We're currently evaluating the impact of migrating to Fabric Runtime 2.0 . We don't make use of spark jobs/notebooks as part of our environment. However, when reviewing the Fabric Capacity Metrics report, and looking at item history, if we filter on the SparkCore items, we see that LH items appear as utilizing this. This appears to be due to SQL ran through the browser Fabric UI.

My question is do we need to consider this in our testing or is this passed to Microsoft managed spark runtime, similar to how the Delta table upgrade in LH/WH is managed by Microsoft? And will this continue to work if the runtime is not updated?

Thanks!

LakeHouse SQL Endpoint connection disruptions

BenT — Wed, 28 Oct 2026 15:18:29 GMT

Hello,

Since a couple of weeks, we are constantly experiencing the connection issue to the SQL Endpoint our Lakehouses whenever we try to run the Python code on a local computer.
we always have to try to establish the connection to the SQL Endpoint by running a query in Fabric or trying to connect to another on-premisses database, before being able to running any code (multiple times a day), but still the issue persist and we would have to do this multiple times for it to start working.

Are there any disruptions on the SQL Endpoint connections?

Regards,

Benjamin T

How are you detecting drift between Fabric environments?

hijesh — Tue, 27 Oct 2026 13:01:36 GMT

How do you know that what’s running in PROD is actually the same version that passed through DEV and UAT?

That’s a problem I’ve been exploring recently in Microsoft Fabric.

I built a Warehouse Drift Detection solution that compares SQL objects and their definitions across:

DEV → UAT → PREPROD → PROD

It identifies scenarios such as:

objects that haven’t been promoted;
objects missing from an environment;
definitions that differ between environments;
PROD differing from DEV/UAT;
potential direct changes in an individual environment.

It also maintains comparison history and, where definitions differ, shows the actual SQL differences.

The idea is simple:

Make it easy to see where Fabric Warehouse environments have diverged and understand exactly what changed.

I thought I’d share the approach here as this is probably a challenge other teams working with multiple Fabric environments encounter as well.

How are you currently handling environment drift in Microsoft Fabric?

Fabric Warehouse as source in MS Access

JVDA98 — Tue, 27 Oct 2026 07:08:15 GMT

Hi All,

I have a client that is still using an Access application for some very specific modules. We recently migrated the on-prem database to Fabric & now some of the connections of that Access application needs to change to the Fabric Warehouse.

Using the ODBC driver, this seems to work, the only issue is that it looks like the user that is using the application has to have "VIEWER" access on the workspace, otherwise the connection cannot be successfully performed.

We've tried by removing the viewer access, and giving granualar connect + readData permission on the specific warehouse, but this throws a connection error (even before we can select a warehouse).
It seems like it needs to have "connect" permissions on the "server".

Does anyone have experience with this set-up? Best practices? Service principal,... doesn't seem an option at first sight.

Thanks !

Item Reference variable not saved on Lakehouse shortcut Target Item

arehman36 — Fri, 23 Oct 2026 16:43:13 GMT

Hello everyone,

I have a workspace with a Lakehouse called Test_LH. Test_LH contains a schema shortcut to a warehouse called TEST_WH which is in a different workspace. I have a library variable that contains an Item Reference to TEST_WH. When i press manage shortcut on my schema shortcut inside Test_LH, and then press 'Edit Target' -> 'Target Item', and then select my library varialbe that contains an item Reference to TEST_WH, the selection does not save. Now, if i press manage shortcut again, it will show that the Target Item is TEST_WH, rather than the library variable. I have pressed "Update all variables" but this has not fixed the issue.

Here are some things I have tried

Logging out of Fabric, closing all my tabs and logging in again.
Double checked to ensure that my item reference was correct

I am not sure why It will not show the Target Item to be my variable after selecting it.

Thanks,

Issue with the DL model deployement

Ar_Sh — Thu, 22 Oct 2026 06:19:38 GMT

I have a Direct Lake semantic model connected to a Fabric Warehouse. After deploying it from DEV to QA, the existing data source connection was not configured correctly, so I had to create a new connection using Workspace Identity.Is this expected behavior with Fabric deployment pipelines, or is there a way to preserve the target environment's connection settings?

erratic connection Power BI Desktop --> Fabric SQL DW

vanessa_fvg — Wed, 21 Oct 2026 14:50:04 GMT

We have an ongoing issue that when you connect to Fabric SQL DW via Power BI Desktop through imported mode ONLY SOMETIMES it gives this error below and other times you can connect easily. this is not for a big dataset it is literally 2000 rows.

Microsoft SQL: A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: Named Pipes Provider, error: 40 - Could not open a connection to SQL Server)

I have changed the timeout values it makes no difference. When i look at the monitor in the SQL DW it doesn't even show that its querying. Not sure where this is getting stumped.

any ideas of where i can diagnose what is going on?

Trasaction deadlocked on lock resources

jnlucas — Wed, 21 Oct 2026 13:07:12 GMT

Hi,

I created a Dataflow Gen2 in Microsoft Fabric that loads multiple tables into a warehouse. Every time I run the dataflow, I receive the following error message:

Couldn't refresh the entity because of an issue with the mashup document MashupException.Error: Microsoft SQL: Transaction (Process ID 206) was deadlocked on lock resources with another process and has been chosen as the deadlock victim. Rerun the transaction. Details: Reason = DataSource.Error;ErrorCode = 10478;DataSourceKind = Warehouse;DataSourcePath = Warehouse;DataSourceKind.2 = SQL;DataSourcePath.2 = qenzxdde3pdenc5pgrl56uss44-3nktrgkfckbedi5tclkxsvoogi.datawarehouse.fabric.microsoft.com;WH_Bronce;Message = Transaction (Process ID 206) was deadlocked on lock resources with another process and has been chosen as the deadlock victim. Rerun the transaction.;ErrorCode = -2146232060;Number = 1205;Class = 13;State = 51;ConnectionId = 090e3c20-db6d-492b-b2a6-48814400d26f;Microsoft.Data.Mashup.Error.Context = User GatewayObjectId: 714e5a6b-cca0-4a5a-bb6b-7c891e9991ba". Código de error: 104100. (Id. de solicitud: 4926601a-0aa6-4cdf-86d4-e84f3cbd1767).

Any idea what it might be?

Thank you in advance.

How to know Warehouse name on Query Window (SMSS)

alfBI — Wed, 14 Oct 2026 06:34:41 GMT

Hi,

I am using SSMS to connect Fabric WH. I have created server registrations to have friendly names, but even with that I have a problem when opening multiple query windows against different WH.

Every window is connected to a different WH but the only way to distinguish it is checking the wonderful String ID shown on the right bottom bar. Unfortunately the friendly name used to regisyter the server does not appear here.

Any idea about how to distinguish easily which WH corresponds to every opened query window?

Thx

NetSuite data in Fabric: where do you put KPI logic, Spark notebooks, Warehouse or DAX?

kiara-mehra35 — Mon, 12 Oct 2026 13:27:16 GMT

Hi all,

I’d like to ask how you’d structure KPI logic in a Fabric setup built on NetSuite data, because I keep going back and forth on where it should live.

The main source is NetSuite ERP, covering finance, sales, CRM, inventory and procurement, along with a few other enterprise systems. Fabric Data Pipelines handle the scheduled transfers into OneLake, and the Lakehouse is organized in Bronze, Silver and Gold layers to separate raw, refined and curated data. Fabric Notebooks with Spark do the cleansing, transformations, business rules and KPI calculations before anything reaches reporting. A Fabric Data Warehouse then serves structured datasets for cross-functional reporting, and Power BI semantic models with DAX measures and RLS sit on top, feeding 50+ dashboards across finance, sales, operations, inventory and executive reporting. Security runs through Entra ID, access controls, lineage and encryption.

The part I’m unsure about is KPI logic. Right now some of it is calculated in Spark before the data reaches the Warehouse, and the rest is defined as DAX measures in the semantic models. How do you decide what goes where? Do you push everything you can into the Gold layer and keep DAX thin, or keep the business definitions in the semantic model so they’re easier to change?

I’d also like to hear how you handle NetSuite ingestion. With pipelines running on a schedule, how do you deal with incremental loads, late-arriving changes and deletes? Do you ever do full reloads for certain tables?

On the Lakehouse vs. Warehouse split, what made you choose one over the other for the serving layer? Did you run into duplicated logic or extra copies of the data between them?

And with 50+ reports, do you keep one semantic model per domain or a few shared ones? How do you manage RLS in that setup, especially as self-service use grows and more people start building their own reports on top?

Has anyone gone through this and changed their approach later?

How are teams structuring Microsoft Fabric Warehouse for scalable ELT and reporting workloads?

codeautomation — Wed, 07 Oct 2026 21:13:48 GMT

I’m looking at architecture patterns for Microsoft Fabric Warehouse in environments where multiple data sources, transformation jobs, and Power BI workloads all depend on the same platform.

A typical flow might look like:

Source systems → ingestion → staging → transformations → Fabric Warehouse → semantic models → Power BI

I’m particularly interested in how teams are handling:

staging vs curated schemas
ELT transformations inside the Warehouse
incremental loading patterns
data quality checks before publishing
CI/CD across dev, test, and production
workload separation for engineering vs reporting
monitoring failed loads and long-running queries
deciding when to use Warehouse vs Lakehouse for the same solution

For teams running Fabric Warehouse at scale, what architecture patterns have worked best for keeping the platform maintainable and avoiding duplicated transformations?

Unable to Connect Azure Synapse Database to Power BI Desktop

ShivaPr — Wed, 07 Oct 2026 06:17:41 GMT

Hi Team,

While I am trying to connect Azure Synapse DB to Power BI Desktop getting error.

Steps performed

1.Obtained the Azure Synapse SQL endpoint and database credentials.
2.Opened Power BI Desktop and selected Azure Synapse Analytics as the data source.
3.Entered the Synapse SQL endpoint and database connection details.
4.Attempted to establish the connection.
5.Received an error indicating that Power BI Desktop is unable to establish a connection to the Azure Synapse database.

Can some one help me how to resolve it ?

DataflowsStagingLakehouse Size

nicolasf — Mon, 05 Oct 2026 15:25:42 GMT

Hello,

My DataflowsStagingLakehouse system storage is 103GB! I have many DF Gen2 with enabled staging, but I really don't understand why this lakehouse is so big!

How can I explain me that this storage is so big?

What are the good pratices for this kind of lakehouse ?

Near-real-time CDC into Fabric Warehouse: MERGE vs Mirroring vs Lakehouse?

jubinsoni — Sat, 03 Oct 2026 00:01:37 GMT

Hi all,

I need operational data from an Azure SQL Database available in Fabric for reporting with roughly 5 to 15 minutes of latency. There are about 30 tables, mostly updates, with some deletes. Business users will query it in T-SQL and Power BI.

I see three main approaches and would like to understand the trade-offs:

Mirroring the source database into Fabric, then building curated tables in a Warehouse on top of it
Incremental MERGE into Warehouse tables from a pipeline or copy job that reads changed rows (CDC or a watermark column)
Eventstream CDC → Lakehouse Delta tables, queried through the SQL analytics endpoint

Questions:

Which approach is recommended at this latency and scale, and where does each one break down?
How are deletes handled in each? Mirroring is the most direct, but what about MERGE-based loads?
How much does running MERGE frequently cost on a Warehouse (capacity usage, small-file or fragmentation effects)?
If you mirror, is it better to build curated tables in a Warehouse or use views over the mirrored database?

Thanks!