Forum Discussion
Issues with Snapshot Isolation Transaction Failures in Cross-Warehouse Ingestion
I have a Workspace that includes:
- Lakehouse lh_bronze: Where I create views at the SQL endpoint of the lakehouse.
- Warehouse wh_silver: Where data from the views in lakehouse bronze is stored.
- Data Pipelines: I have multiple SQL statements running concurrently to load data from lakehouse bronze to warehouse silver using cross-warehouse ingestion.
The SQL statements are structured as follows:
BEGIN TRY
DROP TABLE IF EXISTS [wh_silver].[staging].[table_A];
CREATE TABLE [wh_silver].[staging].[table_A]
AS
SELECT * FROM [lh_bronze].[export].[v_table_A];
BEGIN TRANSACTION
DELETE tgt
FROM [wh_silver].[dbo].[table_A] tgt
INNER JOIN [wh_silver].[staging].[table_A] src
ON tgt.pk_col = src.pk_col;
INSERT INTO [wh_silver].[dbo].[table_A]
SELECT * FROM [wh_silver].[staging].[table_A];
COMMIT TRANSACTION
END TRY
BEGIN CATCH
IF @@TRANCOUNT > 0
ROLLBACK TRANSACTION;
END CATCH;
I schedule these to run about 10 times a day and encounter approximately 2-4 error notifications daily with the following message:
Snapshot isolation transaction failed in database 'lh_bronze' because the object accessed by the statement has been modified by a DDL statement in another concurrent transaction since the start of this transaction. It is disallowed because the metadata is not versioned. A concurrent update to metadata can lead to inconsistency if mixed with snapshot isolation.
Statement ID: {D1533248-6EF4-4D8E-9BDB-C339B0083307} | Query hash: 0xFD91EA61D6EFE5A8 | Distributed request ID: {6D66C75B-BD4E-42DC-9B2C-628AF26F298A}
Statement ID: {2C3AA46C-A5AB-4554-BEA3-27347EB20256}
I have been searching for a solution on Google for the past month but have not been able to resolve the issue.
I would greatly appreciate your assistance in fixing this error and identifying the underlying problem.
8 Replies
- v-cboorla-msft
Microsoft Employee
Thanks for using Microsoft Fabric Community.
This error can occur if you are querying metadata under snapshot isolation and there is a concurrent DDL statement that updates the metadata that is being accessed under snapshot isolation. SQL Server does not support versioning of metadata. For this reason, there are restrictions on what DDL operations can be performed within an explicit transaction running under snapshot isolation.
Please refer the below documentation for additional information.
Link 1 : MSSQLSERVER_3961 - SQL Server | Microsoft Learn
Link 2 : Snapshot isolation transaction fails when querying metadata - Microsoft Community Hub
I hope this information helps. Please do let us know if you have any further queries.
Thank you.
- nguyenhieubisFrequent Visitor
Thanks for your response.
I also referred to the two links you mentioned. However, after reviewing my SQL code, I can confirm that I am not using any DDL statements within the TRANSACTION block.
I am only running parallel SQL statements as described in the original problem.
Thank you.
- v-cboorla-msft
Microsoft Employee
Could you please confirm whether you are getting above error message on each execution. Or you are observing this error on few executions.
Thank you.
- AndyDDC
Most Valuable Professional
Hi, it could be that another process is modifying the table. Can you read through this blog I wrote a while ago and see if any of it applies to your scenario?
https://www.serverlesssql.com/transaction-isolation-in-fabric-warehouses/