Forum Discussion

nguyenhieubis's avatar
nguyenhieubis
Frequent Visitor
2 years ago

Issues with Snapshot Isolation Transaction Failures in Cross-Warehouse Ingestion

I have a Workspace that includes:

  • Lakehouse lh_bronze: Where I create views at the SQL endpoint of the lakehouse.
  • Warehouse wh_silver: Where data from the views in lakehouse bronze is stored.
  • Data Pipelines: I have multiple SQL statements running concurrently to load data from lakehouse bronze to warehouse silver using cross-warehouse ingestion.

The SQL statements are structured as follows:

 

 

BEGIN TRY
    DROP TABLE IF EXISTS [wh_silver].[staging].[table_A];
    CREATE TABLE [wh_silver].[staging].[table_A]
    AS
    SELECT * FROM [lh_bronze].[export].[v_table_A];
    
    BEGIN TRANSACTION
        DELETE tgt
        FROM [wh_silver].[dbo].[table_A] tgt
        INNER JOIN [wh_silver].[staging].[table_A] src
            ON tgt.pk_col = src.pk_col;
        
        INSERT INTO [wh_silver].[dbo].[table_A]
        SELECT * FROM [wh_silver].[staging].[table_A];
    COMMIT TRANSACTION
END TRY
BEGIN CATCH
    IF @@TRANCOUNT > 0
    ROLLBACK TRANSACTION;
END CATCH;

 

 

 

I schedule these to run about 10 times a day and encounter approximately 2-4 error notifications daily with the following message:

 

 

Snapshot isolation transaction failed in database 'lh_bronze' because the object accessed by the statement has been modified by a DDL statement in another concurrent transaction since the start of this transaction. It is disallowed because the metadata is not versioned. A concurrent update to metadata can lead to inconsistency if mixed with snapshot isolation.
Statement ID: {D1533248-6EF4-4D8E-9BDB-C339B0083307} | Query hash: 0xFD91EA61D6EFE5A8 | Distributed request ID: {6D66C75B-BD4E-42DC-9B2C-628AF26F298A}
Statement ID: {2C3AA46C-A5AB-4554-BEA3-27347EB20256}

 

 

 

I have been searching for a solution on Google for the past month but have not been able to resolve the issue.

I would greatly appreciate your assistance in fixing this error and identifying the underlying problem.

8 Replies

  • v-cboorla-msft's avatar
    v-cboorla-msft
    Icon for Microsoft Employee rankMicrosoft Employee

    Hi nguyenhieubis 

     

    Thanks for using Microsoft Fabric Community.

     

    This error can occur if you are querying metadata under snapshot isolation and there is a concurrent DDL statement that updates the metadata that is being accessed under snapshot isolation. SQL Server does not support versioning of metadata. For this reason, there are restrictions on what DDL operations can be performed within an explicit transaction running under snapshot isolation.

    Please refer the below documentation for additional information.

    Link 1 : MSSQLSERVER_3961 - SQL Server | Microsoft Learn

    Link 2 : Snapshot isolation transaction fails when querying metadata - Microsoft Community Hub

     

    I hope this information helps. Please do let us know if you have any further queries.

     

    Thank you.

    • nguyenhieubis's avatar
      nguyenhieubis
      Frequent Visitor

      Hi v-cboorla-msft 

       

      Thanks for your response.

       

      I also referred to the two links you mentioned. However, after reviewing my SQL code, I can confirm that I am not using any DDL statements within the TRANSACTION block.

      I am only running parallel SQL statements as described in the original problem.

       

      Thank you.

      • v-cboorla-msft's avatar
        v-cboorla-msft
        Icon for Microsoft Employee rankMicrosoft Employee

        Hi nguyenhieubis 

         

        Could you please confirm whether you are getting above error message on each execution. Or you are observing this error on few executions.

         

        Thank you.