Forum Discussion

Antongig's avatar
Antongig
Frequent Visitor
1 year ago
Solved

Cannot Remove Invalid Table "_delta_log._commits" — SQL Sync Blocked

Hello, I'm experiencing a persistent issue in Microsoft Fabric Lakehouse.   Lakehouse: Silver_Lakehouse Issue: SQL Sync fails due to a phantom table _delta_log._commits. Error Message: Delta ta...
data warehouse
  • BhaveshPatel's avatar
    BhaveshPatel
    1 year ago

    Hi Antongig 

     

    Somewhere you deleted the transaction log file or data is corrupted due to _delta_log. For that, you need to delete the whole table ( single table ) and recreate the table using Python or scala commands which ever you think is relevant. 

     

    Also, Which database are you using as a source. I mean to say Azure SQL. SQL Server...etc. 

Antongig's avatar
Antongig
Frequent Visitor
1 year ago

Hi Bhavesh, thanks for replying!

I use Silver_Lakehouse as part of a production data pipeline where:

  • Tables are copied from Bronze to Silver using Spark notebooks and pipelines

  • SQL sync is required for warehouse access and reporting

  • Several downstream reports and jobs depend on these tables

The issue is that Fabric tries to sync a non-existent table _delta_log._commits, likely due to a failed write or interrupted notebook cell.
I’ve tried:

  • Dropping the table via Spark SQL (DROP TABLE)

  • Deleting folders via file paths

  • Recreating and overwriting the table

  • Listing all tables (it doesn't appear)

Nothing worked. Now SQL sync always fails.
I cannot delete the Lakehouse easily because many pipelines and references are linked to it by name.

Do you have any suggestion for clearing this ghost table or triggering a hard SQL sync refresh? 

BhaveshPatel's avatar
BhaveshPatel
Icon for Super User rankSuper User
1 year ago

Hi Antongig 

 

Somewhere you deleted the transaction log file or data is corrupted due to _delta_log. For that, you need to delete the whole table ( single table ) and recreate the table using Python or scala commands which ever you think is relevant. 

 

Also, Which database are you using as a source. I mean to say Azure SQL. SQL Server...etc. 

  • Antongig's avatar
    Antongig
    Frequent Visitor
    1 year ago

    The Lakehouse uses Delta Lake format underneath on OneLake storage. 

    I already tried deleting and recreating the table, but Fabric’s SQL sync still reports _delta_log._commits as a table, even though it doesn’t exist anymore. I tried:

    spark.sql("DROP TABLE IF EXISTS Silver_Lakehouse.`_delta_log._commits`")

    and I also tried:

    tables_path = "/lakehouse/default/Tables"
for i in os.listdir(tables_path):
    if "commits" in i:
        print(i)

    in my notebook in fabric. Both does absolutely nothing.

      • v-menakakota's avatar
        v-menakakota
        Icon for Community Support rankCommunity Support
        1 year ago

        Hi Antongig ,

        We’d like to follow up on your support ticket. Has the issue been successfully addressed?

        If a solution was provided, we’d truly appreciate it if you could share your experience or any relevant details, your input could be valuable to other community members.

         

        Thanks again for being an active part of the Microsoft Fabric Community!