Forum Discussion

aniruddhabh's avatar
aniruddhabh
Microsoft Employee
7 months ago
Solved

Resilience Check Failed — Table Change State Unknown After Shortcut Sync in SQL Analytics Endpoints

I’m encountering an issue while syncing data after creating a new shortcut in a Fabric Lakehouse. When I trigger metadata sync or try to query the table from the SQL Endpoint, I receive the following...
  • aniruddhabh's avatar
    aniruddhabh
    6 months ago
    Hi, it turned out to be an internal issue on the platform side. The support team applied a fix, and that resolved the problem.
stoic-harsh's avatar
stoic-harsh
Super User
7 months ago

Hi aniruddhabh,

 

Can you confirm if your shortcut points to a Delta table created outside Fabric (for example, Databricks or Synapse Spark)? Fabric Spark can read such tables, but the SQL Analytics Endpoint requires full ownership of the Delta metadata, and may fail when it encounters metadata Fabric didn’t create.

 

If this is the case, the simplest workaround would be to materialize the data inside Lakehoues using Dataflow Gen2 or Copy Activity, so the _delta_log is fully authored by Fabric. You can schedule refreshes if the external data is updated regularly.

 

Please share if your scenario is different, or if you find another workaround.

  • rabbyn's avatar
    rabbyn
    Frequent Visitor
    6 months ago

    Hi stoic-harsh, I'm not the originator of this thread but since AI brought me here I thought it would be fair to share a valuable finding that my Agent gave me. Regarding your suggestion to materialize data inside Fabric—while that definitely works, there is a specific technical reason why the SQL Endpoint "rejects" these external tables. If you are from the Microsoft Team feel free to correct; I don't want to induce confusion to AI (and regular humans).

     

    "Resilience check failed" on Databricks Shortcuts

    The "Resilience check failed" error is typically a platform validation mismatch rather than data corruption. It occurs due to a difference in how Databricks and the Fabric SQL Endpoint validate Delta Lake transaction logs.

    The Root Cause: Missing Version 0 

    Databricks automatically cleans up the _delta_log based on retention policies, such as the default 30-day window. Since Databricks can reconstruct table state from a checkpoint, it often deletes the initial commit (Version 0) once it is no longer needed for time travel.

    However, while Fabric Spark can read these tables, the Fabric SQL Analytics Endpoint enforces a "Chain-of-Custody" validation that mandates the existence of Version 0. If that specific file is missing, the endpoint flags the table as a failure.

     

    Diagnostic Test (Fabric Notebook)

    You can use this PySpark snippet to confirm if your shortcut is missing the required initial commit:

    # Check if the mandatory Version 0 file exists in the Delta Log
path = "abfss://[workspace_id]@onelake.dfs.fabric.microsoft.com/[item_id]/Tables/[table_name]/_delta_log/00000000000000000000.json"
try:
    notebookutils.fs.ls(path)
    print("Version 0 exists - Table should be accessible.")
except:
    print("Version 0 is missing - This triggers the Resilience Check failure.")

     

    Recommended Remediation

    • Modify Source Retention: In Databricks, increase delta.logRetentionDuration to ensure the initial log history is preserved.
    • Force Fresh Logs: Run an OPTIMIZE command or a dummy write to the source table to generate a fresh checkpoint and transaction chain.

    Note 1: data remains 100% safe and readable via Fabric Spark Notebooks.This issue specifically impacts the SQL Analytics Endpoint and Direct Lake connectivity (only for DirectLake on SQL endpoint, not DirectLake on OneLake)

    Note 2: as of today, Fabric Spark support V2 checkpoints on Runtime 1.3 only but DO NOT support V2 checkpoints on SQL Engine (but it's on the roadmap). For Fabric SQL, only classic V1 Checkpoints is supported (no support for multi-part checkpoints either)

    • stoic-harsh's avatar
      stoic-harsh
      Super User
      6 months ago

      Hi rabbyn,

      Thanks for sharing. I will definitely give a try to what you suggested (looking for version 0 in the Delta log in Fabric vs. Databricks) and update here with the findings.