Lakehouse Delta Table Bug

Hi AnthonySottile ,

Yes, this is a known issue that can occur in Fabric Lakehouse environments when there's a mismatch between the metadata layers used by Spark and the SQL endpoint.

Here's what's likely happening:

• The SQL endpoint uses its own metadata catalog and can often access tables even if the Spark engine has trouble resolving them.
• The Spark engine (used when loading via notebook > Load Data > Spark) relies on the Delta Lake transaction log and metadata files. If these are corrupted, missing, or out of sync, Spark may not recognize the table.

Possible Fixes:

1. Refresh the Lakehouse metadata
   Try restarting the Lakehouse or reloading the workspace. Sometimes metadata caching causes this issue.

2. Check for _delta_log integrity
   Navigate to the Lakehouse file explorer and ensure the _delta_log folder exists and is intact for that table.

3. Re-register the table
   If the table was manually created or altered, try dropping and recreating the table using Spark SQL to ensure it's properly registered in the Spark catalog.

4. Use spark.read.format("delta") manually
   In your notebook, try loading the table manually using:
   
   

df = spark.read.format("delta").load("Tables/YourTableName")