Mirrored Metadata Catalog for Databricks - No Data Agent?

Hi dbeavon3​ ,

If it helps, here's a quick notebook snippet to check which Delta features your Databricks tables are using. Attach the lakehouse with the shortcuts and run it.

import pandas as pd

rows = []
for db in spark.catalog.listDatabases():
    for t in spark.catalog.listTables(db.name):
        name = f"`{db.name}`.`{t.name}`"
        try:
            d = spark.sql(f"DESCRIBE DETAIL {name}").collect()[0].asDict()
            props = d.get("properties") or {}
            rows.append({
                "table": name,
                "reader_version": d.get("minReaderVersion"),
                "writer_version": d.get("minWriterVersion"),
                "features": ", ".join(d.get("tableFeatures") or []),
                "column_mapping": props.get("delta.columnMapping.mode", ""),
                "deletion_vectors": props.get("delta.enableDeletionVectors", ""),
                "error": ""
            })
        except Exception as e:
            rows.append({"table": name, "error": str(e)[:300]})

display(pd.DataFrame(rows))

Then run this on the SQL analytics endpoint to see which tables actually made it through.

SELECT s.name AS schema_name, t.name AS table_name

FROM sys.tables t

JOIN sys.schemas s ON t.schema_id = s.schema_id

ORDER BY schema_name, table_name;

Any table that shows up in the notebook but not on the SQL endpoint is a likely suspect, and the features column should hint at why. Tables with a high reader version or features like deletion vectors or column mapping are the first ones I'd look at.

AI-assisted drafting: AI was used to help structure and phrase this response. I reviewed and validated the technical content before posting.