Python Notebook recreating lakehouse table after deleting issue

Let's first programmatically delete the table(s) properly, and thus let the underlying system perform all the necessary cleanup.

Single table

PySpark

spark.sql("DROP TABLE IF EXISTS schema_name.table_name")

If multiple tables in a given schema:

PySpark

schema = "your_schema_name"

# List all tables in schema
tables = spark.catalog.listTables(schema)

# Drop each table
for table in tables:
    table_name = table.name
    print(f"Dropping table {schema}.{table_name}")
    spark.sql(f"DROP TABLE IF EXISTS {schema}.{table_name}")

Combining these, let's look at a full PySpark solution:

schema = "your_schema_name"

# Step 1: List all tables in the schema
tables = spark.catalog.listTables(schema)

# Step 2: Drop all tables
for t in tables:
    print(f"Dropping table {schema}.{t.name}")
    spark.sql(f"DROP TABLE IF EXISTS {schema}.{t.name}")

# Step 3: Recreate all tables with a sample schema
# WARNING: Replace sample_schema with actual schemas if you want original structures back

sample_schema = "id INT, name STRING"

for t in tables:
    table_full_name = f"{schema}.{t.name}"
    print(f"Recreating table {table_full_name}")
    spark.sql(f"""
        CREATE TABLE {table_full_name} ({sample_schema})
        USING DELTA
        LOCATION '/mnt/datalake/{schema}/{t.name}'
    """)

If you still get complaints about the table names, this means the lakehouse server hasn't update its state yet. 

Hence, what I would do:

0_create pipeline

1_create a Notebook activity to run the delete Pyspark code above

2_use the Web activity with the new lakehouse REST API to force the lakehouse to sync (instead of a Wait activity because we can't know how long it's gonna take)

3_on success, next activity is a Notebook recreating the tables (somehow, you would need to save the table names somewhere, maybe another lakehouse dedicated only to store metadata)

I had the same issue with lakehouse tables on which write I/O wasn't finished by the time the next activity is ready to run. that was before the new REST API, so at the time I used a PySpark script to force the lakehouse to refresh and sync all metadata. and it worked. 

Haven't used the Web activity, so just a suggestion, and to be tested.