Python Notebook recreating lakehouse table after deleting issue

[MangoMagic]

During development & testing, have to regularly delete lakehouse tables through Explorer side panel and try to recreate it using python code, but it often comes with an error as there are some leftover files in the filesystem for Table folder (probably a log folder).

Is there a better way to create lakehouse tables in Python notebook that doesn't cause an error when trying to recreate table with same name?

write_deltalake(
    delta_table_path,
    df,
    mode="overwrite",
    schema_mode="overwrite",
    engine="rust",
    storage_options={"allow_unsafe_rename": "true"}
)

---------------------------------------------------------------------------
DeltaError                                Traceback (most recent call last)
Cell In[40], line 12
      7 df = df.apply(lambda col: col if col.name in ["revision","_ETL_Time"]
      8         else col.apply(lambda x: json.dumps(x) 
      9         if not isinstance(x, str) else x))
     10 #df.info()
---> 12 write_deltalake(
     13     delta_table_path,
     14     df,
     15     mode="overwrite",
     16     schema_mode="overwrite",
     17     engine="rust",
     18     storage_options={"allow_unsafe_rename": "true"}
     19 )
     21 display(df)

File ~/jupyter-env/python3.11/lib/python3.11/site-packages/deltalake/writer.py:304, in write_deltalake(table_or_uri, data, schema, partition_by, mode, file_options, max_partitions, max_open_files, max_rows_per_file, min_rows_per_group, max_rows_per_group, name, description, configuration, schema_mode, storage_options, partition_filters, predicate, large_dtypes, engine, writer_properties, custom_metadata)
    301     return
    303 data = RecordBatchReader.from_batches(schema, (batch for batch in data))
--> 304 write_deltalake_rust(
    305     table_uri=table_uri,
    306     data=data,
    307     partition_by=partition_by,
    308     mode=mode,
    309     table=table._table if table is not None else None,
    310     schema_mode=schema_mode,
    311     predicate=predicate,
    312     name=name,
    313     description=description,
    314     configuration=configuration,
    315     storage_options=storage_options,
    316     writer_properties=(
    317         writer_properties._to_dict() if writer_properties else None
    318     ),
    319     custom_metadata=custom_metadata,
    320 )
    321 if table:
    322     table.update_incremental()

DeltaError: Delta transaction failed, version 0 already exists.

[lbendlin]

in your notebook code include a "remove if exists"  step that clears up the remaining artifacts before you attempt to create the new artifact