Python Notebook recreating lakehouse table after deleting issue

[MangoMagic]

During development & testing, have to regularly delete lakehouse tables through Explorer side panel and try to recreate it using python code, but it often comes with an error as there are some leftover files in the filesystem for Table folder (probably a log folder).

Is there a better way to create lakehouse tables in Python notebook that doesn't cause an error when trying to recreate table with same name?

write_deltalake(
    delta_table_path,
    df,
    mode="overwrite",
    schema_mode="overwrite",
    engine="rust",
    storage_options={"allow_unsafe_rename": "true"}
)

---------------------------------------------------------------------------
DeltaError                                Traceback (most recent call last)
Cell In[40], line 12
      7 df = df.apply(lambda col: col if col.name in ["revision","_ETL_Time"]
      8         else col.apply(lambda x: json.dumps(x) 
      9         if not isinstance(x, str) else x))
     10 #df.info()
---> 12 write_deltalake(
     13     delta_table_path,
     14     df,
     15     mode="overwrite",
     16     schema_mode="overwrite",
     17     engine="rust",
     18     storage_options={"allow_unsafe_rename": "true"}
     19 )
     21 display(df)

File ~/jupyter-env/python3.11/lib/python3.11/site-packages/deltalake/writer.py:304, in write_deltalake(table_or_uri, data, schema, partition_by, mode, file_options, max_partitions, max_open_files, max_rows_per_file, min_rows_per_group, max_rows_per_group, name, description, configuration, schema_mode, storage_options, partition_filters, predicate, large_dtypes, engine, writer_properties, custom_metadata)
    301     return
    303 data = RecordBatchReader.from_batches(schema, (batch for batch in data))
--> 304 write_deltalake_rust(
    305     table_uri=table_uri,
    306     data=data,
    307     partition_by=partition_by,
    308     mode=mode,
    309     table=table._table if table is not None else None,
    310     schema_mode=schema_mode,
    311     predicate=predicate,
    312     name=name,
    313     description=description,
    314     configuration=configuration,
    315     storage_options=storage_options,
    316     writer_properties=(
    317         writer_properties._to_dict() if writer_properties else None
    318     ),
    319     custom_metadata=custom_metadata,
    320 )
    321 if table:
    322     table.update_incremental()

DeltaError: Delta transaction failed, version 0 already exists.

[BhaveshPatel]

Hi @MangoMagic 

You should delete the single table through notebook by right clicking Delta Table and Delete the table. 

Delete Single Delta Table at a time from Explorer Side Panel. There is no alternative way.

 Alternatively, You can drop and recreate the delta table 

DROP TABLE IF EXISTS DimTable

CREATE TABLE IF NOT EXISTS DimTable
USING DELTA

This is managed delta table ( Table with metadata )

[v-venuppu]

Hi @MangoMagic ,

Thank you for reaching out to Microsoft Fabric Community.

Thank you @lbendlin for the prompt response.

I wanted to check if you had the opportunity to review the information provided and resolve the issue..?Please let us know if you need any further assistance.We are happy to help.

Thank you.

[lbendlin]

in your notebook code include a "remove if exists"  step that clears up the remaining artifacts before you attempt to create the new artifact