Python Notebook: writing in existing table error due to datatype Null

Hi Testaaaaaa,

Thanks for reaching MS Fabric community support.

please refer to below code to ensure that Nan values are handled properly:

df = pd.DataFrame(all_items)

# Replace NaN with None for all columns
df = df.fillna(None)

# Replace NaN in any specific column with a placeholder value (for example, 0 for integers or a default date for datetime columns)
df['some_column'] = df['some_column'].fillna(0).astype(int)
df['Insert_Datetime'] = pd.to_datetime(today[0:-6], errors='coerce').fillna(pd.Timestamp('2025-01-01'))

# Drop columns that are completely NaN
df = df.dropna(axis=1, how='all')

# Check the data before writing
print(df.info())  # Check for any remaining NaN or None values

if not df.empty:
    table_path = f"{lakehouse_details_abfsPath}/Tables/{lakehouse_table}"
    write_deltalake(table_path, df, mode='append', schema_mode='merge', engine='rust', storage_options=storage_options)
    print(f"Number of records added in {lakehouse_table}: {len(df)}")

Thanks,

Prashanth Are

MS fabric community support