Forum Discussion

prathijp's avatar
prathijp
Helper I
1 year ago
Solved

Parquet file to delta table using Notebooks: Issue with Datetime column

I have parquet file in lakehouse with the below columns. When I try to save the file as lakhouse table I am getting error with the date field. Please suggest if any solution available? Also date column is not available in sql endpoint once saved.

dataref = "/Files/Bronze/CT_Value.parquet"

datadf = spark.read.parquet(dataref)
datadf.write.mode("overwrite").format("delta").save("Tables/CT_Value")
 
Error: "Columns of the specified data types are not supported for (ColumnName: '[date] TIMESTAMP_NTZ')."
 
 
File is as below

 

data engineering
Lakehouse
Notebook
  • Ilgar_Zarbali's avatar
    Ilgar_Zarbali
    1 year ago

    The error happens because the TIMESTAMP_NTZ type (timestamp without time zone) doesn’t work with Lakehouse tables in Fabric when using Delta format.

    To fix it, change the date column to a supported type like DATE or normal TIMESTAMP before saving.

    If the column type isn’t supported, it won’t show up in SQL after saving — but converting it will fix that and make it visible.

2 Replies

  • Ilgar_Zarbali's avatar
    Ilgar_Zarbali
    Super User
    1 year ago

    The error happens because the TIMESTAMP_NTZ type (timestamp without time zone) doesn’t work with Lakehouse tables in Fabric when using Delta format.

    To fix it, change the date column to a supported type like DATE or normal TIMESTAMP before saving.

    If the column type isn’t supported, it won’t show up in SQL after saving — but converting it will fix that and make it visible.

  • v-prasare's avatar
    v-prasare
    Community Support
    1 year ago

    prathijp, As we haven’t heard back from you, we wanted to kindly follow up to check if the solution provided for your issue worked? or let us know if you need any further assistance here?

     

    Ilgar_Zarbali, Thanks for your prompt response.

     

     

    Thanks,

    Prashanth Are

    MS Fabric community support

     

    If this post helps, then please consider Accept it as the solution to help the other members find it more quickly and give Kudos if helped you resolve your query