Forum Discussion

n5s5n's avatar
n5s5n
Frequent Visitor
1 year ago

When using Runtime 1.3 in Notebook, errors occur in tables containing Lakehouse date columns.

How can I use Runtime 1.3(Spark 3.5, Delta 2.4) in Notebook to manipulate tables containing LakeHouse datetime columns?   I am developing a conversion process for CSV files.   I stored data from ...
lakehouse
notebook
Anonymous's avatar
Anonymous
Not applicable
1 year ago

Hi n5s5n 

 

Is it possible to identify which column in the delta table might causing this problem? What's the data type of this column? 

Or could you provide some steps to reproduce this issue? I'd like to try to reproduce it.

 

Best Regards,
Jing

n5s5n's avatar
n5s5n
Frequent Visitor
1 year ago

Hi Anonymous 
Thank you for reaching out to me, and I apologize for replying in the wrong way earlier.

 

I'm using Dataflow(Gen2) to insert data into a table. When creating the table, I configured the dataflow to create a new table and I configured the destination type as datetime.

This error occurs on a table that contains a column of type "timestamp (nullable = true)" in Lakehouse. 

Even when I run a PySpark query to create a table with a datetime-type column and then insert data into it from a dataflow, the result is the same(The query was executed with Runtime 1.3.).

 

Regarding the error with the datetime column, when the column contains null values, PySpark queries such as SELECT and UPDATE statements can be executed without any issues.

 

Best Regards,

n5s5n