When using Runtime 1.3 in Notebook, errors occur in tables containing Lakehouse date columns.

Hi n5s5n   

I referred to several similar error posts on other forums. Based on the error messages, I guess that it is possible that Spark 3.5 and Spark 3.4 have different precision or range limits when handling Timestamp type data, which caused data overflow error after conversion.

Error Source: USER 
Error Code: INVALID_ARGUMENT 
Reason: Could not convert Timestamp(1732266818935202, 1732266818935202) to microseconds, Exception: VeloxUserError 
Error Source: USER 
Error Code: ARITHMETIC_ERROR 
Reason: integer overflow: 1732266818935202 * 1000000 
Retriable: False 
Function: checkedMultiply

1. VeloxUserError: Could not convert Timestamp(-9223372036854776, 192000000) to milliseconds · Issue #7161 · facebookincubator/velox · GitHub

2. Dates and timestamps should be converted to Pandas dataframe using microsecond resolution (instead of nanosecond) · Issue #6270 · duckdb/duckdb · GitHub

3. postgresql - "ERROR: timestamp out of range" converting epoch stored as bigint to a timestamp - Database Administrators Stack Exchange

As the data is displayed well in the lakehouse preview window, this indicates the data is stored without any error. It seems like the issue might be occurring during type conversion.

You can try removing the following two configuration lines:

spark.conf.set("spark.sql.session.timeZone", "UTC")
spark.conf.set("spark.sql.parquet.datetimeRebaseModeInRead", "CORRECTED")

and querying delta table directly with PySpark:

df = spark.sql("SELECT * FROM LakehouseName.sampletable LIMIT 1000")
display(df)

Test if this would work. This might help identify if these settings are causing the problem. If the issue persists, we might need to consider other aspects. 

Best Regards,
Jing
Community Support Team