Forum Discussion

n5s5n's avatar
n5s5n
Frequent Visitor
1 year ago

When using Runtime 1.3 in Notebook, errors occur in tables containing Lakehouse date columns.

How can I use Runtime 1.3(Spark 3.5, Delta 2.4) in Notebook to manipulate tables containing LakeHouse datetime columns?   I am developing a conversion process for CSV files.   I stored data from ...
lakehouse
notebook
n5s5n's avatar
n5s5n
Frequent Visitor
1 year ago

Hi Anonymous


I appreciate your advice. I tried the pySpark query you provided, but I got an error message saying "Only UTC adjusted Timestamp is not supported."

Py4JJavaError: An error occurred while calling z:com.microsoft.spark.notebook.visualization.display.getDisplayResultForIPython.
: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 23.0 failed 4 times, most recent failure: Lost task 0.3 in stage 23.0 (TID 127) (vm-cff19570 executor 1): org.apache.gluten.exception.GlutenException: java.lang.RuntimeException: Exception: VeloxRuntimeError
Error Source: RUNTIME
Error Code: INVALID_STATE
Reason: Only UTC adjusted Timestamp is supported.
Retriable: False
Expression: logicalType.TIMESTAMP.isAdjustedToUTC
Context: Split [Hive: abfss://[email protected]/xxxxxxx/Tables/table_name/xxxxxxxx.parquet 0 - 416531] Task Gluten_Stage_23_TID_127
Additional Context: Operator: TableScan[0] 0
Function: makeDecoder
File: /__w/1/s/Velox/velox/dwio/parquet/reader/PageReader.cpp
Line: 711

 

When I first encountered this error, I thought about changing Spark's timezone to UTC and added the following code: 

spark.conf.set("spark.sql.session.timeZone", "UTC")

However, the same error occurs: "Only UTC adjusted Timestamp is not supported."

 

When querying Delta tables using PySpark in runtime 1.2, an error occurred when running the query directly, but by setting the following before execution, the query worked perfectly.

spark.conf.set("spark.sql.session.timeZone", "UTC")
spark.conf.set("spark.sql.parquet.datetimeRebaseModeInRead", "CORRECTED")

 

f there are any parts of my explanation that are difficult to understand, I would appreciate it if you could point them out.

Best Regards,

Anonymous's avatar
Anonymous
Not applicable
1 year ago

Hi n5s5n 

 

Is it possible to identify which column in the delta table might causing this problem? What's the data type of this column? 

Or could you provide some steps to reproduce this issue? I'd like to try to reproduce it.

 

Best Regards,
Jing

  • n5s5n's avatar
    n5s5n
    Frequent Visitor
    1 year ago

    Hi Anonymous 
    Thank you for reaching out to me, and I apologize for replying in the wrong way earlier.

     

    I'm using Dataflow(Gen2) to insert data into a table. When creating the table, I configured the dataflow to create a new table and I configured the destination type as datetime.

    This error occurs on a table that contains a column of type "timestamp (nullable = true)" in Lakehouse. 

    Even when I run a PySpark query to create a table with a datetime-type column and then insert data into it from a dataflow, the result is the same(The query was executed with Runtime 1.3.).

     

    Regarding the error with the datetime column, when the column contains null values, PySpark queries such as SELECT and UPDATE statements can be executed without any issues.

     

    Best Regards,

    n5s5n