Forum Discussion
Legacy Timestamp Rebase Support in Native Execution Engine 1.3
I did not file a support ticket for this, but I believe the issue has been resolved by the subsequent updates/patches from the team. Unless my D365 source stopped producing legacy timestamps, it works if I run this now:
Here is my current configuration in the Environment:
runtime_version: '1.3'
spark_conf:
- spark.sql.session.timeZone: Europe/Copenhagen
- spark.gluten.legacy.timestamp.rebase.enabled: 'true'
- spark.sql.legacy.parquet.datetimeRebaseModeInRead: CORRECTED
- spark.sql.legacy.parquet.datetimeRebaseModeInWrite: CORRECTED
- spark.sql.legacy.parquet.int96RebaseModeInRead: CORRECTED
- spark.sql.legacy.parquet.int96RebaseModeInWrite: CORRECTEDI should note that the following
- spark.gluten.legacy.timestamp.rebase.enabled: 'true'
- spark.sql.legacy.parquet.int96RebaseModeInRead
- spark.sql.legacy.parquet.int96RebaseModeInWriteare not recognized by the Environment UI - if that has any implication at all.
I also tried with 'LEGACY' instead of 'CORRECTED' on all the settings.
Still, the same result:
Caused by: java.lang.RuntimeException: Exception: VeloxUserError
Error Source: USER
Error Code: UNSUPPORTED
Reason: Reading legacy timestamp is not supported.
%%configure
{"conf": {"spark.gluten.legacy.timestamp.rebase.enabled": "true"}}
Give it a try
Nope, unfortunately, still same issue.
I recently encountered the same issue. In my scenario I tried multiple times with setting the configuration for the spark session but I still got the legacy timestamp not supported error. But then I found out that my spark session was attached to a specific environment which overrulled my spark session configurations.
So please be aware of this. In short, if you do not run environment and make the following configuration inside the spark session then it will work. The configurations are:
spark.conf.set("spark.sql.parquet.int96RebaseModeInRead", "CORRECTED")spark.conf.set("spark.sql.parquet.int96RebaseModeInWrite", "CORRECTED")spark.conf.set("spark.sql.parquet.datetimeRebaseModeInRead", "CORRECTED")spark.conf.set("spark.sql.parquet.datetimeRebaseModeInWrite", "CORRECTED")spark.conf.set("spark.sql.legacy.parquet.datetimeRebaseModeInRead", "CORRECTED")spark.conf.set("spark.sql.legacy.parquet.datetimeRebaseModeInWrite", "CORRECTED")spark.conf.set("spark.sql.legacy.timeParserPolicy", "CORRECTED"),spark.conf.set("spark.sql.legacy.parquet.int96RebaseModeInRead", "CORRECTED")spark.conf.set("spark.sql.legacy.parquet.int96RebaseModeInWrite", "CORRECTED")As I stated in previous comments, I am using an Environment, but that's where I'm primarily trying to set the configuration (which is what you're supposed to be able to). Secondly, I tried overriding the presets from the Environment by making a forced configuration in the notebook. None of it works.
My guess is that sessions that are attached to Environments are currently not working as intended, as seems to be the general rule with Environments until now (very unstable). Perhaps some settings are not applied at all and ignored even when set in the notebook. It could be a case where you have to force an Environment rebuild by switching runtimes back and forth (which I've had to do when I encountered issues with custom library publishing).
Next I will have to try to do the rebuild, and after that try to run Native in a notebook without an Environment.