Forum Discussion

dpollozhani's avatar
dpollozhani
Helper I
1 year ago
Solved

Legacy Timestamp Rebase Support in Native Execution Engine 1.3

I'm working on a notebook with shortcuts to Dynamics 365 data stored in a ADLS gen2 container (as a result of Synapse Link + Spark). It seems the parquet files written by the Synapse Link contain leg...
data engineering
environment
notebook
  • dpollozhani's avatar
    dpollozhani
    1 year ago

    I did not file a support ticket for this, but I believe the issue has been resolved by the subsequent updates/patches from the team. Unless my D365 source stopped producing legacy timestamps, it works if I run this now:

     

nilendraFabric's avatar
nilendraFabric
Super User
1 year ago

Hello dpollozhani 

try these as well 

 

spark.sql.legacy.parquet.int96RebaseModeInRead=CORRECTED
spark.sql.legacy.parquet.int96RebaseModeInWrite=CORRECTED
spark.sql.legacy.parquet.datetimeRebaseModeInRead=CORRECTED
spark.sql.legacy.parquet.datetimeRebaseModeInWrite=CORRECTED


Please let me know if this works 

Thanks 

dpollozhani's avatar
dpollozhani
Helper I
1 year ago

Here is my current configuration in the Environment:

runtime_version: '1.3'
spark_conf:
  - spark.sql.session.timeZone: Europe/Copenhagen
  - spark.gluten.legacy.timestamp.rebase.enabled: 'true'
  - spark.sql.legacy.parquet.datetimeRebaseModeInRead: CORRECTED
  - spark.sql.legacy.parquet.datetimeRebaseModeInWrite: CORRECTED
  - spark.sql.legacy.parquet.int96RebaseModeInRead: CORRECTED
  - spark.sql.legacy.parquet.int96RebaseModeInWrite: CORRECTED

  I should note that the following

- spark.gluten.legacy.timestamp.rebase.enabled: 'true'
- spark.sql.legacy.parquet.int96RebaseModeInRead
- spark.sql.legacy.parquet.int96RebaseModeInWrite

 are not recognized by the Environment UI - if that has any implication at all.

 

I also tried with 'LEGACY' instead of 'CORRECTED' on all the settings.

 

Still, the same result:

Caused by: java.lang.RuntimeException: Exception: VeloxUserError
Error Source: USER
Error Code: UNSUPPORTED
Reason: Reading legacy timestamp is not supported.

 

  • nilendraFabric's avatar
    nilendraFabric
    Super User
    1 year ago

    %%configure
    {"conf": {"spark.gluten.legacy.timestamp.rebase.enabled": "true"}}

     

     

    Give it a try 

    • dpollozhani's avatar
      dpollozhani
      Helper I
      1 year ago

      Nope, unfortunately, still same issue.

       

       

       

      • Anonymous's avatar
        Anonymous
        Not applicable
        1 year ago

        I recently encountered the same issue. In my scenario I tried multiple times with setting the configuration for the spark session but I still got the legacy timestamp not supported error. But then I found out that my spark session was attached to a specific environment which overrulled my spark session configurations. 

         

        So please be aware of this. In short, if you do not run environment and make the following configuration inside the spark session then it will work. The configurations are:

        spark.conf.set("spark.sql.parquet.int96RebaseModeInRead", "CORRECTED")
        spark.conf.set("spark.sql.parquet.int96RebaseModeInWrite", "CORRECTED")
        spark.conf.set("spark.sql.parquet.datetimeRebaseModeInRead", "CORRECTED")
        spark.conf.set("spark.sql.parquet.datetimeRebaseModeInWrite", "CORRECTED")
        spark.conf.set("spark.sql.legacy.parquet.datetimeRebaseModeInRead", "CORRECTED")
        spark.conf.set("spark.sql.legacy.parquet.datetimeRebaseModeInWrite", "CORRECTED")
        spark.conf.set("spark.sql.legacy.timeParserPolicy", "CORRECTED"),
        spark.conf.set("spark.sql.legacy.parquet.int96RebaseModeInRead", "CORRECTED")
        spark.conf.set("spark.sql.legacy.parquet.int96RebaseModeInWrite", "CORRECTED")