Forum Discussion

dpollozhani's avatar
dpollozhani
Helper I
1 year ago
Solved

Legacy Timestamp Rebase Support in Native Execution Engine 1.3

I'm working on a notebook with shortcuts to Dynamics 365 data stored in a ADLS gen2 container (as a result of Synapse Link + Spark). It seems the parquet files written by the Synapse Link contain leg...
data engineering
environment
notebook
  • dpollozhani's avatar
    dpollozhani
    1 year ago

    I did not file a support ticket for this, but I believe the issue has been resolved by the subsequent updates/patches from the team. Unless my D365 source stopped producing legacy timestamps, it works if I run this now:

     

nilendraFabric's avatar
nilendraFabric
Super User
1 year ago

Hello dpollozhani 

try these as well 

 

spark.sql.legacy.parquet.int96RebaseModeInRead=CORRECTED
spark.sql.legacy.parquet.int96RebaseModeInWrite=CORRECTED
spark.sql.legacy.parquet.datetimeRebaseModeInRead=CORRECTED
spark.sql.legacy.parquet.datetimeRebaseModeInWrite=CORRECTED


Please let me know if this works 

Thanks 

  • dpollozhani's avatar
    dpollozhani
    Helper I
    1 year ago

    Here is my current configuration in the Environment:

    runtime_version: '1.3'
spark_conf:
  - spark.sql.session.timeZone: Europe/Copenhagen
  - spark.gluten.legacy.timestamp.rebase.enabled: 'true'
  - spark.sql.legacy.parquet.datetimeRebaseModeInRead: CORRECTED
  - spark.sql.legacy.parquet.datetimeRebaseModeInWrite: CORRECTED
  - spark.sql.legacy.parquet.int96RebaseModeInRead: CORRECTED
  - spark.sql.legacy.parquet.int96RebaseModeInWrite: CORRECTED

      I should note that the following

    - spark.gluten.legacy.timestamp.rebase.enabled: 'true'
- spark.sql.legacy.parquet.int96RebaseModeInRead
- spark.sql.legacy.parquet.int96RebaseModeInWrite

     are not recognized by the Environment UI - if that has any implication at all.

     

    I also tried with 'LEGACY' instead of 'CORRECTED' on all the settings.

     

    Still, the same result:

    Caused by: java.lang.RuntimeException: Exception: VeloxUserError
Error Source: USER
Error Code: UNSUPPORTED
Reason: Reading legacy timestamp is not supported.

     

    • nilendraFabric's avatar
      nilendraFabric
      Super User
      1 year ago

      %%configure
      {"conf": {"spark.gluten.legacy.timestamp.rebase.enabled": "true"}}

       

       

      Give it a try 

      • dpollozhani's avatar
        dpollozhani
        Helper I
        1 year ago

        Nope, unfortunately, still same issue.