Forum Discussion

dpollozhani's avatar
dpollozhani
Helper I
1 year ago
Solved

Legacy Timestamp Rebase Support in Native Execution Engine 1.3

I'm working on a notebook with shortcuts to Dynamics 365 data stored in a ADLS gen2 container (as a result of Synapse Link + Spark). It seems the parquet files written by the Synapse Link contain legacy timestamps - at least that's how I interpret this.


Anyhow, I'm trying out the Native Execution Engine for Fabric 1.3, and according to this link, I should just be able to enable the timestamp rebase for Gluten. But even though I've done this in my Environment and restarted my Spark sessions in Fabric, and subsequently tried adding `SET spark.gluten.legacy.timestamp.rebase.enabled = true;` directly to the notebook in question - it doesn't work.

 

I'm still getting the error:
`Error Source: USER`
`Error Code: UNSUPPORTED`
`Reason: Reading legacy timestamp is not supported.`

 

Anyone have any ideas here?

data engineering
environment
notebook

13 Replies

  • nilendraFabric's avatar
    nilendraFabric
    Super User
    1 year ago

    Hello dpollozhani 

    try these as well 

     

    spark.sql.legacy.parquet.int96RebaseModeInRead=CORRECTED
    spark.sql.legacy.parquet.int96RebaseModeInWrite=CORRECTED
    spark.sql.legacy.parquet.datetimeRebaseModeInRead=CORRECTED
    spark.sql.legacy.parquet.datetimeRebaseModeInWrite=CORRECTED


    Please let me know if this works 

    Thanks 

    • dpollozhani's avatar
      dpollozhani
      Helper I
      1 year ago

      Here is my current configuration in the Environment:

      runtime_version: '1.3'
spark_conf:
  - spark.sql.session.timeZone: Europe/Copenhagen
  - spark.gluten.legacy.timestamp.rebase.enabled: 'true'
  - spark.sql.legacy.parquet.datetimeRebaseModeInRead: CORRECTED
  - spark.sql.legacy.parquet.datetimeRebaseModeInWrite: CORRECTED
  - spark.sql.legacy.parquet.int96RebaseModeInRead: CORRECTED
  - spark.sql.legacy.parquet.int96RebaseModeInWrite: CORRECTED

        I should note that the following

      - spark.gluten.legacy.timestamp.rebase.enabled: 'true'
- spark.sql.legacy.parquet.int96RebaseModeInRead
- spark.sql.legacy.parquet.int96RebaseModeInWrite

       are not recognized by the Environment UI - if that has any implication at all.

       

      I also tried with 'LEGACY' instead of 'CORRECTED' on all the settings.

       

      Still, the same result:

      Caused by: java.lang.RuntimeException: Exception: VeloxUserError
Error Source: USER
Error Code: UNSUPPORTED
Reason: Reading legacy timestamp is not supported.

       

      • nilendraFabric's avatar
        nilendraFabric
        Super User
        1 year ago

        %%configure
        {"conf": {"spark.gluten.legacy.timestamp.rebase.enabled": "true"}}

         

         

        Give it a try 

  • Anonymous's avatar
    Anonymous
    Not applicable
    1 year ago

    Hi dpollozhani ,
    Thank you for the helpful response nilendraFabric .
    In addition to Niledra's suggestion, I kindly request you to update to the latest versions of Fabric Runtime and Spark. These versions include support for the native execution engine and the necessary timestamp handling features.

    If this post helps, please give us Kudos and consider Accept it as a solution to help the other members find it more quickly.
    Thank you for being a aprt of Microsoft Fabric Community Forum!
    Regards,
    Pallavi.

    • dpollozhani's avatar
      dpollozhani
      Helper I
      1 year ago

      I'm already using the latest Fabric runtime, 1.3. According to the documentation (blog) the timestamp handling features are not yet set by default, which is also evident in my examples. 

  • v-prasare's avatar
    v-prasare
    Community Support
    1 year ago

    dpollozhani, please do raise a support ticket this will allow the a dedicated team to investigate and provide a resolution. Please include details of your setup, test results, and the impact on your workflows when submitting the ticket.

    How to create a Fabric and Power BI Support ticket - Power BI | Microsoft Learn

     

     

     

    Thanks,

    Prashanth Are

    MS Fabric Community Support

     

    If this post helps, then please consider Accept it as the solution to help the other members find it more quickly and give Kudos if helped you resolve your query.

  • v-prasare's avatar
    v-prasare
    Community Support
    1 year ago

    We are following up once again regarding your query. Could you please confirm if the issue has been resolved through the support ticket with Microsoft?

    If the issue has been resolved, we kindly request you to share the resolution or key insights here to help others in the community. If we don’t hear back, we’ll go ahead and close this thread.

    Should you need further assistance in the future, we encourage you to reach out via the Microsoft Fabric Community Forum and create a new thread. We’ll be happy to help.

     

    Thank you for your understanding and participation.

    • dpollozhani's avatar
      dpollozhani
      Helper I
      1 year ago

      I did not file a support ticket for this, but I believe the issue has been resolved by the subsequent updates/patches from the team. Unless my D365 source stopped producing legacy timestamps, it works if I run this now:

       

      • v-prasare's avatar
        v-prasare
        Community Support
        1 year ago

        Hi dpollozhani,

        Thanks for the confirmation that your issue got resolved. please accept useful post as answer by doing so, this will help other users find similar asnwers quickly. If we don't hear back from you we will go ahead and close this thread. please raise new thread, we will be happy to help you.

         

         

         

        Thanks,

        Prashanth Are

        MS Fabric community support