topic Re: Legacy Timestamp Rebase Support in Native Execution Engine 1.3 in Data Engineering

Legacy Timestamp Rebase Support in Native Execution Engine 1.3

dpollozhani — Tue, 28 Jan 2025 11:25:55 GMT

I'm working on a notebook with shortcuts to Dynamics 365 data stored in a ADLS gen2 container (as a result of Synapse Link + Spark). It seems the parquet files written by the Synapse Link contain legacy timestamps - at least that's how I interpret this.

Anyhow, I'm trying out the Native Execution Engine for Fabric 1.3, and according to this link, I should just be able to enable the timestamp rebase for Gluten. But even though I've done this in my Environment and restarted my Spark sessions in Fabric, and subsequently tried adding `SET spark.gluten.legacy.timestamp.rebase.enabled = true;` directly to the notebook in question - it doesn't work.

I'm still getting the error:
`Error Source: USER`
`Error Code: UNSUPPORTED`
`Reason: Reading legacy timestamp is not supported.`

Anyone have any ideas here?

Re: Legacy Timestamp Rebase Support in Native Execution Engine 1.3

nilendraFabric — Tue, 28 Jan 2025 12:36:04 GMT

Hello @dpollozhani

try these as well

spark.sql.legacy.parquet.int96RebaseModeInRead=CORRECTED
spark.sql.legacy.parquet.int96RebaseModeInWrite=CORRECTED
spark.sql.legacy.parquet.datetimeRebaseModeInRead=CORRECTED
spark.sql.legacy.parquet.datetimeRebaseModeInWrite=CORRECTED

Please let me know if this works

Thanks

Re: Legacy Timestamp Rebase Support in Native Execution Engine 1.3

Anonymous — Wed, 29 Jan 2025 12:08:38 GMT

Hi @dpollozhani ,
Thank you for the helpful response @nilendraFabric .
In addition to Niledra's suggestion, I kindly request you to update to the latest versions of Fabric Runtime and Spark. These versions include support for the native execution engine and the necessary timestamp handling features.

If this post helps, please give us Kudos and consider Accept it as a solution to help the other members find it more quickly.
Thank you for being a aprt of Microsoft Fabric Community Forum!
Regards,
Pallavi.

Re: Legacy Timestamp Rebase Support in Native Execution Engine 1.3

dpollozhani — Wed, 29 Jan 2025 12:14:04 GMT

Here is my current configuration in the Environment:

runtime_version: '1.3' spark_conf: - spark.sql.session.timeZone: Europe/Copenhagen - spark.gluten.legacy.timestamp.rebase.enabled: 'true' - spark.sql.legacy.parquet.datetimeRebaseModeInRead: CORRECTED - spark.sql.legacy.parquet.datetimeRebaseModeInWrite: CORRECTED - spark.sql.legacy.parquet.int96RebaseModeInRead: CORRECTED - spark.sql.legacy.parquet.int96RebaseModeInWrite: CORRECTED

I should note that the following

- spark.gluten.legacy.timestamp.rebase.enabled: 'true' - spark.sql.legacy.parquet.int96RebaseModeInRead - spark.sql.legacy.parquet.int96RebaseModeInWrite

are not recognized by the Environment UI - if that has any implication at all.

I also tried with 'LEGACY' instead of 'CORRECTED' on all the settings.

Still, the same result:

Caused by: java.lang.RuntimeException: Exception: VeloxUserError Error Source: USER Error Code: UNSUPPORTED Reason: Reading legacy timestamp is not supported.

Re: Legacy Timestamp Rebase Support in Native Execution Engine 1.3

dpollozhani — Wed, 29 Jan 2025 12:18:35 GMT

I'm already using the latest Fabric runtime, 1.3. According to the documentation (blog) the timestamp handling features are not yet set by default, which is also evident in my examples.

Re: Legacy Timestamp Rebase Support in Native Execution Engine 1.3

nilendraFabric — Wed, 29 Jan 2025 12:59:00 GMT

%%configure
{"conf": {"spark.gluten.legacy.timestamp.rebase.enabled": "true"}}

Give it a try

Re: Legacy Timestamp Rebase Support in Native Execution Engine 1.3

dpollozhani — Wed, 29 Jan 2025 14:11:49 GMT

Nope, unfortunately, still same issue.

Re: Legacy Timestamp Rebase Support in Native Execution Engine 1.3

Anonymous — Wed, 29 Jan 2025 21:53:08 GMT

I recently encountered the same issue. In my scenario I tried multiple times with setting the configuration for the spark session but I still got the legacy timestamp not supported error. But then I found out that my spark session was attached to a specific environment which overrulled my spark session configurations.

So please be aware of this. In short, if you do not run environment and make the following configuration inside the spark session then it will work. The configurations are:

spark.conf.set("spark.sql.parquet.int96RebaseModeInRead", "CORRECTED")

spark.conf.set("spark.sql.parquet.int96RebaseModeInWrite", "CORRECTED")

spark.conf.set("spark.sql.parquet.datetimeRebaseModeInRead", "CORRECTED")

spark.conf.set("spark.sql.parquet.datetimeRebaseModeInWrite", "CORRECTED")

spark.conf.set("spark.sql.legacy.parquet.datetimeRebaseModeInRead", "CORRECTED")

spark.conf.set("spark.sql.legacy.parquet.datetimeRebaseModeInWrite", "CORRECTED")

spark.conf.set("spark.sql.legacy.timeParserPolicy", "CORRECTED"),

spark.conf.set("spark.sql.legacy.parquet.int96RebaseModeInRead", "CORRECTED")

spark.conf.set("spark.sql.legacy.parquet.int96RebaseModeInWrite", "CORRECTED")

Re: Legacy Timestamp Rebase Support in Native Execution Engine 1.3

dpollozhani — Wed, 29 Jan 2025 22:41:00 GMT

As I stated in previous comments, I am using an Environment, but that's where I'm primarily trying to set the configuration (which is what you're supposed to be able to). Secondly, I tried overriding the presets from the Environment by making a forced configuration in the notebook. None of it works.

My guess is that sessions that are attached to Environments are currently not working as intended, as seems to be the general rule with Environments until now (very unstable). Perhaps some settings are not applied at all and ignored even when set in the notebook. It could be a case where you have to force an Environment rebuild by switching runtimes back and forth (which I've had to do when I encountered issues with custom library publishing).

Next I will have to try to do the rebuild, and after that try to run Native in a notebook without an Environment.

Re: Legacy Timestamp Rebase Support in Native Execution Engine 1.3

dpollozhani — Thu, 30 Jan 2025 09:39:59 GMT

So running native execution engine in a notebook that is not attached to an Environment, with the configurations as per above, does not work:

Changing everything from "CORRECTED" to "LEGACY" has no effect either:

Don't know if it has anything to do with the fact that the data is accessed via shortcuts (it shouldn't have), or if it's just generally buggy - but I think I might just give up on this for now.

Re: Legacy Timestamp Rebase Support in Native Execution Engine 1.3

v-prasare — Thu, 06 Feb 2025 03:08:22 GMT

@dpollozhani, please do raise a support ticket this will allow the a dedicated team to investigate and provide a resolution. Please include details of your setup, test results, and the impact on your workflows when submitting the ticket.

How to create a Fabric and Power BI Support ticket - Power BI | Microsoft Learn

Thanks,

Prashanth Are

MS Fabric Community Support

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly and give Kudos if helped you resolve your query.

Re: Legacy Timestamp Rebase Support in Native Execution Engine 1.3

v-prasare — Mon, 16 Jun 2025 04:41:00 GMT

We are following up once again regarding your query. Could you please confirm if the issue has been resolved through the support ticket with Microsoft?

If the issue has been resolved, we kindly request you to share the resolution or key insights here to help others in the community. If we don’t hear back, we’ll go ahead and close this thread.

Should you need further assistance in the future, we encourage you to reach out via the Microsoft Fabric Community Forum and create a new thread. We’ll be happy to help.

Thank you for your understanding and participation.

Re: Legacy Timestamp Rebase Support in Native Execution Engine 1.3

dpollozhani — Mon, 16 Jun 2025 07:10:26 GMT

I did not file a support ticket for this, but I believe the issue has been resolved by the subsequent updates/patches from the team. Unless my D365 source stopped producing legacy timestamps, it works if I run this now:

Re: Legacy Timestamp Rebase Support in Native Execution Engine 1.3

v-prasare — Fri, 27 Jun 2025 09:47:27 GMT

Hi @dpollozhani,

Thanks for the confirmation that your issue got resolved. please accept useful post as answer by doing so, this will help other users find similar asnwers quickly. If we don't hear back from you we will go ahead and close this thread. please raise new thread, we will be happy to help you.

Thanks,

Prashanth Are

MS Fabric community support