Forum Discussion

dpollozhani's avatar
dpollozhani
Helper I
1 year ago
Solved

Legacy Timestamp Rebase Support in Native Execution Engine 1.3

I'm working on a notebook with shortcuts to Dynamics 365 data stored in a ADLS gen2 container (as a result of Synapse Link + Spark). It seems the parquet files written by the Synapse Link contain leg...
data engineering
environment
notebook
  • dpollozhani's avatar
    dpollozhani
    1 year ago

    I did not file a support ticket for this, but I believe the issue has been resolved by the subsequent updates/patches from the team. Unless my D365 source stopped producing legacy timestamps, it works if I run this now:

     

Anonymous's avatar
Anonymous
Not applicable
1 year ago

I recently encountered the same issue. In my scenario I tried multiple times with setting the configuration for the spark session but I still got the legacy timestamp not supported error. But then I found out that my spark session was attached to a specific environment which overrulled my spark session configurations. 

 

So please be aware of this. In short, if you do not run environment and make the following configuration inside the spark session then it will work. The configurations are:

spark.conf.set("spark.sql.parquet.int96RebaseModeInRead", "CORRECTED")
spark.conf.set("spark.sql.parquet.int96RebaseModeInWrite", "CORRECTED")
spark.conf.set("spark.sql.parquet.datetimeRebaseModeInRead", "CORRECTED")
spark.conf.set("spark.sql.parquet.datetimeRebaseModeInWrite", "CORRECTED")
spark.conf.set("spark.sql.legacy.parquet.datetimeRebaseModeInRead", "CORRECTED")
spark.conf.set("spark.sql.legacy.parquet.datetimeRebaseModeInWrite", "CORRECTED")
spark.conf.set("spark.sql.legacy.timeParserPolicy", "CORRECTED"),
spark.conf.set("spark.sql.legacy.parquet.int96RebaseModeInRead", "CORRECTED")
spark.conf.set("spark.sql.legacy.parquet.int96RebaseModeInWrite", "CORRECTED")
dpollozhani's avatar
dpollozhani
Helper I
1 year ago

As I stated in previous comments, I am using an Environment, but that's where I'm primarily trying to set the configuration (which is what you're supposed to be able to). Secondly, I tried overriding the presets from the Environment by making a forced configuration in the notebook. None of it works.

 

My guess is that sessions that are attached to Environments are currently not working as intended, as seems to be the general rule with Environments until now (very unstable). Perhaps some settings are not applied at all and ignored even when set in the notebook. It could be a case where you have to force an Environment rebuild by switching runtimes back and forth (which I've had to do when I encountered issues with custom library publishing). 

Next I will have to try to do the rebuild, and after that try to run Native in a notebook without an Environment.

  • dpollozhani's avatar
    dpollozhani
    Helper I
    1 year ago

    So running native execution engine in a notebook that is not attached to an Environment, with the configurations as per above, does not work:

     

    Changing everything from "CORRECTED" to "LEGACY" has no effect either:

     

    Don't know if it has anything to do with the fact that the data is accessed via shortcuts (it shouldn't have), or if it's just generally buggy - but I think I might just give up on this for now.