topic Re: Parquet with timestamp prior 1900- Spark conf in Data Engineering

Parquet with timestamp prior 1900- Spark conf

Mathan1787 — Wed, 09 Aug 2023 07:16:53 GMT

Hi,

I am trying to copying data from azure sql database to lakehouse. I have used copy activity to move the data into parquet files but when i try to open them in notebook, I am getting an error message.

I have to add below code.

spark.sql.legacy.parquet.int96RebaseModeInRead=CORRECTED

spark.sql.legacy.parquet.int96RebaseModeInWrite=CORRECTED

I have tried to set this in spark conf in workspace setting to have it set for my entire workspace but i cant find them in drop down list so i have to manually add it in all the notebook. could you help me here. thanks

Re: Parquet with timestamp prior 1900- Spark conf

DennesTorres — Wed, 09 Aug 2023 13:30:12 GMT

Hi,

If this works but you can't set for the entire workspace because it's not available in the dropdown, should you register this as an issue, asking for correction?

You can using this link: https://community.fabric.microsoft.com/t5/Issues/idb-p/Issues

Kind Regards,

Dennes

Re: Parquet with timestamp prior 1900- Spark conf

DennesTorres — Wed, 09 Aug 2023 19:53:15 GMT

Hi,

Let me fix this previous reply:

The drop down are open, you can add configurations even if they are not present in the dropdown. If they are wrong, the spark session will not start and will result in an error message.

Kind Regards,

Dennes