Forum Discussion
Helper IParquet with timestamp prior 1900- Spark conf
Hi,
I am trying to copying data from azure sql database to lakehouse. I have used copy activity to move the data into parquet files but when i try to open them in notebook, I am getting an error message.
I have to add below code.
spark.sql.legacy.parquet.int96RebaseModeInRead=CORRECTED
spark.sql.legacy.parquet.int96RebaseModeInWrite=CORRECTED
I have tried to set this in spark conf in workspace setting to have it set for my entire workspace but i cant find them in drop down list so i have to manually add it in all the notebook. could you help me here. thanks
Hi,
Let me fix this previous reply:
The drop down are open, you can add configurations even if they are not present in the dropdown. If they are wrong, the spark session will not start and will result in an error message.
Kind Regards,Dennes
2 Replies
DennesTorresPower Participant
Hi,
If this works but you can't set for the entire workspace because it's not available in the dropdown, should you register this as an issue, asking for correction?
You can using this link: https://community.fabric.microsoft.com/t5/Issues/idb-p/Issues
Kind Regards,Dennes
- DennesTorres
Hi,
Let me fix this previous reply:
The drop down are open, you can add configurations even if they are not present in the dropdown. If they are wrong, the spark session will not start and will result in an error message.
Kind Regards,Dennes