DateTime timezone changes while copying to .parquet

[Noeleke1301]

I've set up a data pipeline that copies data from our oracle database to a Fabric lakehouse. However, when using .parquet I can see times being changed (likely due to daylight savings), f.e. 2024-03-29 00:00 to 2024-03-28 22:00

When I use a .json extension, the time stays the same as in my source.

How can I tell my pipeline to not change the time when using .parquet as a sink?

As I'm copying all tables dynamically, there is no way for me to set up mapping.

[v-cboorla-msft]

Hi @Noeleke1301 

Thanks for using Microsoft Fabric Community.
At this time, we are reaching out to the internal team to get some help on this.
We will update you once we hear back from them.

Appreciate your patience.

Thanks

[Noeleke1301 · Accepted solution]

Hi,

I think I figured out the problem. My source system doesn't have any timezone specification in their datetime fields. I've read that parquet self-assigns a timezone to the value if the source doesn't give any.

Since the Fabric capacity is probably in another timezone, it changes the default timezone of my .parquet files.

I've had to set the correct timezone in my spark notebook to correct this. More info on:

Spark: queries with datetime and time zones - KeesTalksTech

[v-cboorla-msft]

Hi @Noeleke1301 

Glad that you were able to find some insights and thank you for sharing the same with the community as it can be helpful to others.

Please continue using Fabric Community for further queries.

Thanks