Spark Settings not available for environment

[smoqt]

Several Spark notebooks are throwing an error and suggest I set the following configurations.

spark.sql.parquet.datetimeRebaseModeInWrite

spark.sql.parquet.datetimeRebaseModeInRead

I can do this from the notebook but would prefer to do it at the environment level.  However, these settings are not available in the Spark Settings dropdown for my environment.  I can (and did) set the following configurations at the environment level but they did not resolve my issue.

spark.sql.legacy.parquet.datetimeRebaseModeInWrite

spark.sql.legacy.parquet.datetimeRebaseModeInRead

Runtime information:

Runtime 1.3 (Spark 3.5, Delta 3.2)

Will these settings be made available soon?

[v-tsaipranay · Accepted solution]

Hi @smoqt ,

Thank you for reaching out to the Microsoft Fabric Community.

you’re correct that the properties spark.sql.parquet.datetimeRebaseModeInWrite and spark.sql.parquet.datetimeRebaseModeInRead are not currently available in the Spark Settings dropdown at the environment level in Microsoft Fabric (Runtime 1.3, Spark 3.5). At this time, Fabric environments do not support setting these specific Spark configurations globally via the UI or backend configuration.

While the legacy configurations (spark.sql.legacy.parquet.datetimeRebaseModeInWrite and spark.sql.legacy.parquet.datetimeRebaseModeInRead) are available in the environment settings, they do not apply the same behavior in Spark 3.5+. This is due to changes in how Spark handles datetime rebasing starting in Spark 3.x, where the non-legacy keys take precedence for Parquet write/read operations.

I am including image please refer for your understanding:

As a result, the only current workaround is to set these properties explicitly within each notebook using spark.conf.set() at the start of your code.

We understand the need for setting such properties at the environment level for broader consistency, and we recommend submitting this as a feature request on the Microsoft Fabric Ideas Forum. Feedback there helps guide prioritization for feature enhancements like this.

If this post helps, then please give us Kudos and consider Accept it as a solution to help the other members find it more quickly.

Thankyou.