Forum Discussion
Spark Environment Properties not applied during runtime from Pipeline Notebook Activity
Hi jisaac
unlike Databricks, Microsoft Fabric doesn't currently support initialization scripts for Spark environments. In Databricks, these scripts let you apply settings automatically when a cluster starts, but Fabric doesn’t offer that feature yet.you can manually set the Spark configs at the top of each notebook to ensure the necessary settings (like handling pre-1900 timestamps) are always applied. Alternatively, if you disable High Concurrency mode in pipeline runs, each notebook will start a fresh Spark session and correctly pick up the environment settings.
Another option is to create a small helper notebook that contains all your spark.conf.set(...) lines and use %run at the top of your main notebooks. This way, you keep things consistent without repeating code everywhere.
If the issue still persists we recommend you to raise a support ticket.You can submit a ticket through the Microsoft Power BI Support Portal:
How to create a Fabric and Power BI Support ticket - Power BI | Microsoft Learn
Thank you.
Thanks for your reply, but this is what I am hoping to avoid.
The last line in my post says
> How do I ensure these properties are set without explicitly setting them at the top of my notebook every time?
I was not aware of the %%configure magic, but I was setting these configs at the top of my notebook already. I'm wanting to know why the "Spark Properties" tab exists if it does not apply to sessions run by pipelines.
Could you please check If high concurrency mode for pipelines is enabled in the workspace settings
Yes, it is enabled. nilendraFabric
So this is why spark properties are not attached while running through pipelines
When you run a notebook interactively, a new Spark session is typically created for that notebook. During this process, the Spark Properties defined in the Spark Properties tab (or Default Spark Environment) are applied to the session at initialization. This ensures that any custom configurations you set are available for your interactive session.
In high concurrency mode, notebooks executed via pipelines often share an existing Spark session rather than creating a new one. If a session is already running, it will use the configurations it was initialized with, and the Spark Properties from the environment will not be reapplied. This is an optimization to avoid the overhead of starting new sessions for every pipeline activity.
Please accept the answer if this is helpful
Two things that I observe that are inconsistent with what you've said and confuse me:
- Whether the session starts from the first or second notebook run, it should still be using the same default properties, no? Both notebooks are set to run with the same environment. Are you saying that high concurrency sessions always start with no environment or properties?
- The settings are clearly not carrying over from one notebook to another. Observe the screen shots of two notebooks run under one HC session.
First notebook run:
Second notebook run, settings are not changed from previous run:
The settings reverted in the second notebook, so they are clearly being reset between notebooks.