Forum Discussion
Spark Environment Properties not applied during runtime from Pipeline Notebook Activity
Hi jisaac
unlike Databricks, Microsoft Fabric doesn't currently support initialization scripts for Spark environments. In Databricks, these scripts let you apply settings automatically when a cluster starts, but Fabric doesn’t offer that feature yet.you can manually set the Spark configs at the top of each notebook to ensure the necessary settings (like handling pre-1900 timestamps) are always applied. Alternatively, if you disable High Concurrency mode in pipeline runs, each notebook will start a fresh Spark session and correctly pick up the environment settings.
Another option is to create a small helper notebook that contains all your spark.conf.set(...) lines and use %run at the top of your main notebooks. This way, you keep things consistent without repeating code everywhere.
If the issue still persists we recommend you to raise a support ticket.You can submit a ticket through the Microsoft Power BI Support Portal:
How to create a Fabric and Power BI Support ticket - Power BI | Microsoft Learn
Thank you.
Hi jisaac
Thank's for clarification jisaac.
The reason your Spark Environment properties are not applying when running the notebook from a pipeline is due to High Concurrency mode. In this mode, notebooks often reuse existing Spark sessions, and once a session starts without certain configurations, it continues using those initial settings even for subsequent notebook executions. This prevents the Spark Properties set in the Fabric UI from being applied dynamically during pipeline runs.
To resolve this:
- Disable High Concurrency mode for pipeline executions. This ensures that every pipeline-triggered notebook starts a new Spark session, correctly applying the Spark Environment settings.
- If High Concurrency mode must remain enabled, you’ll need to manually enforce these properties by explicitly setting them at the start of each notebook.
- Another alternative is to configure an initialization script in the Spark Environment, which ensures that the required properties are always applied when a session starts.
By following these steps, your Spark Environment settings will be consistently applied when executing notebooks through pipelines. Please let me know if you need further assistance.
If this solution helps, please consider giving us Kudos and accepting it as the solution so that it may assist other members in the community.
Thank you.
Can you describe what you meant by point 3, initialization scripts in the Spark Environment? Searching the internet I found no mention of this for Fabric. There is a mention of something similar for DataBricks, but would that apply to Fabric?
Hi jisaac
unlike Databricks, Microsoft Fabric doesn't currently support initialization scripts for Spark environments. In Databricks, these scripts let you apply settings automatically when a cluster starts, but Fabric doesn’t offer that feature yet.you can manually set the Spark configs at the top of each notebook to ensure the necessary settings (like handling pre-1900 timestamps) are always applied. Alternatively, if you disable High Concurrency mode in pipeline runs, each notebook will start a fresh Spark session and correctly pick up the environment settings.
Another option is to create a small helper notebook that contains all your spark.conf.set(...) lines and use %run at the top of your main notebooks. This way, you keep things consistent without repeating code everywhere.
If the issue still persists we recommend you to raise a support ticket.You can submit a ticket through the Microsoft Power BI Support Portal:
How to create a Fabric and Power BI Support ticket - Power BI | Microsoft Learn
Thank you.
