Skip to main content
cancel
Showing results for 
Search instead for 
Did you mean: 

Be one of the first to start using Fabric Databases. View on-demand sessions with database experts and the Microsoft product team to learn just how easy it is to get started. Watch now

Reply
shietpas_cbts
Frequent Visitor

mssparkutils.notebook.run not running notebooks in isolated session context

When calling multiple notebooks using the mssparkutils, they do not run in thread-safe manner and will share session context veriables. This is true if you call two notebooks sequentially using mssparkutils.notebook.run sequentally in a notebook or if you call in parallel using mssparkutils.notebook.runMultiple(). In either case, values set in session using 

spark.conf.set will be visible by other other notebooks. And if running in parallel, the notebooks may interfere with each other if using the same variable names.
 
Temporary Views are also shared across their sessions, which could also create issues.
 
DBUtils runs notebooks in isoluation and I believe mssparkutils should also run in isolation.
1 ACCEPTED SOLUTION
Anonymous
Not applicable

Hi @shietpas_cbts ,

Apologies for the delay in response, we have an update from the internal team -

spark.conf.set will be share in child notebooks, but local variable in the notebook will not be shared by other notebooks.



Hope this is helpful. Please let me know incase of further queries.

View solution in original post

4 REPLIES 4
Anonymous
Not applicable

Hi @shietpas_cbts ,

Thanks for using Fabric Community.
At this time, we are reaching out to the internal team to get some help on this .
We will update you once we hear back from them.

Anonymous
Not applicable

Hi @shietpas_cbts ,

Apologies for the delay in response, we have an update from the internal team -

spark.conf.set will be share in child notebooks, but local variable in the notebook will not be shared by other notebooks.



Hope this is helpful. Please let me know incase of further queries.

Anonymous
Not applicable

Glad to know that your query got resolved. Please continue using Fabric Community on your further queries.

Anonymous
Not applicable

Hi  @shietpas_cbts 

The Issues forum is for posts dealing with Power BI related errors. I have moved your post to the corresponding forum. Thank you for your cooperation.

 

Best Regards,
Community Support Team _ Ailsa Tao

Helpful resources

Announcements
Las Vegas 2025

Join us at the Microsoft Fabric Community Conference

March 31 - April 2, 2025, in Las Vegas, Nevada. Use code MSCUST for a $150 discount!

Dec Fabric Community Survey

We want your feedback!

Your insights matter. That’s why we created a quick survey to learn about your experience finding answers to technical questions.

ArunFabCon

Microsoft Fabric Community Conference 2025

Arun Ulag shares exciting details about the Microsoft Fabric Conference 2025, which will be held in Las Vegas, NV.

December 2024

A Year in Review - December 2024

Find out what content was popular in the Fabric community during 2024.