Skip to main content
cancel
Showing results for 
Search instead for 
Did you mean: 

Get certified in Microsoft Fabric—for free! For a limited time, the Microsoft Fabric Community team will be offering free DP-600 exam vouchers. Prepare now

Reply
shietpas_cbts
Frequent Visitor

mssparkutils.notebook.run not running notebooks in isolated session context

When calling multiple notebooks using the mssparkutils, they do not run in thread-safe manner and will share session context veriables. This is true if you call two notebooks sequentially using mssparkutils.notebook.run sequentally in a notebook or if you call in parallel using mssparkutils.notebook.runMultiple(). In either case, values set in session using 

spark.conf.set will be visible by other other notebooks. And if running in parallel, the notebooks may interfere with each other if using the same variable names.
 
Temporary Views are also shared across their sessions, which could also create issues.
 
DBUtils runs notebooks in isoluation and I believe mssparkutils should also run in isolation.
1 ACCEPTED SOLUTION
Anonymous
Not applicable

Hi @shietpas_cbts ,

Apologies for the delay in response, we have an update from the internal team -

spark.conf.set will be share in child notebooks, but local variable in the notebook will not be shared by other notebooks.



Hope this is helpful. Please let me know incase of further queries.

View solution in original post

4 REPLIES 4
Anonymous
Not applicable

Hi @shietpas_cbts ,

Thanks for using Fabric Community.
At this time, we are reaching out to the internal team to get some help on this .
We will update you once we hear back from them.

Anonymous
Not applicable

Hi @shietpas_cbts ,

Apologies for the delay in response, we have an update from the internal team -

spark.conf.set will be share in child notebooks, but local variable in the notebook will not be shared by other notebooks.



Hope this is helpful. Please let me know incase of further queries.

Anonymous
Not applicable

Glad to know that your query got resolved. Please continue using Fabric Community on your further queries.

v-yetao1-msft
Community Support
Community Support

Hi  @shietpas_cbts 

The Issues forum is for posts dealing with Power BI related errors. I have moved your post to the corresponding forum. Thank you for your cooperation.

 

Best Regards,
Community Support Team _ Ailsa Tao

Helpful resources

Announcements
Oct Fabric Update Carousel

Fabric Monthly Update - October 2024

Check out the October 2024 Fabric update to learn about new features.

September Hackathon Carousel

Microsoft Fabric & AI Learning Hackathon

Learn from experts, get hands-on experience, and win awesome prizes.

October NL Carousel

Fabric Community Update - October 2024

Find out what's new and trending in the Fabric Community.