Forum Discussion

smpa01's avatar
smpa01
Community Champion
2 years ago
Solved

Stopping Spark Session inside/outside ForEach

Usually, spark.stop() is recommended as a best practice to releasing resources like memory, CPU, and network connections back to the cluster and for other reasons. I usually do that for all my notebo...
pipelines
frithjof_v's avatar
frithjof_v
Community Champion
2 years ago

I am guessing you don't need to use spark.stop() in Fabric.

 

Are you starting the Spark session also by using code? Something like this:

 

from pyspark.sql import SparkSession

spark = SparkSession.builder \

    .appName("Simple DataFrame Example") \

    .getOrCreate()

 

I don't think that is necessary in Fabric also. I guess sessions are managed by Fabric. When running a notebook interactively (using the notebook editor interface) I guess it's a good idea to click 'Stop session' when finished. However when running a Notebook in a Data pipeline, I think Fabric manages the session and stops the session when it's not needed anymore. Ref. the blog post link in my previous comment.

 

I am also guessing that using spark.stop() inside the Notebook can make you unable to take advantage of high concurrency spark sessions.

 

However I'm not sure about any of this, as I don't have enough knowledge or experience with this.

 

Hoping to get others' insights and thoughts on this 😃

 

frithjof_v's avatar
frithjof_v
Community Champion
2 years ago

I started a discussion on Reddit to try to learn more about the topic: 

 

spark.stop() - is it needed? : r/MicrosoftFabric (reddit.com)

 

I also noticed there is an alterantive to spark.stop(), which is mssparkutils.session.stop()

 

Anyway, I'm not sure if it's necessary.

 

I am still not entirely sure what to believe regarding session start/stop in Fabric.

 

There is also the option to use a Master notebook and use that notebook to call other notebooks. Then I think you can share the same session among notebooks. I think this approach utilizes the high concurrency feature.

 

EDIT: I think the Reddit discussion has made me understand more about it. I recommend checking out the Reddit discussion (link above).

  • smpa01's avatar
    smpa01
    Community Champion
    2 years ago

    frithjof_v  thanks for this.

    High concurrency is not shipped yet(off-topic)

     

    Therefore, if measures are not taken for a large_array,  the pipeline will error out if you are calling a notebook inside forEach for a large array to perform operation on the same table. E.g.

     

    //pseudo code 
const large_array = [1,2,...20]

//updates to be utilize in upsert
const updates = updates

//target
const target = delta_fact

//forEach activity in pipeline sequntial execution on a subset of target
forEach eleemnt of large_array {
perform Delta Table Merge sequentially
      where each large_array[element] = target[element]

}