Stopping Spark Session inside/outside ForEach

frithjof_v  thanks for this.

High concurrency is not shipped yet(off-topic)

Therefore, if measures are not taken for a large_array,  the pipeline will error out if you are calling a notebook inside forEach for a large array to perform operation on the same table. E.g.

//pseudo code 
const large_array = [1,2,...20]

//updates to be utilize in upsert
const updates = updates

//target
const target = delta_fact

//forEach activity in pipeline sequntial execution on a subset of target
forEach eleemnt of large_array {
perform Delta Table Merge sequentially
      where each large_array[element] = target[element]

}