Parallel Execution

Hi DennesTorres,

High concurrency mode in Spark allows you to share the Spark Compute to execute notebooks in an instant fashion instead of waiting to spin up new compute for each notebook. The runMultiple run uses the compute engine's multi-threading to run the different notebooks. In a way, both of them help achieve high concurrency using the same compute power.

High concurrency mode that you enable while using Spark is more for interactive approach at this moment. So let's say you are parallely working in many notebooks and you want to share the same compute so that you can start instantly as well as save costs, high concurrency mode is the way to go. For doing the same in code, you are using runMultiple which uses the same approach. 

It is true that pipelines don't do concurrent execution using the same session. But it is in the roadmap: https://learn.microsoft.com/en-us/fabric/release-plan/data-engineering#concurrency

So for now, as you mentioned, we can use pipeline to run a notebook that uses runMultiple to run different notebooks to achieve concurrency.

Hi,

I got an explanation from the support about the never ending session. 

According to them, the runmultiple DAG should never be used with more than 50 activities. Strange things could happen if we try, such as the session never ending.

Kind Regards,

Dennes