Forum Discussion

jisaac's avatar
jisaac
Helper II
1 year ago
Solved

Running multiple instances of the same Notebook concurrently

I have a pipeline that needs to runs two Notebook activities (A and B) before running the third Notebook activity (C).

Activities A and B both run the same Notebook but with different parameters: they're doing the same operation on different tables. After both of those tables have been operated on by the Notebooks, it is supposed to run Activity C which runs an operation on both of those tables.

 

The problem is that, despite being able to give it parameters, the Notebooks error when they are run at the same time. The second instance to run causes the first run to Fail with a strange error. Say the first run is A and the second is B by a few second delay. A fails after B starts. Clicking on A to view the Notebook snapshot pulls up two identical snapshots for B, with no trace of the A run left behind.

 

I get the same behavior with or without High Concurrency sessions setup.

 

Is there a way to run the same Notebook twice at the same time with different parameters?

data pipeline
notebook
  • Anonymous's avatar
    Anonymous
    1 year ago

    Hi jisaac ,

     

    I recommend using a single call to a notebook that calls mssparkutils.notebook.runMultiple() in order to execute multiple calls to the notebook in a single Spark session. This approach allows Spark to utilize its ability to efficiently perform concurrent tasks.

     

    For more detailed information, please refer to:

    Microsoft Spark Utilities (MSSparkUtils) for Fabric - Microsoft Fabric | Microsoft Learn

     

    Best Regards,
    Yang
    Community Support Team

     

    If there is any post helps, then please consider Accept it as the solution  to help the other members find it more quickly.
    If I misunderstand your needs or you still have problems on it, please feel free to let us know. Thanks a lot!

2 Replies