Forum Discussion

AntonisJ's avatar
AntonisJ
New Member
2 years ago

Fabric Notebook parallel execution through Data Pipeline, no isolation

I have a Notebook that accepts parameters, the functions also populate my log table, 

when i run this notebook in parallel from a pipeline, say 4 times using different parameters, the logs show that parameters are overlapping each session. 

 

Question is, can the same Notebook run in an isolated session in parallel, when triggered multiple times from a Data Pipeline?

 

Thank you in advance

Data Integration

2 Replies

  • Anonymous's avatar
    Anonymous
    Not applicable
    2 years ago

    Hi AntonisJ ,

    Yes, the same notebook can be run in parallel in an isolated session when triggered multiple times from a data pipeline. You might consider configuring your pipeline to create a separate instance for each run to ensure that each notebook execution runs in a completely isolated environment.

     

    Best Regards,

    Ada Wang

    If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

    • AntonisJ's avatar
      AntonisJ
      New Member
      2 years ago

      thanks Anonymous 

      how do i configure a pipeline to create a separate instance for each run?