Fabric Notebook parallel execution through Data Pipeline, no isolation

[AntonisJ]

I have a Notebook that accepts parameters, the functions also populate my log table, 

when i run this notebook in parallel from a pipeline, say 4 times using different parameters, the logs show that parameters are overlapping each session. 

Question is, can the same Notebook run in an isolated session in parallel, when triggered multiple times from a Data Pipeline?

Thank you in advance

[Anonymous]

Hi @AntonisJ ,

Yes, the same notebook can be run in parallel in an isolated session when triggered multiple times from a data pipeline. You might consider configuring your pipeline to create a separate instance for each run to ensure that each notebook execution runs in a completely isolated environment.

Best Regards,

Ada Wang

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

[AntonisJ]

thanks @Anonymous 

how do i configure a pipeline to create a separate instance for each run?