Forum Discussion
Super UserHigh Concurrency session in Data pipeline
Took 7 notebooks here , attached to same lakehouse ,env and present in same workspace.
In my case it should ideally create2 sessions 1 session (for 5 notebooks) another for ( 2 notebooks) according to the document.
https://learn.microsoft.com/en-us/fabric/data-engineering/configure-high-concurrency-session-notebooks-in-pipelines
But in my case it has created 4 sessions (1 for 4 notebooks , 1 session for remaining 3 notebooks). which is strange. But grouping and reducing time in execution is working as expected.
Resolver IIIHi Srisakthi ,
this is an interesting question! I just tried this out and for me, two high concurrency sessions were started to process 10 notebooks running in parallel from a data pipeline. When I added an 11th notebook, a third HC session was started.
This observation alligns with the upper limit of 5 notebooks which can be connected to a single HC session when started up manually.
Hope this helps!
Kind regards,
Niels