Forum Discussion
Super UserHigh Concurrency session in Data pipeline
Took 7 notebooks here , attached to same lakehouse ,env and present in same workspace.
In my case it should ideally create2 sessions 1 session (for 5 notebooks) another for ( 2 notebooks) according to the document.
https://learn.microsoft.com/en-us/fabric/data-engineering/configure-high-concurrency-session-notebooks-in-pipelines
But in my case it has created 4 sessions (1 for 4 notebooks , 1 session for remaining 3 notebooks). which is strange. But grouping and reducing time in execution is working as expected.
Super UserHi ObungiNiels ,
Thanks for your quick response. Yeah what i have understood is as per documentation, per high concurrency session can hold 5 notebooks. What I wonder is suppose if i have 6 notebooks which is attached to lakehouse1 then HC1 will be attached to only 5 notebooks and other one will be attached to HC2 right?
Regards,
Srisakthi
Resolver IIIHi Srisakthi , yes this is what happened to me. Here a screenshot from my Monitor of a pipeline in which I ran a notebook 11 times:
As you can see from the HC session id attached to the activity name, 5 notebooks were assigned to HC session starting with aa1a, another 5 to 0b99 and the final notebook ran under session f222.
So it behaves in a way that high concurrency session are being filled up to the limit of 5 and new session are started when necessary. 🙂