High Concurrency session in Data pipeline

[Srisakthi]

Hello Everyone,

Consider a scenario where I have 10 notebooks and a datapipeline which uses session tag to call all those notebooks, in this case how many high concurrency session will be created ?

Regards,

Srisakthi

[Srisakthi]

Thank you @ObungiNiels  @MonicaCVL  for your valuable insights. 

If we have more than 10 notebooks then it is aligining 5 notebooks per HC.

Regards,

Srisakthi

[MonicaCVL]

Hi @Srisakthi 
I have tried 2 scenarios ,

Use Case 1: Parallel Execution

• With High Concurrency (HC): 20 mins 40 secs, 9985 CUs.
• Without HC: 29 mins 35 secs, 10809.3 CUs.
• Savings: 31% time, 823.6 CUs.

Use Case 2: Sequential Execution

• With HC: 20 mins 40 secs, 12093.67 CUs.
• Without HC: 27 mins 36 secs, 10864.16 CUs.
• Savings: 31% time, but higher CU consumption.

Conclusion: HC saves time in both cases. For cost efficiency, HC is better in parallel execution but not in sequential execution. Choose HC if speed is a priority.

[Srisakthi]

@MonicaCVL ,

Any idea how many notebooks are getting attached per HC for your use case?

Regards,

Srisakthi

[MonicaCVL · Accepted solution]

@Srisakthi ,

Took 7 notebooks here , attached to same lakehouse ,env and present in same workspace.

In my case it should ideally create2 sessions 1 session (for 5 notebooks) another for ( 2 notebooks) according to the document.
https://learn.microsoft.com/en-us/fabric/data-engineering/configure-high-concurrency-session-noteboo...

But in my case it has created 4 sessions (1 for 4 notebooks , 1 session for remaining 3 notebooks). which is strange. But grouping and reducing time in execution is working as expected.

[v-nmadadi-msft]

Hi @Srisakthi ,

As we haven’t heard back from you, we wanted to kindly follow up to check if the solution provided by the community members for the issue worked. If our response addressed, please mark it as Accept as solution and click Yes if you found it helpful.

Thanks and regards

[v-nmadadi-msft]

Hi @Srisakthi ,

I wanted to check if you had the opportunity to review the information provided. Please feel free to contact us if you have any further questions. If our responses has addressed your query, please accept it as a solution and give a 'Kudos' so other members can easily find it.

Thank you.

[v-nmadadi-msft]

Hi @Srisakthi ,

May I ask if you have resolved this issue? If so, please mark the helpful reply and accept it as the solution. This will be helpful for other community members who have similar problems to solve it faster.

Thank you.

[Srisakthi]

Hi @ObungiNiels ,

Thanks for your quick response. Yeah what i have understood is as per documentation, per high concurrency session can hold 5 notebooks. What I wonder is suppose if i have 6 notebooks which is attached to lakehouse1 then HC1 will be attached to only 5 notebooks and other one will be attached to HC2 right?

Regards,

Srisakthi

[ObungiNiels]

Hi @Srisakthi , yes this is what happened to me. Here a screenshot from my Monitor of a pipeline in which I ran a notebook 11 times:

As you can see from the HC session id attached to the activity name, 5 notebooks were assigned to HC session starting with aa1a, another 5 to 0b99 and the final notebook ran under session f222. 

So it behaves in a way that high concurrency session are being filled up to the limit of 5 and new session are started when necessary. 🙂 

[ObungiNiels]

Adding on this, the number of parallel sessions is contrained by what can be achieved with the capacity in place. When running a small capacity (F2 or F4), it is possible that only one HC concurrency session will be started at the same time while all other notebooks are being queued until the first session resolves. 

[ObungiNiels]

Hi @Srisakthi ,

this is an interesting question! I just tried this out and for me, two high concurrency sessions were started to process 10 notebooks running in parallel from a data pipeline. When I added an 11th notebook, a third HC session was started.

This observation alligns with the upper limit of 5 notebooks which can be connected to a single HC session when started up manually. 

Hope this helps! 

Kind regards,

Niels