Forum Discussion
High Concurrency session in Data pipeline
Took 7 notebooks here , attached to same lakehouse ,env and present in same workspace.
In my case it should ideally create2 sessions 1 session (for 5 notebooks) another for ( 2 notebooks) according to the document.
https://learn.microsoft.com/en-us/fabric/data-engineering/configure-high-concurrency-session-notebooks-in-pipelines
But in my case it has created 4 sessions (1 for 4 notebooks , 1 session for remaining 3 notebooks). which is strange. But grouping and reducing time in execution is working as expected.
Hi Srisakthi
I have tried 2 scenarios ,
Use Case 1: Parallel Execution
- With High Concurrency (HC): 20 mins 40 secs, 9985 CUs.
- Without HC: 29 mins 35 secs, 10809.3 CUs.
- Savings: 31% time, 823.6 CUs.
Use Case 2: Sequential Execution
- With HC: 20 mins 40 secs, 12093.67 CUs.
- Without HC: 27 mins 36 secs, 10864.16 CUs.
- Savings: 31% time, but higher CU consumption.
Conclusion: HC saves time in both cases. For cost efficiency, HC is better in parallel execution but not in sequential execution. Choose HC if speed is a priority.
Anonymous ,
Any idea how many notebooks are getting attached per HC for your use case?
Regards,
Srisakthi
Took 7 notebooks here , attached to same lakehouse ,env and present in same workspace.
In my case it should ideally create2 sessions 1 session (for 5 notebooks) another for ( 2 notebooks) according to the document.
https://learn.microsoft.com/en-us/fabric/data-engineering/configure-high-concurrency-session-notebooks-in-pipelines
But in my case it has created 4 sessions (1 for 4 notebooks , 1 session for remaining 3 notebooks). which is strange. But grouping and reducing time in execution is working as expected.