Forum Discussion
Resolver IIISeeking best practices and scenarios suitable for using Session tags for HC for notebooks in
Hi Team,
We used session tags in our data pipelines which has HC notebooks. That pipeline always failed with error 'java.lang.AssertionError: assertion failed: Byte array does not have correct length'. The stage was always tried 4 times and it failed the 5th time.
Need to understand the best practices or the scenarios when the session tags for High Concurreny notebooks should be used and how many concurrent sessions can be supported for parallel runs.
Please feel free to ask more questions on the above ask if the questions is not clear. Would want to discuss every aspect of the above functionality.
Thanks
Ati Puri
Session tags in high concurrency clusters and notebooks are not a general purpose parallelisation feature. They are a resource routing and session affinity mechanism. If used incorrectly inside orchestrated pipelines, they often trigger unstable interpreter states, which is why you are hitting the AssertionError: Byte array does not have correct length -> it is a known pattern when too many tasks attempt to bind to the same interpreter pool.
Below are the best practices and realworld scenarios, I have tried to summarise, where session tags work well and where they should not be used.
When you should use session tags (valid scenarios)
Ensures that specific users or groups consistently land on the same interpreter session for caching, library loading, or maintaining state.
When multiple interactive users run small queries simultaneously and you want cleaner separation (for billing or debugging).
Routing workloads with specialised dependencies : Example: Some notebooks require heavy ML libraries while others do not. Tags ensure those workloads attach to interpreters pre-warmed with those libraries.
Tags prevent noisy-neighbour behaviour on HC clusters for interactive usage.
These are designed for interactive or semiinteractive use, not pipeline heavy parallel orchestration.
When you should NOT use session tags?
If you have multiple pipeline activities spawning notebooks with the same tag, they all fight for the same interpreter slot. Result: retries, corruption, serialization errors, and your exact error.
Heavy ETL or long-running batches : HC mode is not meant for this.
Tasks that must be fully isolated (per-run state) : HC + session tags do not give strict isolation.
Highly parallel fan-out workloads : HC limits concurrency because it relies on a shared JVM and interpreter pool.
How many concurrent sessions does HC actually support?
This is the part mostly misunderstand imo.
HC clusters typically support 8 to 20 concurrent notebook sessions depending on node size.
But tags reduce that headroom further because sessions with the same tag bind to a single interpreter route.
If your pipeline triggers >3 - 5 parallel activities with the same tag, failures are common.
Your pipeline failing after the 5th retry is classic “session pool exhaustion + corrupted state”
Recommended approach for pipelines
Do not use HC mode for orchestrated pipelines unless absolutely required.
Remove session tags for pipeline-triggered notebooks.
Scale cluster based on workload, not sessions.
If strict isolation is needed, use standard or single-user mode.
4 Replies
- nielsvdc
Super User
Hi ati_puri, in Fabric pipelines, it's recommended to enable high concurrency (HC) for notebooks when your pipeline triggers multiple notebooks simultaneously, especially if those notebooks perform lightweight transformations or quick data preparation tasks. A single high-concurrency session can share resources across a maximum of 5 notebooks identified by the same session tag. When more than 5 notebooks are submitted with the same tag, the system will automatically create a new high-concurrency session to host the subsequent notebook steps. More information can be found here Configure high concurrency mode for Fabric notebooks in pipelines.
However, avoid using HC for notebooks that involve heavy memory usage, intensive compute operations, or long-running jobs. But keep in mind that the assigned Fabric capacity restricts the number of concurrent non-HC notebook sessions you can run. justblindbaek is explaining these sometimes complicated to understand limits here Running Multiple Spark Notebooks on the Same Capacity – justB smart.Hope this helps. If so, please give a Kudos 👍 or mark as Accepted Solution ✔️.
- Vinodh247
Session tags in high concurrency clusters and notebooks are not a general purpose parallelisation feature. They are a resource routing and session affinity mechanism. If used incorrectly inside orchestrated pipelines, they often trigger unstable interpreter states, which is why you are hitting the AssertionError: Byte array does not have correct length -> it is a known pattern when too many tasks attempt to bind to the same interpreter pool.
Below are the best practices and realworld scenarios, I have tried to summarise, where session tags work well and where they should not be used.
When you should use session tags (valid scenarios)
Ensures that specific users or groups consistently land on the same interpreter session for caching, library loading, or maintaining state.
When multiple interactive users run small queries simultaneously and you want cleaner separation (for billing or debugging).
Routing workloads with specialised dependencies : Example: Some notebooks require heavy ML libraries while others do not. Tags ensure those workloads attach to interpreters pre-warmed with those libraries.
Tags prevent noisy-neighbour behaviour on HC clusters for interactive usage.
These are designed for interactive or semiinteractive use, not pipeline heavy parallel orchestration.
When you should NOT use session tags?
If you have multiple pipeline activities spawning notebooks with the same tag, they all fight for the same interpreter slot. Result: retries, corruption, serialization errors, and your exact error.
Heavy ETL or long-running batches : HC mode is not meant for this.
Tasks that must be fully isolated (per-run state) : HC + session tags do not give strict isolation.
Highly parallel fan-out workloads : HC limits concurrency because it relies on a shared JVM and interpreter pool.
How many concurrent sessions does HC actually support?
This is the part mostly misunderstand imo.
HC clusters typically support 8 to 20 concurrent notebook sessions depending on node size.
But tags reduce that headroom further because sessions with the same tag bind to a single interpreter route.
If your pipeline triggers >3 - 5 parallel activities with the same tag, failures are common.
Your pipeline failing after the 5th retry is classic “session pool exhaustion + corrupted state”
Recommended approach for pipelines
Do not use HC mode for orchestrated pipelines unless absolutely required.
Remove session tags for pipeline-triggered notebooks.
Scale cluster based on workload, not sessions.
If strict isolation is needed, use standard or single-user mode.
v-karpurapudCommunity Support
Hi ati_puri
Thank you for submitting your question to the Microsoft Fabric Community Forum, and thanks to nielsvdc and Vinodh247 for offering helpful suggestions.
Could you let us know if the suggested solution resolved your issue? If it's still pending, please let us know any further details so we can help.
Best regards,
Community Support Team
