High Concurrency in pipelines not working as intended

[tommy24pt]

Hi, 

So I have a Pipeline which has 3 parallel invoke pipeline activities to ingest data. Basically each of these invoked pipelines ingests data through a dataflow gen 2 and then I use a notebook to perform an addition to the tables ingested (its one single notebook but in 3 different pipelines). Then when the 3 invoked pipelines are done, another pipeline (lets call it "B") is invoked. This B pipeline has a for each activity that for each record from performs a cleaning task (through a notebook) and then invokes another pipeline for that has a notebook for testing according to some quality tests (the B pipeline invokes this last mentioned pipeline). Then there are other invoked pipelined in the main pipeline, but i think you get the idea.

The thing is: Between the first 3 parallel activities and the cleaning notebook, the high concurrency session ends and another one is automatically created, but still have to wait 5 minutes for it to be setup.  I dont get why the session ends (appearing as "Stopped Session Timed Out) when actually the default 20 minutes had not been reached

I should also say that I tried to deactivate 2 out of the 3 ingestions and it ended up joining both the first notebook and the cleaning notebook in a session. But when i tried it again later it didn't work that way.

In total there were 6 notebooks used in several places of the main pipeline (i mean that one notebook might be inside more than one pipeline invoked or even inside a loop). 

I know that the maximum number of notebooks to share resources is 5 but in this case it changes to another HC when only 1 notebook was 3 times. 

Hope it wasn't confusing. 

In case I need to give more information, feel free to ask!

Thanks in advance.

[v-bmanikante]

Hi @tommy24pt ,

We wanted to kindly follow up to check if the solution provided for the issue worked? or Let us know if you need any further assistance?

If our response addressed, please mark it as Accept as solution and click Yes if you found it helpful.

Regards,

B Manikanteswara Reddy

[v-bmanikante · Accepted solution]

Hi @tommy24pt ,

Thank you for reaching out to Microsoft Fabric Community Forum.

@nilendraFabric Thank you for your quick response.

You can try these steps:

>>>Keep transitions tight between activities
If there's a long pause (like more than a minute or two) between activities like ingestion and cleaning, Fabric might shut down the session early — even if the official timeout hasn't passed. To keep using the same session, try to make sure the next step starts quickly.

>>>Avoid calling the same notebook in parallel
Even if it’s the exact same notebook, running it in parallel (like in different branches or pipelines) is treated as separate sessions. That quickly eats up your High Concurrency (HC) limit. If you can, try combining the logic into one notebook and use parameters to handle different data one after the other - it’s more session-friendly.

>>> Try running steps one after another instead of all at once
Running tasks like data cleaning and quality checks in sequence (instead of in parallel) can help Fabric reuse the same session more easily. It might take a bit longer, but it can save time overall by avoiding session spin-ups and delays. Worth considering if smooth session reuse is a priority.

>>>Use the monitoring tools
You can check out Lakehouse sessions under the Monitoring section in your workspace. It shows when sessions start, get reused, or shut down. It’s really useful to compare this with your pipeline logs to spot delays or figure out why a session didn’t get reused.

please refer to the official documentation for more details. 
Configure high concurrency mode for notebooks in pipelines - Microsoft Fabric | Microsoft Learn

If this post helps, then please consider Accepting as solution to help the other members find it more quickly, don't forget to give a "Kudos" – I’d truly appreciate it!

Regards,
B Manikanteswara Reddy

[nilendraFabric]

Hi @tommy24pt 

All notebook activities that should share a session must use exactly the same session tag. Any typo or mismatch creates a new session

https://www.red-gate.com/simple-talk/business-intelligence/microsoft-fabric/high-concurrency-data-pi...

There is a practical limit to how many notebooks can run concurrently in a single session (often cited as 5). Exceeding this may force new sessions or queue notebooks, causing delays and possible timeouts.

If notebooks are triggered in different pipeline branches or at different times, session reuse is not guaranteed unless the session is still alive and the tags/settings match.
• ForEach loops and parallel activities can complicate session management, especially if the timing causes the session to idle and time out.