High Concurrency setting in Pipelines not working

[BMooreBUNN]

I have posted this on Reddit as well, but it seems that if I turn on the high concurrency sessions in pipelines features I get various erorrs (livy session error comes up a  lot but I've gotten other errors as well).

If I turn the setting off and remove the session tags everything starts working again, but then each notebook takes 2.5mins to spin up and drags out the total runtime (i have private link enabled). Is anyone else encountering this as well? I have ensured the notebooks are connecting to the same lakehouses and I'm not installing any additional libraries.

[v-pnaroju-msft · Accepted solution]

Hi BMooreBUNN,

The feature 'High Concurrency Mode for Notebooks in Pipelines' is currently in the preview stage. You may access further details regarding this feature through the link provided below. Additionally, a relevant screenshot is attached for your reference:
What's New? - Microsoft Fabric | Microsoft Learn

Microsoft Fabric offers preview features to allow users to evaluate them and provide valuable feedback to the product team before these features are made generally available (GA).

We kindly request you to share your feedback through the following link:
Azure Synapse Analytics · Community

If you require further assistance to resolve the issue, we recommend creating a Microsoft support ticket. Please refer to the instructions provided in the link below:
How to create a Fabric and Power BI Support ticket - Power BI | Microsoft Learn

If you find this response helpful, kindly mark it as the accepted solution and extend kudos. This will assist other community members who might be encountering similar issues.

Thank you.

[BMooreBUNN]

Thanks for your help. In Fabric itself it doesn't indicate that this is a preview feature so I didn't question that. Typically doesn't it say (Preview) in the name if it's not GA yet?

[v-pnaroju-msft]

Hi BMooreBUNN,

We sincerely apologize for the inconvenience you are facing while executing the high concurrency mode for notebooks in pipelines. Kindly review the links provided below to verify if they resolve the issue:

1.Introducing High Concurrency Mode for Notebooks in Pipelines for Fabric Spark | Microsoft Fabric Blo...

2.High Concurrency Mode for Notebooks in Pipelines for Fabric Spark

3.Configure high concurrency mode for notebooks in pipelines - Microsoft Fabric | Microsoft Learn

Thank you.

[BMooreBUNN]

So here's ultimately the issue. Even if I don't intend to utilize the high concurrency notebook functionality in a pipeline, the setting being on in my workspace causes the pipeline to fail anyways. That means I can't reliably use the setting at all since it being on causes errors in pipelines where two notebooks are utilized. This isn't about tuning a specific pipeline to make it work really.

[v-pnaroju-msft]

Hi BMooreBUNN,

We have not received a response from you regarding the query and were following up to check if you have found a resolution. If you have identified a solution, we kindly request you to share it with the community, as it may be helpful to others facing a similar issue.

If you find the response helpful, please mark it as the accepted solution and provide kudos, as this will help other members with similar queries.

Thank you.

[BMooreBUNN]

So if I turn the pipeline concurrency setting on, I start getting pipeline failures where I have multiple notebooks in the same pipeline. Sometimes even the first notebook in the pipeline fails.

It seems the setting itself is an issue. I have posted this multiple times on the r/MicrosoftFabric subreddit and other users report the same issue, and have had to disable the functionality as well.

[v-pnaroju-msft]

Hi @BMooreBUNN,

We sincerely appreciate your query on the Microsoft Fabric Community Forum.

To address the issue, we kindly request you to follow the troubleshooting steps mentioned below:

1. Ensure that the high concurrency setting is correctly enabled in the pipeline configuration. Double-check the settings to verify that they have been applied as intended.
2. Examine the resource allocation for the cluster. High concurrency settings may necessitate additional resources (CPU and memory) to manage multiple sessions effectively. Please ensure that the cluster is provisioned with adequate resources to handle the expected workload.
3. Temporarily simplify the pipeline by reducing the number of concurrent notebooks or sessions. This approach can help identify whether the issue is associated with resource constraints or configuration settings.
4. Verify that all notebooks requiring a shared session have matching session tags to ensure proper compatibility.

If you find this response helpful, kindly mark it as the accepted solution and extend kudos. This will assist other community members who might be encountering similar issues.

Thank you.