Users limits for notebooks sessions within Fabric capacity.

[Anonymous]

Hi..I would like to understand one thing about Fabric capacity. 

We have tried a trial capacity. But we struggle when 4 users running interactive Spark queries or notebooks running from a pipeline. We always hit a limit of capacity  

It was the only reason we moved our notebooks back to Databricks (still connected to OneLake). 

To be honest I do not want to pay 8K/month for 2 users are running pipelines notebooks sequtiontionaly beside. So I have not purchase it and moved to Databrick where I do not care how many users are connected to one cluster. 

What am I missing? is it only limitation of trial? where can I find more info? I am aware to recommend Fabrics as a end-to-end platform to other clients. 

Thanks for explenation. 

[AndyDDC]

There's a few things to unpack here with Spark in Fabric.  Have you read through the doc here about concurrenty limits in Spark in Fabric?  Well worth reading if you haven't Concurrency limits and queueing for Fabric Spark - Microsoft Fabric | Microsoft Learn

Then there's the interactive concurrency when running code in Notebooks, you can utilise the High Concurrency mode to re-use Spark sessions acrossmultiple notebooks: Configure high concurrency mode for notebooks - Microsoft Fabric | Microsoft Learn

With running Notebooks from Data Pipelines, I often see issues where people are using a ForEach loop in a pipeline to execute a notebook, this will create a new spark session and spark cluster for each run of the notebook - there is no way to re-use a session/cluster.

Then there's the actual Spark cluster(s) that have been created in the workspace, ytou can configure different sizes of clusters, it may be that a Small cluster is what you need to run workloads - that will help with concurrency. 

I hope that has been of some help.

[Anonymous]

Hi @AndyDDC , will be honest with you. I have read all but not understood. Will have to invest more time in the future to know to scale capacity to support 5; 10; 20 users. I will give a chance to Copilot, hope it smarter than I am.
As I wrote. we are thinking about moving notebooks from Databricks, where you connect it to cluster and it works 🙂 so this is something I was not expecting from SaaS like Fabric. Hope the UX will imporove in time. 

We have also tried concurrency mode. but it sometimes works sometimes not. Very often it fails to establish a session and swith automatically back to standard session. Also it was a solution only to run multiple (5)  notebooks by one user. But other users cannot to see it and connect to it. 
So still we were limit (with trial capacity) to have one running pipeline and 2 users interactively using notebooks experience. 

Size cluster. yea that was also the idea to use differen capacity to run notebooks within pipeline. But I do not an option to choose a capacity to run within notebook ADF activity. 

[Anonymous]

I have not accepted a solution. High concurency mode did not solved the issue with users limits. 

[Anonymous]

Hi @Anonymous 
Thanks for using Fabric Community.
You can use this feature : Microsoft Spark Utilities (MSSparkUtils) for Fabric - Microsoft Fabric | Microsoft Learn
Hope this helps. Please let me know if you have any further questions.

[Anonymous]

Hi @Anonymous Thanks a lot for a quick response. I have seen this in last updates. Two things:

- this probably will solve only pipeline issue, so I can run more notebooks in pararell, but not users issues. Platform for 8K/M will be still accessible only for 3 unique users, correct ?

- there is overhead to manage notebooks outputs, and other dependend task in pipeline. So still is for us better solution to call Databricks notebooks from pipeline than invest time to this workaround. 

[Anonymous]

Hi @Anonymous 
Thanks for the feedback. The internal team has confirmed that High concurrency mode for pipeline, small sized single node compute are on the roadmap. There is no ETA for deployment.

We'll introduce new features to help mitigate the problem. I request you to please share the feedback on our feedback channel , which would be open for the user community to upvote & comment on. This allows our product teams to effectively prioritize your request against our existing feature backlog and gives insight into the potential impact of implementing the suggested feature.

Hope this helps. Please let me know if you have any further queries.

[Anonymous]

Hi @Anonymous 
We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet.
Otherwise, will respond back with the more details and we will try to help.
Thanks

[Anonymous · Accepted solution]

Hi, still not sure how to design this to use Fabric for more than 2-3 users in interactive way. So we continue with Databricks.