Forum Discussion
Where to Monitor Spark Pools?
Hi frithjof_v
I agree that the spark "pool" concept (in Fabric) is intended to act as a template/metadata. This is different than the "pool" concept in Synapse.
Furthermore, I would agree that Microsoft is monetizing and billing us for our spark sessions and NOT our spark pools. That is a non-technical concern which I won't dig into (again)....
But none of these things are obviating the need to monitor/manage our custom spark pool. The pool has an important presence in our workspace, and we must interact with them whether we like it or not. Here is our custom pool:
Lets consider some real-world scenarios. Suppose we have a bunch of notebooks running on the cluster (starting and completing) for a period of time. Then all of a sudden we might see that new notebooks will not be launched and they will start piling up (queuing). There are a variety of underlying reasons for this that are directly caused by the underlying spark pool. For example, the pool may be in the process of "autoscaling". Or the pool may have unexpectedly reached its max capacity, because some notebooks grabbed an excessive number of dynamically allocated executors. These are the simplest sorts of scenarios for why notebooks won't launch on a pool. Other scenrios involve PG bugs, configuration issues, operating system issues, "transient network failures", tenant throttling issues, Entra ID errors, and so on.
A wide variety of issues can be encounter in cloud-hosted SaaS and PaaS. Yet it is not possible to troubleshoot for them, since Microsoft gives us no visibility to see the underlying pool itself. It would seem that the PG expects us to open support tickets with Mindtree every time our pyspark notebook environment has become frozen. Since I already have too many of those tickets, and since those tickets will generally take a week of effort (or more), I'm not very eager to continue to follow that path. After that week has passed, I would then be dealing with LOTS of severe problems, instead of that one simple problem that I started with.
I'm guessing that the lack of a monitoring console is either an oversight, or it is still on the roadmap, or Microsoft expects their Mindtree vendor to have the role of a monitoring tool. None of these seem acceptable to me. Spark is already a challenging technology for developers to manage, without these "blinders" that we must wear in the Fabric platform.
I think Microsoft does mean well, in trying to make an "easy" platform in Fabric. But I think they have taken things way too far in that regard. (Spark notebooks should not have gone to GA without having a monitoring tool. It violates a basic principle to “make things as simple as possible, but no simpler.”)
I think the Capacity (F SKU) is the limiting factor. The Capacity (F SKU) decides how many Spark VCores you're allowed to use in one point of time. https://learn.microsoft.com/en-us/fabric/data-engineering/spark-job-concurrency-and-queueing
The pool is only a template for creating clusters, and thus decides the min and max limit for how many VCores a cluster created using that pool (template) can use.
The Capacity limits decide how many clusters you can use concurrently, also depending on how many VCores each cluster uses.
I'm not sure if there is a way to monitor how many VCores you're using in total at a single point in time. I only know how to monitor individual clusters (sessions). You would need to add the number of VCores used by each cluster (session), to determine if the Capacity's limits are close to being reached.
Another limit is the CU % limit (also a capacity level limit). That is not only related to Spark, but the combined load from all Fabric workloads. The CU % utilization is found in the Capacity Metrics App.
Hi frithjof_v
Thanks for the link. That was a missing piece of the puzzle, but there are other pieces missing as well.
Assuming a physical spark cluster is created under the covers, there is overhead to launching one. Spark clusters may have head nodes, zookeeper nodes, and worker nodes. These are all implemented as VM's and have boot times. Generally it is only the worker nodes that appear in the Spark History U/I so those are the ones that most people focus on.In addition to these VM's, I think there is also a hidden layer of Hadoop software (Yarn) running on the nodes for the sake of resource management. That has additional overhead, when Spark sessions start and finish (yarn containers are launched for executors, and what-not).
I just created another post ("Queued time is elevated - can this trigger a notebook failure")
That post represents a real-world example of where it would be helpful to have a monitoring U/I. There is a Microsoft bug, and if I want to independently find a workaround, I need a bit of visibility.I don't think I'm running in to the capacity limits of the F sku's. Even if my pool was at the 5 max nodes, it would only be using 8x5 vcores in the workers which is not a lot. I wasn't previously aware of that link you shared, but I think the capacity limits in that documentation were set in place so I wouldn't be able to have a cluster scale up to !50! nodes (causing problems for Microsoft, while not necessarily appearing in my synapse-notebook CU-meter). I think the arbitrary capacity limits (eg. 128 spark cores for F64) were put in place by Microsoft for self-preservation. I think the arbitrary limits are only necessary because of the fact that their CU billing is taking at the notebook level rather than at the cluster level. Else the CU charges themselves would prevent customers from autoscaling to 50.
Not to muddy this discussion, but in that other post, I believe the bug in Livy is due to basic lifecycle events (a sudden autoscaling of the physical cluster from 2 workers to 5 workers or so). I believe those lifecycle events are having buggy side-effects if they take longer than ten minutes. It is just a theory, and is very hard to prove without having some sort of a spark monitoring U/I.It is theoretically possible for customers to build their own U/I by extracting the VM lists of the all these executors from all these notebooks, and collating by VM-name, to see how long-lived each VM was in the lifecycle of the cluster as a whole. It would be a lot of work, and frustrating to do it when Microsoft should be responsible for working on their own bugs.