Forum Discussion
Where to Monitor Spark Pools?
I think the Capacity (F SKU) is the limiting factor. The Capacity (F SKU) decides how many Spark VCores you're allowed to use in one point of time. https://learn.microsoft.com/en-us/fabric/data-engineering/spark-job-concurrency-and-queueing
The pool is only a template for creating clusters, and thus decides the min and max limit for how many VCores a cluster created using that pool (template) can use.
The Capacity limits decide how many clusters you can use concurrently, also depending on how many VCores each cluster uses.
I'm not sure if there is a way to monitor how many VCores you're using in total at a single point in time. I only know how to monitor individual clusters (sessions). You would need to add the number of VCores used by each cluster (session), to determine if the Capacity's limits are close to being reached.
Another limit is the CU % limit (also a capacity level limit). That is not only related to Spark, but the combined load from all Fabric workloads. The CU % utilization is found in the Capacity Metrics App.
Hi frithjof_v
Thanks for the link. That was a missing piece of the puzzle, but there are other pieces missing as well.
Assuming a physical spark cluster is created under the covers, there is overhead to launching one. Spark clusters may have head nodes, zookeeper nodes, and worker nodes. These are all implemented as VM's and have boot times. Generally it is only the worker nodes that appear in the Spark History U/I so those are the ones that most people focus on.
In addition to these VM's, I think there is also a hidden layer of Hadoop software (Yarn) running on the nodes for the sake of resource management. That has additional overhead, when Spark sessions start and finish (yarn containers are launched for executors, and what-not).
I just created another post ("Queued time is elevated - can this trigger a notebook failure")
That post represents a real-world example of where it would be helpful to have a monitoring U/I. There is a Microsoft bug, and if I want to independently find a workaround, I need a bit of visibility.
I don't think I'm running in to the capacity limits of the F sku's. Even if my pool was at the 5 max nodes, it would only be using 8x5 vcores in the workers which is not a lot. I wasn't previously aware of that link you shared, but I think the capacity limits in that documentation were set in place so I wouldn't be able to have a cluster scale up to !50! nodes (causing problems for Microsoft, while not necessarily appearing in my synapse-notebook CU-meter). I think the arbitrary capacity limits (eg. 128 spark cores for F64) were put in place by Microsoft for self-preservation. I think the arbitrary limits are only necessary because of the fact that their CU billing is taking at the notebook level rather than at the cluster level. Else the CU charges themselves would prevent customers from autoscaling to 50.
Not to muddy this discussion, but in that other post, I believe the bug in Livy is due to basic lifecycle events (a sudden autoscaling of the physical cluster from 2 workers to 5 workers or so). I believe those lifecycle events are having buggy side-effects if they take longer than ten minutes. It is just a theory, and is very hard to prove without having some sort of a spark monitoring U/I.
It is theoretically possible for customers to build their own U/I by extracting the VM lists of the all these executors from all these notebooks, and collating by VM-name, to see how long-lived each VM was in the lifecycle of the cluster as a whole. It would be a lot of work, and frustrating to do it when Microsoft should be responsible for working on their own bugs.