Forum Discussion
Where to Monitor Spark Pools?
Hi dbeavon3 ,
Thank you for reaching out to Microsoft Fabric Support about the monitoring issues you're facing with your custom Spark pool in Microsoft Fabric. We understand the troubles you're having, particularly with the capacity metrics app and not being able to see Spark pool utilization and performance.
The documents you shared mainly cover monitoring Apache Spark capacity consumption, billing, and utilization reporting. They don't specifically address monitoring Spark pools.
I've attached the official Microsoft documents that directly tackle your concern. These documents explain where and how to monitor your custom Apache Spark pools, and how to view specific details about each Spark pool. Please look at the attached link for clear and detailed guidance.
Link: How to monitor Apache Spark pools in Synapse Studio - Azure Synapse Analytics | Microsoft Learn.
Considering your Microsoft Fabric (F64) capacity, whether you need an upgrade depends on your current workload demands and limitations. It might be a good idea to think about upgrading to a higher SKU (like F64 to F64v2 or another high-performance option).
- Since multiple notebooks share your pool during peak hours, it may be beneficial to increase capacity or optimize resource allocation to ensure smoother performance. Adjusting the auto-scaling settings can also help, if applicable.
- For a more detailed view of the Spark jobs and tasks in your pool, enable Spark job monitoring through the Spark UI or check Spark logs. This will give you a clearer picture of how the workload is distributed.
Link: Use the Monitoring hub to manage Apache Spark applications - Microsoft Fabric | Microsoft Learn.
- You can monitor your Spark pool using custom REST APIs. These APIs provide real-time data on your pool's performance, including resource usage, job execution times, and other metrics. You can integrate this data into a monitoring dashboard for better insights into when the pool might be overwhelmed or when resource costs are high.
Link: Collect Apache Spark applications metrics using APIs - Azure Synapse Analytics | Microsoft Learn.
- You may want to look into using Azure Monitor for setting up custom metrics and alerts for your Spark pool. This can help you keep track of the pool's health and resource usage over time. Even though there are some ongoing issues with the Fabric capacity metrics app, Azure Monitor is a solid alternative for collecting resource usage data and creating alerts.
If my answer addressed your query, kindly mark it as the Accepted Solution to assist others.
I'd also be grateful for a 'Kudos' if you found my response useful!
V-yubandi-msft
Thanks for the reply
>>These documents explain where and how to monitor your custom Apache Spark pools (in synapse)
Yes, I already aware of how the Synapse PaaS works, but this is a question that focuses on Fabric.
>> It is the new Fabric that doesn't seem to be working as expected, or giving visibility to monitor pools.
It might be a good idea to think about upgrading to a higher SKU (like F64 to F64v2 or another high-performance option).
We use F64. There is no such thing as "F64v2". Is that an internal code-word?
>> You can monitor your Spark pool using custom REST APIs. These APIs provide real-time data on your pool's performance, including resource usage, job execution times, and other metrics
Can you share a link to the corresponding thing in Fabric?
Please verify that we are talking about Fabric and not Synapse? Either that API is not in Fabric, or you are assuming that Fabric and Synapse have the exact same set of features (ie. all the docs for Synapse are equally relevant to Fabric). This second scenario seems pretty unlikely.
If there was already some Fabric documentation related to monitoring pools in Fabric, then I probably would have found it. I already saw all of the docs related to the Synapse platform, but those do not apply to Fabric, (AFAIK). I would appreciate if you would only reference the information that pertains directly to Fabric, so we don't introduce additional confusion.... This is a topic that other customers will have in the future, and it would be well if we restricted the discussion to focus only on Fabric and avoid confusing Fabric and Synapse platforms. This discussion may be used by Microsoft partners as well. Eg. I may also refer the Mindtree engineers to it, when asking follow-up questions about your suggestions .
Hi dbeavon3 ,
Thank you for your feedback and apologize for the confusion regarding Synapse and Fabric earlier.
You are correct that, at this moment, there isn't an equivalent of the Synapse monitoring tools in Microsoft Fabric for Spark pools. My apologies for any misunderstanding. The F64 (Fabric) SKU doesn't have an "F64v2" variant like some of the Synapse SKUs. My earlier suggestion was more generic, and we appreciate your patience.
A special thanks to frithjof_v , for their valuable contribution. Currently, Fabric lacks a dedicated, unified tool for the monitoring of Spark pools. We are closely following the progress that Fabric's monitoring capabilities are making and will keep you updated on any developments that might address your needs. In the meantime, continue to monitor the system.
Thank you again for your patience as we go through this. We continue to look out for any new development that would increase the monitoring options in Microsoft Fabric.