Forum Discussion
Where to Monitor Spark Pools?
V-yubandi-msft
Thanks for the reply
>>These documents explain where and how to monitor your custom Apache Spark pools (in synapse)
Yes, I already aware of how the Synapse PaaS works, but this is a question that focuses on Fabric.
>> It is the new Fabric that doesn't seem to be working as expected, or giving visibility to monitor pools.
It might be a good idea to think about upgrading to a higher SKU (like F64 to F64v2 or another high-performance option).
We use F64. There is no such thing as "F64v2". Is that an internal code-word?
>> You can monitor your Spark pool using custom REST APIs. These APIs provide real-time data on your pool's performance, including resource usage, job execution times, and other metrics
Can you share a link to the corresponding thing in Fabric?
Please verify that we are talking about Fabric and not Synapse? Either that API is not in Fabric, or you are assuming that Fabric and Synapse have the exact same set of features (ie. all the docs for Synapse are equally relevant to Fabric). This second scenario seems pretty unlikely.
If there was already some Fabric documentation related to monitoring pools in Fabric, then I probably would have found it. I already saw all of the docs related to the Synapse platform, but those do not apply to Fabric, (AFAIK). I would appreciate if you would only reference the information that pertains directly to Fabric, so we don't introduce additional confusion.... This is a topic that other customers will have in the future, and it would be well if we restricted the discussion to focus only on Fabric and avoid confusing Fabric and Synapse platforms. This discussion may be used by Microsoft partners as well. Eg. I may also refer the Mindtree engineers to it, when asking follow-up questions about your suggestions .
Hi dbeavon3 ,
Thank you for your feedback and apologize for the confusion regarding Synapse and Fabric earlier.
You are correct that, at this moment, there isn't an equivalent of the Synapse monitoring tools in Microsoft Fabric for Spark pools. My apologies for any misunderstanding. The F64 (Fabric) SKU doesn't have an "F64v2" variant like some of the Synapse SKUs. My earlier suggestion was more generic, and we appreciate your patience.
A special thanks to frithjof_v , for their valuable contribution. Currently, Fabric lacks a dedicated, unified tool for the monitoring of Spark pools. We are closely following the progress that Fabric's monitoring capabilities are making and will keep you updated on any developments that might address your needs. In the meantime, continue to monitor the system.
Thank you again for your patience as we go through this. We continue to look out for any new development that would increase the monitoring options in Microsoft Fabric.