Capacity Metrics Dashboard - Notebook Pipeline Run reported incorrectly (time off by an hour?)

[dbeavon3]

We have been trying to use Fabric notebooks for about a week. 

Trying to manage "CU" costs has been particular hard because:

 - Our capacity metrics app was suffering from a two month outage in the north central region (finally resolved on Friday), and

 - The Fabric app does not seem to have any console for monitoring Spark clusters.  In Synapse there used to be cluster-level management but in Fabric, the only monitoring tool that allows customers to keep an eye on the costs of Spark resources is the so-called "capacity metrics app".  It has never been a very friendly tool for admin purposes, and that goes double in the context of Spark.

As I try to use the metrics app I'm finding that the date/times listed in the "background operations" appear to be wrong by an hour.  My timezone is East US, and the ultimate basis for comparison is UTC (reported in the Spark History UI).     

Some places report the date/times correctly like the ADF "pipelines" as follows.  It shows that the pipeline ran for about an hour and forty minutes starting at 4 AM.

... continued...

That corresponds to about 9 AM UTC.

... but in the capacity metrics app, the time is off by an hour.

It seems to say that my run started at 5 AM.  There is no USA timezone where that could be possible.  I'd guess some developer introduced some really bad date/time math, for the sake of the details that are shown here.

It is frustrating to have such a basic type of issue in the metrics app.  I think the app should at least indicate what timezone it is referring to.  Or perhaps it could have a toggle to show UTC, assuming the developers are incapable of presenting time in the user's preference of timezone.   I'm guessing this issue doesn't go any further than the U/I itself.  But it can make customers nervous when these bugs appear in the tools that we are using to manage our costs.  If Microsoft can't present the date/time correctly, it makes us wonder about the likelihood that are CU's are being miscalculated as well!

[frithjof_v]

Which UTC offset have you used in the Capacity Metrics App's semantic model settings?

Also, just to make sure regarding Spark Monitoring, have you checked the Monitoring Hub and the monitoring of individual Spark sessions which is possible there?

Have you familiarized yourself with the contents of this documentation: https://learn.microsoft.com/en-us/fabric/data-engineering/spark-detail-monitoring

[dbeavon3]

I should include that I'm running v.1.5.1 of the capacity metrics app.

My original theory was that this bug was specific to the North Central US region.

However I think it is probably even more simple than that.  It seems to be affecting East US as well.  It appears that all the date/times for these synapse notebooks are being reported as UTC-4 instead of UTC-5.  That may be the result of not taking the seasonal differences into account.  This app may be unconditionally using the UTC-4 math to display our start/stop times in the eastern timezone. 

I have opened a support ticket, but it hasn't gotten to Microsoft yet.  Currently with their partners at Mindtree.

The only mystery is why I'm the first to complain about this.  Perhaps it is a recent regression or something like that.

[v-prasare]

Hi @dbeavon3,

Thanks for actively participating in MS Fabric Community Support.

As mentioned you have already raised a support ticket, While you're waiting for the support ticket resolution, your feedback about improving the Capacity Metrics App particularly in terms of time zone clarity, Spark cluster monitoring, and admin usability is highly relevant and insightful. It's important that our product team hears about this directly from users like you, who are deeply engaged with the platform and rely on it for essential operations, these insights will undoubtedly contribute to a better experience for all users.

I’d encourage you to submit your detailed feedback and ideas via Microsoft's official feedback channels, such as the Microsoft Fabric Ideas. Feedback submitted here is often reviewed by the product teams and can lead to meaningful improvement.

Thanks,

Prashanth Are

MS Fabric Community Support

[dbeavon3 · Accepted solution]

Hi @v-prasare 

Its funny, but I see that there are dozens and dozens of other posts about the timezone problem in the capacity metrics app.  

I didn't see those at first.  

No customer would expect that we should be updating a timezone offset two times a year, in a configuration parameter.  Nor would we expect that there is only one timezone that is relevant to the entire capacity.  This had the feeling of a bug, but I don't think Microsoft considers it a bug.  It appears in the microsoft documentation.

I'm probably going to close the case now.  Whenever I install this app in the future, I will just use UTC.  Less confusing.

[Element115]

Exactly right and don't get me started on this issue in Power BI Desktop.  It's outrageous and a total disaster how badly the ball has been dropped with timezone management. So much for time intelligence, huh?  😂

[v-prasare]

@dbeavon3, Thank you for your detailed observations and for identifying the underlying issue with the timezone offset in the Capacity Metrics App. We sincerely appreciate the time and effort you put into exploring this matter and sharing your findings.

please accept your answer, this will help others users to find answer to similar issues more quickly

Thanks,

Prashanth Are