Fabric Monitor is Truncating the History of Notebooks Prematurely

[dbeavon3]

I'm midway thru a support case.  All of a sudden we've started losing the notebook executions from the Fabric "Monitor" blade.

It is only 1/20 and we are already losing notebook executions from 1/15, when my support case was opened.   See below. 

Notice that pipelines are appearing for the days prior to 1/16 but the Notebooks are not.

This is very convenient for the PG, if they don't need to work on fixing bugs, after their logs are lost.  The PG simply needs to procrastinate for a few days, and all the history is lost.  The customer is forced to start over again and again, with new cases and new examples of the Fabric bugs. 

Has anyone seen this premature truncation of notebooks?  I would have expected these details to remain visible to me for at least a month.  Not five days.  Is it possible to customize the holding period for these notebooks?  Is it possible to recover the lost history? 

[nilendraFabric · Accepted solution]

Hi @dbeavon3 

According to the official documentation, the Monitor hub in Microsoft Fabric should retain activity history for a longer period:
1. Each Fabric item displays a maximum of 100 activities.
2. History is kept for 30 days and can be viewed using the Historical view option.

https://learn.microsoft.com/sl-si/fabric/admin/monitoring-hub

can it be the case for this item number of activities is more than 100 , so the previous activities are truncated?

please accept it solution if this helps.

thanks

[nilendraFabric]

@dbeavon3

please accept this solution, so other community users can get help as well.

[nilendraFabric]

@dbeavon3

please accept this solution, so other community users can get help as well.

[dbeavon3 · Accepted solution]

@nilendraFabric 

Thank you very much for your time.  Have you ever engaged with the PG about this platform?

The monitoring experience should go back 30 days at the very least.  Whoever built this U/I is unlikely to have familiarity with spark.

Being able to reviewing spark/notebook history is critical. 

Consider you have four business units at your company and you run a per-business-unity notebook execution once per hour (for sales data or whatever).  That makes 24x4.  By the end of the day your execution history is rolling over and you can't even see the executions for the prior day, let alone the prior week.  It can feel like a shock not to be able to pull up those executions.

>> kept for 30 days and can be viewed using the Historical view option.

The "historical view" option is probably what I am looking for, and it wasn't obvious why those wouldn't simply be shown in the normal list.  In fact I was complaining to a (well qualified) Mindtree TA for ten mins about the lack of any meaningful history for spark.  I don't think he actually knew about the "historical view" either.  For others who are looking, you can find them in the context menu like so:

[nilendraFabric]

Hi @dbeavon3 Agreed with you here. Hopefully someone from PG is reading this here. Else please post a idea, it's worthwhile and I think very straightforward to implement in Monitor Hub

thanks

[nilendraFabric]

Hello @dbeavon3 ,

their is not a direct way in Fabric to achieve this. 

but Consider implementing additional logging within notebooks themselves to maintain a longer execution history externally.

please accept this solution and give kudos if this information is helpful 

thanks

[dbeavon3]

Spark has executor logs (stderr/stdout) that are critical for diagnosing a wide variety of issues.  It makes no sense for customers to reinvent the wheel.  We need a way to capture those logs, rather than lose them after just a few days.  Do you know of a programmatic interface for retrieving those logs?

[nilendraFabric · Accepted solution]

Hi @dbeavon3 

According to the official documentation, the Monitor hub in Microsoft Fabric should retain activity history for a longer period:
1. Each Fabric item displays a maximum of 100 activities.
2. History is kept for 30 days and can be viewed using the Historical view option.

https://learn.microsoft.com/sl-si/fabric/admin/monitoring-hub

can it be the case for this item number of activities is more than 100 , so the previous activities are truncated?

please accept it solution if this helps.

thanks

[dbeavon3]

Hi @nilendraFabric 

That is probably the reason.  We run some notebooks well over 30 times a day.  If they are only sharing 100 prior activities, then it leaves almost no time to report a bug.  The Mindtree organization takes several days before they send bugs to Microsoft, and my that time all the details are lost!!! 

.... Especially if the bug happens on a Friday, and the case is opened on a Monday.

https://learn.microsoft.com/sl-si/fabric/admin/monitoring-hub#view-the-monitor-hub-display

I'm truly shocked that we would lose access to all our Fabric logs after only a few days.  Do you know if there is a way to capture them before they are removed?