topic Re: Lost monitoring details for notebook in Data Engineering

Lost monitoring details for notebook

dbeavon3 — Fri, 18 Jul 2025 18:59:42 GMT

I downloaded, deleted then uploaded a notebook with a new name (ipynb). ... It is largely identical.

How do I find the Monitor activity for those spark sessions the pre-date this change? Is it gone forever? I guess it wouldn't surprise me.

Seems like Fabric shouldn't be so cavalier about blowing away the entire history of jobs running on spark. I suppose I should have added some sort of additional KQL logging functionality?

Re: Lost monitoring details for notebook

Element115 — Sat, 19 Jul 2025 00:31:49 GMT

Cavalier? LOL Let me tell you how cavalier it is: Fabric just blew up all my partitions on one lakehouse table the other week and changed the table name. Just like that. I guess it was in a bad mood or something. 😉 All joking aside, probably a dev that pushed by mistake some new code to production that should have gone through testing first, or the testing was so botched that disaster ensued.

Microsoft is working on a fix. So we wait and count the hours.

Re: Lost monitoring details for notebook

lbendlin — Sat, 19 Jul 2025 00:24:20 GMT

You can get the "Workspace Monitoring" eventhouse if you are willing to pay the extra cost.

In general you should assume that artifact logging is a 30 day window, and there is no SLA whatsoever. Fun fact: The Capacity Metrics App Kusto database has an artificial ingestion limit of 100K operations per 30 second slot. Any halfway busy capacity above F128 can easily blow through that limit, and you end up flying blind for x percent of your operations.

Re: Lost monitoring details for notebook

Element115 — Sat, 19 Jul 2025 00:34:23 GMT

'Flying blind' would be a great marketing campaign! LOL Which makes me wonder how many enterprise clients are willing to put up with this and for how long. If I were CEO... you can guess how this sentence ends. 😉

Re: Lost monitoring details for notebook

lbendlin — Sat, 19 Jul 2025 00:41:29 GMT

To their credit Microsoft are willing to listen to our rantings. I don't think they have yet internalized the business ROI centric view (for example our single focus in all this is on CU auditing which seems to be a novel concept for them). We'll see if and how they incorporate our input.

Re: Lost monitoring details for notebook

cald147 — Sat, 19 Jul 2025 12:13:05 GMT

Yeah, once you delete and re upload the notebook with a new name, the old Spark session history is usually gone. Sadly, Fabric doesn’t keep that history by default. Adding custom KQL logging next time is a smart move to keep track.

Re: Lost monitoring details for notebook

v-saisrao-msft — Mon, 21 Jul 2025 11:11:35 GMT

Hi dbeavon3,

Thank you cald147 lbendlin Element115

Please create a logging table to store logs. Once the table is ready, insert a log entry from the notebook, and update this entry after the process completes successfully.

Since Fabrics deletes logs from monitoring when a notebook is removed, this method can be used as an alternative workaround.

Thank you.

Re: Lost monitoring details for notebook

dbeavon3 — Mon, 21 Jul 2025 12:21:27 GMT

Hi v-saisrao-msft

What resource are you referring to when you mention a table. A Data Warehouse table? That seems like a pretty expensive way to solve the problem

I will probably dump to temp tables in a lakehouse or something.

Ideally there would be a way to preserve Spark monitoring logs, regardless of whether the notebooks are temporarily removed. Is it possible that Microsoft doesn't realize the importance of preserving operational logs?

Re: Lost monitoring details for notebook

v-saisrao-msft — Wed, 23 Jul 2025 09:58:34 GMT

Hi dbeavon3,

You can choose to use a warehouse, Lakehouse, or SQL database, depending on what database you are using in your model. Consider setting up a logging table in your selected database to store the logs.

Thank you.

Re: Lost monitoring details for notebook

v-saisrao-msft — Mon, 28 Jul 2025 04:32:25 GMT

Hi dbeavon3,

Have you had a chance to review the solution we shared earlier? If the issue persists, feel free to reply so we can help further.

Thank you.

Re: Lost monitoring details for notebook

dbeavon3 — Tue, 29 Jul 2025 00:02:28 GMT

Doing our own logging shouldn't be the workaround. Microsoft shouldn't just blow away the activity logs for a notebook.

Re: Lost monitoring details for notebook

Element115 — Tue, 29 Jul 2025 16:44:35 GMT

Ain't that the sorry truth!