Skip to main content
cancel
Showing results for 
Search instead for 
Did you mean: 

Enhance your career with this limited time 50% discount on Fabric and Power BI exams. Ends August 31st. Request your voucher.

Reply
dbeavon3
Memorable Member
Memorable Member

Lost monitoring details for notebook

I downloaded, deleted then uploaded a notebook with a new name (ipynb).  ... It is largely identical.

 

How do I find the Monitor activity for those spark sessions the pre-date this change?  Is it gone forever?  I guess it wouldn't surprise me.

 

Seems like Fabric shouldn't be so cavalier about blowing away the entire history of jobs running on spark.  I suppose I should have added some sort of additional KQL logging functionality?

 

1 ACCEPTED SOLUTION
v-saisrao-msft
Community Support
Community Support

Hi @dbeavon3,

Thank you @cald147 @lbendlin @Element115 

Please create a logging table to store logs. Once the table is ready, insert a log entry from the notebook, and update this entry after the process completes successfully.

Since Fabrics deletes logs from monitoring when a notebook is removed, this method can be used as an alternative workaround.

 

Thank you.

View solution in original post

11 REPLIES 11
v-saisrao-msft
Community Support
Community Support

Hi @dbeavon3,

Have you had a chance to review the solution we shared earlier? If the issue persists, feel free to reply so we can help further.

 

Thank you.

Doing our own logging shouldn't be the workaround.  Microsoft shouldn't just blow away the activity logs for a notebook.

Ain't that the sorry truth!

v-saisrao-msft
Community Support
Community Support

Hi @dbeavon3,

Thank you @cald147 @lbendlin @Element115 

Please create a logging table to store logs. Once the table is ready, insert a log entry from the notebook, and update this entry after the process completes successfully.

Since Fabrics deletes logs from monitoring when a notebook is removed, this method can be used as an alternative workaround.

 

Thank you.

Hi @v-saisrao-msft 

What resource are you referring to when you mention a table.  A Data Warehouse table?  That seems like a pretty expensive way to solve the problem

I will probably dump to temp tables in a lakehouse or something.

Ideally there would be a way to preserve Spark monitoring logs, regardless of whether the notebooks are temporarily removed.  Is it possible that Microsoft doesn't realize the importance of preserving operational logs?

Hi @dbeavon3,

You can choose to use a warehouse, Lakehouse, or SQL database, depending on what database you are using in your model. Consider setting up a logging table in your selected database to store the logs.

 

Thank you.

cald147
New Member

Yeah, once you delete and re upload the notebook with a new name, the old Spark session history is usually gone. Sadly, Fabric doesn’t keep that history by default. Adding custom KQL logging next time is a smart move to keep track.

lbendlin
Super User
Super User

You can get the "Workspace Monitoring"  eventhouse if you are willing to pay the extra cost.

 

In general you should assume that artifact logging is a 30 day window, and there is no SLA whatsoever.  Fun fact: The Capacity Metrics App Kusto database has an artificial ingestion limit of 100K operations per 30 second slot.  Any halfway busy capacity above F128 can easily blow through that limit, and you end up flying blind for x percent of your operations.

'Flying blind' would be a great marketing campaign! LOL Which makes me wonder how many enterprise clients are willing to put up with this and for how long.  If I were CEO... you can guess how this sentence ends. 😉

To their credit Microsoft are willing to listen to our rantings. I don't think they have yet internalized the business ROI centric view  (for example our single focus in all this is on CU auditing which seems to be a novel concept for them).  We'll see if and how they incorporate our input.

Element115
Super User
Super User

Cavalier?  LOL Let me tell you how cavalier it is:  Fabric just blew up all my partitions on one lakehouse table the other week and changed the table name. Just like that.  I guess it was in a bad mood or something.  😉  All joking aside, probably a dev that pushed by mistake some new code to production that should have gone through testing first, or the testing was so botched that disaster ensued.

 

Microsoft is working on a fix. So we wait and count the hours.

Helpful resources

Announcements
Fabric July 2025 Monthly Update Carousel

Fabric Monthly Update - July 2025

Check out the July 2025 Fabric update to learn about new features.

August 2025 community update carousel

Fabric Community Update - August 2025

Find out what's new and trending in the Fabric community.