topic Re: Stream Notebook logs in Data Engineering

Stream Notebook logs

AdarshPanasri — Tue, 05 Nov 2024 09:08:59 GMT

Hello Everyone, I have setup a logging mechanism using the logging library from python, while my code generates the logs and persist them in the lakehouse, I want to stream the logs real-time so that I can monitor the status of the job, has someone tried something similar and can suggest something that I can implement to achieve this. Thanks

Re: Stream Notebook logs

Anonymous — Wed, 06 Nov 2024 06:43:29 GMT

Hi @AdarshPanasri ,

I have not found a way to stream logs in real time in Fabric, so I'd be grateful if others could advise.

Those who want to monitor the status of jobs can use the Microsoft Fabric Monitoring Center, where users are able to monitor Microsoft Fabric activity from a central location.

By clicking on the Monitor pane you can select the type of filter you want and then you can see its running status and also open it to view the details.

More details about Monitor can be found in this document:

Use the Monitor hub - Microsoft Fabric | Microsoft Learn

Best Regards,
Yang
Community Support Team

If there is any post helps, then please consider Accept it as the solution to help the other members find it more quickly.
If I misunderstand your needs or you still have problems on it, please feel free to let us know. Thanks a lot!

Re: Stream Notebook logs

Anonymous — Thu, 14 Nov 2024 02:05:38 GMT

Hi @AdarshPanasri ,

Is my follow-up just to ask if the problem has been solved?

If so, can you accept the correct answer as a solution or share your solution to help other members find it faster?

Thank you very much for your cooperation!

Best Regards,
Yang
Community Support Team

Re: Stream Notebook logs

Srisakthi — Thu, 14 Nov 2024 07:29:29 GMT

Hi @AdarshPanasri ,

Not sure whether a direct way available as of now, but you can try this option

Go to your notebook, select Run-> Viewallruns and then go to your specific application , you can see as below

There should be some api's available to fetch the logs, but I havent explored. Thought this would give some information.

Regards,

Srisakthi

Re: Stream Notebook logs

vgiatti — Thu, 20 Mar 2025 10:27:55 GMT

You can use the method proposed by Dennes. See:

the real pain is that (as of today) the tables are ingested in Azure Log Analytics as "Custom Table (classic)", and these tables cannot be added to an Azure Events Hub, which is the only "make-sense" real-time source to be used in a Fabric Eventstream pipeline.