Forum Discussion
Fabric pipelines audit
Hi community,
I am looking for a way to log pipelines details into an audit delta table. To be more specific, I want to log the start time, end time, status, error message if any etc. Having this by activity name would be even better.
I was thinking to use a notebook that's being called at the end of each pipeline, so somehow to have only a notebook that can be used in all of my pipelines (because i have a lot of them).
Is this something that you were able to build? Any guidance would be very much appreciated.
thanks!
2 Replies
- Olufemi7
Super User
Hello Eleanordum93,
You may not need a notebook for this. Fabric Workspace Monitoring provides pipeline and activity-level execution logs, including status, timestamps, activity name and pipeline run ID.You can query these logs and persist the fields you need into your own audit table.
Microsoft docs:
Enable workspace monitoring in Microsoft Fabric - tayloramy
Hi Eleanordum93,
I would recommend creating a UDF that accepts the things you want to log as parameters, and then saves them to a warehouse or SQL database. THen you can call that UDF from the pipeline (or from a notebook, or from anywhere else really) to log something.
This is better than using a notebook because it doesn't require a spark session to be created so will be cheaper on compute/CU usage.
