Fabric pipelines audit

Hi everyone! Thank you for your kind help! Being FY end, my time was limited, but will work on a solution taking into consideration your answers and will come back! 
I am thinking to work on the stored procedure that was suggested.

Hi Eleanordum93​ 

Thanks for sharing your scenario. The suggestions above provide a good starting point for building a centralized audit solution across multiple Fabric pipelines.

I would recommend using Workspace Monitoring for standard pipeline and activity-level telemetry, combined with a reusable logging component for any additional information you need to persist.

Workspace Monitoring exposes pipeline and activity execution details, including pipeline and activity names, run IDs, status, start and end timestamps, duration, and failure type. You can query these logs using KQL and use them for centralized monitoring and reporting.

One limitation to consider is that Workspace Monitoring does not currently provide full error details and diagnostics. If capturing the exact activity error message in your own audit table is a requirement, you can use a reusable logging component, such as a Fabric User Data Function or a Warehouse stored procedure, to persist the additional details without starting a Spark session solely for logging.

This hybrid approach helps avoid duplicating logging logic across pipelines while keeping the audit design maintainable.
For implementation details, please refer to the official Microsoft documentation:

Enable Workspace Monitoring (Preview) in Microsoft Fabric - Microsoft Fabric | Microsoft Learn
Choose How to Monitor Pipeline Runs - Microsoft Fabric | Microsoft Learn

I hope this helps you build a centralized and reusable pipeline auditing solution.

Hello Eleanordum93​,

You may not need a notebook for this. Fabric Workspace Monitoring provides pipeline and activity-level execution logs, including status, timestamps, activity name and pipeline run ID.

You can query these logs and persist the fields you need into your own audit table.

Microsoft docs:
Enable workspace monitoring in Microsoft Fabric

Hi Eleanordum93​, 

Both good suggestions above. One thing that helps with having lots of pipelines is building a small reusable "logging" pipeline (or UDF, as tayloramy said) with parameters like pipeline name, run ID, activity name, status, start time, end time and error message. Then each pipeline just calls it with an Invoke Pipeline activity on the success and failure paths, so the logic lives in one place.

The values you pass in are mostly available as dynamic content already, for example @pipeline().Pipeline, @pipeline().RunId, @pipeline().TriggerTime and @activity('YourActivity').error.message on the failure branch.

If you want to keep it cheap, a Stored Procedure or Script activity writing to a Warehouse avoids spinning up Spark, and Warehouse tables are stored as Delta in OneLake anyway. You can also use Workspace Monitoring for the detailed history and keep your own table for just the fields you care about.

Hi Eleanordum93​,

I would probably avoid putting a logging notebook at the end of every pipeline unless you specifically need Spark.

Fabric now has Workspace Monitoring, which can capture pipeline and activity-level execution telemetry centrally for the whole workspace.

For pipelines, Microsoft exposes details such as:

• pipeline name and run ID
• activity name and activity type
• status
• start and end timestamps
• duration
• failure type

That means a large part of your audit requirement can come from the monitoring logs without adding the same logging activity to every pipeline.

There is one important limitation though: Microsoft currently documents that Workspace Monitoring does not include full error details/diagnostics.

So if you specifically need the exact activity error message in your Delta audit table, I would use a hybrid approach:

Workspace Monitoring for standard run/activity telemetry

+

one reusable logging component only for the extra fields you need, such as detailed error text or business-specific metadata.

For that reusable component, I would prefer either a Fabric User Data Function or a Stored Procedure activity over a notebook if the job is simply to insert an audit row.

That keeps the logging lightweight and avoids starting a Spark session just to write a few values.

A practical pattern could be:

pipeline/activity runs
-> Workspace Monitoring
-> central monitoring/audit table

and on failure:

failure branch
-> reusable logging function/procedure
-> persist detailed error message + any custom metadata

That gives you one central audit design without having to duplicate full logging logic across every pipeline.

AI-assisted drafting: AI was used to help structure and phrase this response. I reviewed and validated the technical content before posting.

Hi Eleanordum93​ 

Following up to confirm if the earlier responses addressed your query. If not, please share your questions and we’ll assist further.

Hi Eleanordum93​ 

Have you had a chance to look through the responses shared earlier? If anything is still unclear, we’ll be happy to provide additional support.

Hi everyone! 
I started testing the Monitoring database and indeed it gives a lot of information, but i cannot link the FabricDataPipelinesActivityRunsLogs and ItemJobsEventsLogs. 
I.E: I have the activities from a pipeline with their status, but cannot see from which pipeline they are coming from. Couldn't find a lot of information this database related queries. 
I found some samples online, but not very helpful. Just trying to create a live report for now, to see what failed today, which step from which pipeline etc. Any thoughts? 

Thank you!