Fabric pipelines audit

Hi Eleanordum93​,

I would probably avoid putting a logging notebook at the end of every pipeline unless you specifically need Spark.

Fabric now has Workspace Monitoring, which can capture pipeline and activity-level execution telemetry centrally for the whole workspace.

For pipelines, Microsoft exposes details such as:

• pipeline name and run ID
• activity name and activity type
• status
• start and end timestamps
• duration
• failure type

That means a large part of your audit requirement can come from the monitoring logs without adding the same logging activity to every pipeline.

There is one important limitation though: Microsoft currently documents that Workspace Monitoring does not include full error details/diagnostics.

So if you specifically need the exact activity error message in your Delta audit table, I would use a hybrid approach:

Workspace Monitoring for standard run/activity telemetry

+

one reusable logging component only for the extra fields you need, such as detailed error text or business-specific metadata.

For that reusable component, I would prefer either a Fabric User Data Function or a Stored Procedure activity over a notebook if the job is simply to insert an audit row.

That keeps the logging lightweight and avoids starting a Spark session just to write a few values.

A practical pattern could be:

pipeline/activity runs
-> Workspace Monitoring
-> central monitoring/audit table

and on failure:

failure branch
-> reusable logging function/procedure
-> persist detailed error message + any custom metadata

That gives you one central audit design without having to duplicate full logging logic across every pipeline.

AI-assisted drafting: AI was used to help structure and phrase this response. I reviewed and validated the technical content before posting.