Fabric pipelines audit

Hi Eleanordum93​ 

Thanks for sharing your scenario. The suggestions above provide a good starting point for building a centralized audit solution across multiple Fabric pipelines.

I would recommend using Workspace Monitoring for standard pipeline and activity-level telemetry, combined with a reusable logging component for any additional information you need to persist.

Workspace Monitoring exposes pipeline and activity execution details, including pipeline and activity names, run IDs, status, start and end timestamps, duration, and failure type. You can query these logs using KQL and use them for centralized monitoring and reporting.

One limitation to consider is that Workspace Monitoring does not currently provide full error details and diagnostics. If capturing the exact activity error message in your own audit table is a requirement, you can use a reusable logging component, such as a Fabric User Data Function or a Warehouse stored procedure, to persist the additional details without starting a Spark session solely for logging.

This hybrid approach helps avoid duplicating logging logic across pipelines while keeping the audit design maintainable.
For implementation details, please refer to the official Microsoft documentation:

Enable Workspace Monitoring (Preview) in Microsoft Fabric - Microsoft Fabric | Microsoft Learn
Choose How to Monitor Pipeline Runs - Microsoft Fabric | Microsoft Learn

I hope this helps you build a centralized and reusable pipeline auditing solution.