Forum Discussion
Fabric pipelines audit
Hi Eleanordum93,
I would recommend creating a UDF that accepts the things you want to log as parameters, and then saves them to a warehouse or SQL database. THen you can call that UDF from the pipeline (or from a notebook, or from anywhere else really) to log something.
This is better than using a notebook because it doesn't require a spark session to be created so will be cheaper on compute/CU usage.
Continued ContributorHi Eleanordum93,
Both good suggestions above. One thing that helps with having lots of pipelines is building a small reusable "logging" pipeline (or UDF, as tayloramy said) with parameters like pipeline name, run ID, activity name, status, start time, end time and error message. Then each pipeline just calls it with an Invoke Pipeline activity on the success and failure paths, so the logic lives in one place.
The values you pass in are mostly available as dynamic content already, for example @pipeline().Pipeline, @pipeline().RunId, @pipeline().TriggerTime and @activity('YourActivity').error.message on the failure branch.
If you want to keep it cheap, a Stored Procedure or Script activity writing to a Warehouse avoids spinning up Spark, and Warehouse tables are stored as Delta in OneLake anyway. You can also use Workspace Monitoring for the detailed history and keep your own table for just the fields you care about.