Forum Discussion

Eleanordum93's avatar
Eleanordum93
Regular Visitor
21 days ago
Solved

Fabric pipelines audit

Hi community, I am looking for a way to log pipelines details into an audit delta table. To be more specific, I want to log the start time, end time, status, error message if any etc. Having this by...
data pipeline
lakehouse
Notebook
  • tayloramy's avatar
    tayloramy
    21 days ago

    Hi Eleanordum93​,

    I would recommend creating a UDF that accepts the things you want to log as parameters, and then saves them to a warehouse or SQL database. THen you can call that UDF from the pipeline (or from a notebook, or from anywhere else really) to log something. 

    This is better than using a notebook because it doesn't require a spark session to be created so will be cheaper on compute/CU usage.  

Olufemi7's avatar
Olufemi7
Icon for Super User rankSuper User
21 days ago

Hello Eleanordum93​,

You may not need a notebook for this. Fabric Workspace Monitoring provides pipeline and activity-level execution logs, including status, timestamps, activity name and pipeline run ID.

You can query these logs and persist the fields you need into your own audit table.

Microsoft docs:
Enable workspace monitoring in Microsoft Fabric