Forum Discussion

Eleanordum93's avatar
Eleanordum93
Regular Visitor
1 month ago
Solved

Fabric pipelines audit

Hi community, I am looking for a way to log pipelines details into an audit delta table. To be more specific, I want to log the start time, end time, status, error message if any etc. Having this by...
data pipeline
lakehouse
Notebook
  • tayloramy's avatar
    tayloramy
    1 month ago

    Hi Eleanordum93​,

    I would recommend creating a UDF that accepts the things you want to log as parameters, and then saves them to a warehouse or SQL database. THen you can call that UDF from the pipeline (or from a notebook, or from anywhere else really) to log something. 

    This is better than using a notebook because it doesn't require a spark session to be created so will be cheaper on compute/CU usage.  

tayloramy's avatar
tayloramy
Icon for Super User rankSuper User
1 month ago

Hi Eleanordum93​,

I would recommend creating a UDF that accepts the things you want to log as parameters, and then saves them to a warehouse or SQL database. THen you can call that UDF from the pipeline (or from a notebook, or from anywhere else really) to log something. 

This is better than using a notebook because it doesn't require a spark session to be created so will be cheaper on compute/CU usage.