Forum Discussion

Eleanordum93's avatar
Eleanordum93
Regular Visitor
21 days ago
Solved

Fabric pipelines audit

Hi community, I am looking for a way to log pipelines details into an audit delta table. To be more specific, I want to log the start time, end time, status, error message if any etc. Having this by...
data pipeline
lakehouse
Notebook
  • tayloramy's avatar
    tayloramy
    21 days ago

    Hi Eleanordum93​,

    I would recommend creating a UDF that accepts the things you want to log as parameters, and then saves them to a warehouse or SQL database. THen you can call that UDF from the pipeline (or from a notebook, or from anywhere else really) to log something. 

    This is better than using a notebook because it doesn't require a spark session to be created so will be cheaper on compute/CU usage.  

Eleanordum93's avatar
Eleanordum93
Regular Visitor
8 days ago

Hi everyone! 
I started testing the Monitoring database and indeed it gives a lot of information, but i cannot link the FabricDataPipelinesActivityRunsLogs and ItemJobsEventsLogs. 
I.E: I have the activities from a pipeline with their status, but cannot see from which pipeline they are coming from. Couldn't find a lot of information this database related queries. 
I found some samples online, but not very helpful. Just trying to create a live report for now, to see what failed today, which step from which pipeline etc. Any thoughts? 

Thank you!