Forum Discussion

Eleanordum93's avatar
Eleanordum93
Regular Visitor
21 days ago
Solved

Fabric pipelines audit

Hi community, I am looking for a way to log pipelines details into an audit delta table. To be more specific, I want to log the start time, end time, status, error message if any etc. Having this by...
data pipeline
lakehouse
Notebook
  • tayloramy's avatar
    tayloramy
    21 days ago

    Hi Eleanordum93​,

    I would recommend creating a UDF that accepts the things you want to log as parameters, and then saves them to a warehouse or SQL database. THen you can call that UDF from the pipeline (or from a notebook, or from anywhere else really) to log something. 

    This is better than using a notebook because it doesn't require a spark session to be created so will be cheaper on compute/CU usage.  

Eleanordum93's avatar
Eleanordum93
Regular Visitor
13 days ago

Hi everyone! Thank you for your kind help! Being FY end, my time was limited, but will work on a solution taking into consideration your answers and will come back! 
I am thinking to work on the stored procedure that was suggested.

  • v-aatheeque's avatar
    v-aatheeque
    Icon for Community Support rankCommunity Support
    9 days ago

    Hi Eleanordum93​ 

    Following up to confirm if the earlier responses addressed your query. If not, please share your questions and we’ll assist further.