Forum Discussion

DiKi-I's avatar
DiKi-I
Post Partisan
11 months ago
Solved

Best practice for error logging

Can someone share the best practices around pipeline creation like erro handling. logging etc in fabric.
one lake
  • neetesh91's avatar
    neetesh91
    11 months ago

    Hi DiKi-I : The following best practices can be applied for effective error logging in Fabric pipelines.

     

    1. Capture Errors with Try–Catch (On Failure Path)
     
    • Use the pipeline activity dependency (OnFailure, OnCompletion) to branch into an error-handling activity.
    • Example: If a Copy Data activity fails → trigger a Stored Procedure, Notebook, or Dataflow to log the error.
     
    2. Create a Centralized Error Logging Table in Lakehouse or Warehouse, below is one the example of schema for Error logging table.
     
    •         ActivityName
    •         RunId
    •         ErrorCode
    •         ErrorMessage
    •         ActivityType
    •         StartTime, 
    •         EndTime
    •         FailureTimestamp
    •         InputParameters / SourceTarget (optional, but may be useful in debugging)
     
    3.Similar to ADF, Fabric pipelines also expose system variables that can be used to capture runtime values in the error log table. A few examples are listed below.
     
    • @pipeline().RunId
    • @pipeline().DataFactory
    • @activity().Activity
    • @activity().Error.Message
    • @utcNow()
     
    4. Build a proc/notebook/Dataflow to write error logs and trigger it via OnFailure or OnCompletion .
     
     
    Thanks.
tayloramy's avatar
tayloramy
Super User
11 months ago

Hi DiKi-I, 

 

The best practices I can mention is to be consistent. 

I've built a full job orchestration system to handle consistent metadata collection for all jobs (pipelines and notebooks supported so far) and to send error notificaitons. 


You can build these sorts of systems into pipelines, but if every developer does it slightly differently then you can run into problems consolidating the data later on, or you may run into the situation where one developer does not implement it at all. 

 

In my system, I start pipelines via the API, and then query the API for the pipeline status. Once it completes, I write metadata like how long it took, the result status (success, failure, etc) as well as the runID so I can dynamically build the URL to go to the run page. 
This way if something fails, I can send an email with the link and the developer can click the link to see exactly what part of the pipeline failed. 

 

If this helped, consider giving some kudos. If I answered your question or helped solve your problem, mark this post as the solution to help future forum users find it.