Forum Discussion

DiKi-I's avatar
DiKi-I
Post Partisan
11 months ago
Solved

Best practice for error logging

Can someone share the best practices around pipeline creation like erro handling. logging etc in fabric.
one lake
  • neetesh91's avatar
    neetesh91
    11 months ago

    Hi DiKi-I : The following best practices can be applied for effective error logging in Fabric pipelines.

     

    1. Capture Errors with Try–Catch (On Failure Path)
     
    • Use the pipeline activity dependency (OnFailure, OnCompletion) to branch into an error-handling activity.
    • Example: If a Copy Data activity fails → trigger a Stored Procedure, Notebook, or Dataflow to log the error.
     
    2. Create a Centralized Error Logging Table in Lakehouse or Warehouse, below is one the example of schema for Error logging table.
     
    •         ActivityName
    •         RunId
    •         ErrorCode
    •         ErrorMessage
    •         ActivityType
    •         StartTime, 
    •         EndTime
    •         FailureTimestamp
    •         InputParameters / SourceTarget (optional, but may be useful in debugging)
     
    3.Similar to ADF, Fabric pipelines also expose system variables that can be used to capture runtime values in the error log table. A few examples are listed below.
     
    • @pipeline().RunId
    • @pipeline().DataFactory
    • @activity().Activity
    • @activity().Error.Message
    • @utcNow()
     
    4. Build a proc/notebook/Dataflow to write error logs and trigger it via OnFailure or OnCompletion .
     
     
    Thanks.
v-venuppu's avatar
v-venuppu
Community Support
11 months ago

Hi DiKi-I ,

Thank you for reaching out to Microsoft Fabric Community.

Thank you neetesh91 Udo_S for the prompt response.

I wanted to check if you had the opportunity to review the information provided and resolve the issue..?Please let us know if you need any further assistance.We are happy to help.

Thank you.