Forum Discussion

DiKi-I's avatar
DiKi-I
Post Partisan
11 months ago
Solved

Best practice for error logging

Can someone share the best practices around pipeline creation like erro handling. logging etc in fabric.
one lake
  • neetesh91's avatar
    neetesh91
    11 months ago

    Hi DiKi-I : The following best practices can be applied for effective error logging in Fabric pipelines.

     

    1. Capture Errors with Try–Catch (On Failure Path)
     
    • Use the pipeline activity dependency (OnFailure, OnCompletion) to branch into an error-handling activity.
    • Example: If a Copy Data activity fails → trigger a Stored Procedure, Notebook, or Dataflow to log the error.
     
    2. Create a Centralized Error Logging Table in Lakehouse or Warehouse, below is one the example of schema for Error logging table.
     
    •         ActivityName
    •         RunId
    •         ErrorCode
    •         ErrorMessage
    •         ActivityType
    •         StartTime, 
    •         EndTime
    •         FailureTimestamp
    •         InputParameters / SourceTarget (optional, but may be useful in debugging)
     
    3.Similar to ADF, Fabric pipelines also expose system variables that can be used to capture runtime values in the error log table. A few examples are listed below.
     
    • @pipeline().RunId
    • @pipeline().DataFactory
    • @activity().Activity
    • @activity().Error.Message
    • @utcNow()
     
    4. Build a proc/notebook/Dataflow to write error logs and trigger it via OnFailure or OnCompletion .
     
     
    Thanks.
v-venuppu's avatar
v-venuppu
Community Support
11 months ago

Hi DiKi-I ,

Thank you for reaching out to Microsoft Fabric Community.

Thank you NandanHegde tayloramy for the prompt response.

Here are few Microsoft Documentations that might help:
1.You will find topics like related to Pipeline creation,Error handling and logging,Monitoring and troubleshooting,Best practices for building and managing pipelines in the below documentation.

Azure Data Factory Documentation - Azure Data Factory | Microsoft Learn

2.You will find topics like related to Data pipeline orchestration,Error handling and logging,Pipeline monitoring and troubleshooting in the below documentation:

Azure Synapse Analytics - Azure Synapse Analytics | Microsoft Learn

3.Go to below link and search for Microsoft Fabric.

Microsoft Learn: Build skills that open doors in your career

This will pull up the latest and most relevant information for Microsoft Fabric and its components, including pipeline management, logging, and error handling.