Forum Discussion

DiKi-I's avatar
DiKi-I
Post Partisan
11 months ago
Solved

Best practice for error logging

Can someone share the best practices around pipeline creation like erro handling. logging etc in fabric.
one lake
  • neetesh91's avatar
    neetesh91
    11 months ago

    Hi DiKi-I : The following best practices can be applied for effective error logging in Fabric pipelines.

     

    1. Capture Errors with Try–Catch (On Failure Path)
     
    • Use the pipeline activity dependency (OnFailure, OnCompletion) to branch into an error-handling activity.
    • Example: If a Copy Data activity fails → trigger a Stored Procedure, Notebook, or Dataflow to log the error.
     
    2. Create a Centralized Error Logging Table in Lakehouse or Warehouse, below is one the example of schema for Error logging table.
     
    •         ActivityName
    •         RunId
    •         ErrorCode
    •         ErrorMessage
    •         ActivityType
    •         StartTime, 
    •         EndTime
    •         FailureTimestamp
    •         InputParameters / SourceTarget (optional, but may be useful in debugging)
     
    3.Similar to ADF, Fabric pipelines also expose system variables that can be used to capture runtime values in the error log table. A few examples are listed below.
     
    • @pipeline().RunId
    • @pipeline().DataFactory
    • @activity().Activity
    • @activity().Error.Message
    • @utcNow()
     
    4. Build a proc/notebook/Dataflow to write error logs and trigger it via OnFailure or OnCompletion .
     
     
    Thanks.
tayloramy's avatar
tayloramy
Super User
11 months ago

Hi DiKi-I, 

 

I don't have any documentation for my system that I can share yet, though I do plan to make some blogs posts about it once it is completed. 
I currently don't have permissions on the forum to make blog posts, so I need to continue posting and contributing to the forum until the admins deem me as worthy. 

 

At a high level, the orchestration reads a configuration "registry" of jobs (across Bronze, Silver, and Gold) that declares each item’s workspace, type (notebook or pipeline), targets, and dependencies (e.g., bronze:<id>, silver:<id>).

It resolves those dependencies into a DAG, kicks off everything it safely can in parallel, and enforces ordering where required; if a prerequisite fails, dependents are marked skipped due to dependency rather than run.

 

Launches use Fabric’s scheduler/REST APIs with a warm-up/poll loopto check the job status. As jobs progress, the orchestrator records start/end times, status (success/failed/skipped_dependency), identifiers, and other metadata into meta tables.

The whole thing is parameterized by environment (DEV/TEST/PROD) and layer so one notebook can drive all tiers, and it stays safe to re-run after partial failures.

 

If this helped, consider giving some kudos. If I answered your question or helped solve your problem, mark this post as the solution to help future forum users find it.