Forum Discussion
Solved
Best practice for error logging
Can someone share the best practices around pipeline creation like erro handling. logging etc in fabric.
Hi DiKi-I : The following best practices can be applied for effective error logging in Fabric pipelines.
1. Capture Errors with Try–Catch (On Failure Path)- Use the pipeline activity dependency (OnFailure, OnCompletion) to branch into an error-handling activity.
- Example: If a Copy Data activity fails → trigger a Stored Procedure, Notebook, or Dataflow to log the error.
2. Create a Centralized Error Logging Table in Lakehouse or Warehouse, below is one the example of schema for Error logging table.- ActivityName
- RunId
- ErrorCode
- ErrorMessage
- ActivityType
- StartTime,
- EndTime
- FailureTimestamp
- InputParameters / SourceTarget (optional, but may be useful in debugging)
3.Similar to ADF, Fabric pipelines also expose system variables that can be used to capture runtime values in the error log table. A few examples are listed below.- @pipeline().RunId
- @pipeline().DataFactory
- @activity().Activity
- @activity().Error.Message
- @utcNow()
4. Build a proc/notebook/Dataflow to write error logs and trigger it via OnFailure or OnCompletion .Thanks.
Hi DiKi-I,
Here's how I manage this:
First I start a job from the API
POST v1/workspaces/{ws_id}/items/{item_id}/jobs/instances?jobType={job_type}
this request returns the job instance ID.
Then I am polling the API every 15 seconds or so to wait for a job to finish:
GET v1/workspaces/{ws_id}/items/{item_id}/jobs/instances/{job_instance_id}
Once the job's status is complete, the API endpoint above returns the metadata I am collecting with overall status (success, fail), start times, end times, failure reasons, etc.
Here are the API docs:
Job Scheduler - REST API (Core) | Microsoft Learn
If this helped, consider giving some kudos. If I answered your question or helped solve your problem, mark this post as the solution to help future forum users find it.