Forum Discussion
Best practice for error logging
Hi DiKi-I : The following best practices can be applied for effective error logging in Fabric pipelines.
1. Capture Errors with Try–Catch (On Failure Path)- Use the pipeline activity dependency (OnFailure, OnCompletion) to branch into an error-handling activity.
- Example: If a Copy Data activity fails → trigger a Stored Procedure, Notebook, or Dataflow to log the error.
2. Create a Centralized Error Logging Table in Lakehouse or Warehouse, below is one the example of schema for Error logging table.- ActivityName
- RunId
- ErrorCode
- ErrorMessage
- ActivityType
- StartTime,
- EndTime
- FailureTimestamp
- InputParameters / SourceTarget (optional, but may be useful in debugging)
3.Similar to ADF, Fabric pipelines also expose system variables that can be used to capture runtime values in the error log table. A few examples are listed below.- @pipeline().RunId
- @pipeline().DataFactory
- @activity().Activity
- @activity().Error.Message
- @utcNow()
4. Build a proc/notebook/Dataflow to write error logs and trigger it via OnFailure or OnCompletion .Thanks.
Hi DiKi-I ,
As some members have already pointed out, there is probably not a single best practice, but I will describe an approach we are following.
I will explain this in detail and with a concrete example, since I believe this is more helpful than a generic description.
The example looks as follows:
1. I have a pipeline ("PIP_Run_DF2_TST") that executes a Dataflow Gen2 (could also be a Notebook or a Copy activity) on a schedule.
2. If the execution of the dataflow fails, the error message along with a timestamp should be written into a Lakehouse table.
3. From this table, you can then, for example, create a simple semantic model and visualize it in a Power BI dashboard.
The example consists of these components:
1. A dataflow "DF2_Do_Something_TST" that should be monitored for errors.
2. A pipeline "PIP_Run_DF2_TST" that executes the dataflow on a schedule (e.g., hourly).
3. A notebook "NB_ErrorLog_TST", to which the error parameters are passed from the pipeline.
4. A Lakehouse "LH_Error_Events_TST" with a table "errorlog", into which the error parameters should be continuously written.
Let’s assume you want these values to be returned in case of an error and stored in the Lakehouse:
EventTime: Timestamp (Datetime) of the error
DataflowName: Name of the dataflow whose execution failed
ErrorDescription: Description of the error
First create an empty Lakehouse "LH_Error_Events_TST".
The Lakehouse table "errorlog" can, for example, be created via a temporary PySpark notebook used only to create the table (with the Lakehouse connected as data item):
%%sql
CREATE TABLE IF NOT EXISTS dbo.errorlog(
EventTime timestamp,
DataflowName STRING,
ErrorDescription STRING
) USING DELTA
Then create an (initially) empty notebook "NB_ErrorLog_TST", connected to the Lakehouse as a data item.
Next, create a pipeline with a Dataflow activity and a Notebook activity.
The Notebook activity is connected to the Dataflow activity via the On fail output.
The Notebook activity is linked to the (still empty) notebook "NB_ErrorLog_TST".
Now create three parameters of type "String" (Datetime is not available here) in the Notebook activity with these values:
| Name | Value | Description |
| p_event_ts | @utcNow() | Timestamp of the error |
| p_dataflow_name | DF2_Do_Something_TST | Name of the dataflow (hardcoded) |
| p_error | @string(activity('Dataflow1').Error.Message) | Error message |
The pipeline should look like this:
(A detailed list of parameter-values can be found here: https://learn.microsoft.com/en-us/fabric/data-factory/expression-language)
Finally open the notebook "NB_ErrorLog_TST", which receives the error parameters from the pipeline and writes them into the Lakehouse.
The notebook concists of three simple code cells:
1. A cell with the import statements.
2. A cell with the parameter definitions.
3. A cell in which the parameter values from the pipeline are written into a dataframe, which is then appended as a new row to the Lakehouse table.
Cell 1:
# Import statements
from datetime import datetime
from pyspark.sql import Row
from delta.tables import DeltaTableCell 2:
# Parameters with random default-values - parameter-names must be excactly the same as the pipeline-paramter-names
# "Toggle parameter cell" must be activated
p_events_ts = datetime.now()
p_dataflow_name = "Default DataflowName"
p_error = "Default Error"Important: "Toggle parameter cell" must be activatet for cell 2.
Cell 3:
# At "On fail" error-values are passed over from pipeline to notebook and can be stored in a dataframe
df = spark.createDataFrame([
Row(EventTime=p_events_ts,
DataflowName=p_dataflow_name,
ErrorDescription=p_error)
])
df.write.mode("append").saveAsTable("errorlog") # Dataframe with error-values will be appended to table "errorlog"
That's it.
The only thing left to do is to schedule the pipeline and wait for an error to occur.
If an error occurs ("On fail"), the error values are passed from the pipeline to the notebook, and then from the notebook to the lakehouse table.
You can reuse the notebook in as many pipelines as you like, and you can extend the parameters to suit your needs. And you can replace the dataflow activity with any activity that has an "On fail" output.
And of course, you are not limited to the "On fail" output. Instead, you can also connect the notebook to "On success" or "On completion" and adjust the parameters accordingly.
I hope this helps.
Best regards,
Udo