Best practice for error logging

Hi DiKi-I ,
As some members have already pointed out, there is probably not a single best practice, but I will describe an approach we are following.

I will explain this in detail and with a concrete example, since I believe this is more helpful than a generic description.

The example looks as follows:

1. I have a pipeline ("PIP_Run_DF2_TST") that executes a Dataflow Gen2 (could also be a Notebook or a Copy activity) on a schedule.

2. If the execution of the dataflow fails, the error message along with a timestamp should be written into a Lakehouse table.

3. From this table, you can then, for example, create a simple semantic model and visualize it in a Power BI dashboard.

The example consists of these components:

1. A dataflow "DF2_Do_Something_TST" that should be monitored for errors.

2. A pipeline "PIP_Run_DF2_TST" that executes the dataflow on a schedule (e.g., hourly).

3. A notebook "NB_ErrorLog_TST", to which the error parameters are passed from the pipeline.

4. A Lakehouse "LH_Error_Events_TST" with a table "errorlog", into which the error parameters should be continuously written.

Let’s assume you want these values to be returned in case of an error and stored in the Lakehouse:

• EventTime: Timestamp (Datetime) of the error

• DataflowName: Name of the dataflow whose execution failed

• ErrorDescription: Description of the error

First create an empty Lakehouse "LH_Error_Events_TST". 

The Lakehouse table "errorlog" can, for example, be created via a temporary PySpark notebook used only to create the table (with the Lakehouse connected as data item):

%%sql
CREATE TABLE IF NOT EXISTS dbo.errorlog(
    EventTime           timestamp,
    DataflowName        STRING,
    ErrorDescription    STRING
) USING DELTA

Then create an (initially) empty notebook "NB_ErrorLog_TST", connected to the Lakehouse as a data item.

Next, create a pipeline with a Dataflow activity and a Notebook activity.

The Notebook activity is connected to the Dataflow activity via the On fail output.

The Notebook activity is linked to the (still empty) notebook "NB_ErrorLog_TST".

Now create three parameters of type "String" (Datetime is not available here) in the Notebook activity with these values:

The pipeline should look like this:

(A detailed list of parameter-values can be found here: https://learn.microsoft.com/en-us/fabric/data-factory/expression-language)

Finally open the notebook "NB_ErrorLog_TST", which receives the error parameters from the pipeline and writes them into the Lakehouse.

The notebook concists of three simple code cells:

1. A cell with the import statements.

2. A cell with the parameter definitions.

3. A cell in which the parameter values from the pipeline are written into a dataframe, which is then appended as a new row to the Lakehouse table.

Cell 1:

# Import statements
from datetime import datetime
from pyspark.sql import Row
from delta.tables import DeltaTable

Cell 2:

# Parameters with random default-values - parameter-names must be excactly the same as the pipeline-paramter-names
# "Toggle parameter cell" must be activated
p_events_ts = datetime.now()
p_dataflow_name = "Default DataflowName"
p_error = "Default Error"

Important: "Toggle parameter cell" must be activatet for cell 2.

Cell 3:

# At "On fail" error-values are passed over from pipeline to notebook and can be stored in a dataframe
df = spark.createDataFrame([
    Row(EventTime=p_events_ts,
        DataflowName=p_dataflow_name,
        ErrorDescription=p_error)
])

df.write.mode("append").saveAsTable("errorlog") # Dataframe with error-values will be appended to table "errorlog"

That's it.

The only thing left to do is to schedule the pipeline and wait for an error to occur.
If an error occurs ("On fail"), the error values are passed from the pipeline to the notebook, and then from the notebook to the lakehouse table.
You can reuse the notebook in as many pipelines as you like, and you can extend the parameters to suit your needs. And you can replace the dataflow activity with any activity that has an "On fail" output.

And of course, you are not limited to the "On fail" output. Instead, you can also connect the notebook to "On success" or "On completion" and adjust the parameters accordingly.

I hope this helps.

Best regards,
Udo

Hi DiKi-I, 

The best practices I can mention is to be consistent. 

I've built a full job orchestration system to handle consistent metadata collection for all jobs (pipelines and notebooks supported so far) and to send error notificaitons. 

You can build these sorts of systems into pipelines, but if every developer does it slightly differently then you can run into problems consolidating the data later on, or you may run into the situation where one developer does not implement it at all. 

In my system, I start pipelines via the API, and then query the API for the pipeline status. Once it completes, I write metadata like how long it took, the result status (success, failure, etc) as well as the runID so I can dynamically build the URL to go to the run page. 
This way if something fails, I can send an email with the link and the developer can click the link to see exactly what part of the pipeline failed. 

If this helped, consider giving some kudos. If I answered your question or helped solve your problem, mark this post as the solution to help future forum users find it.   

Do you have any documentation around it, if you could share?

Hi DiKi-I, 

I don't have any documentation for my system that I can share yet, though I do plan to make some blogs posts about it once it is completed. 
I currently don't have permissions on the forum to make blog posts, so I need to continue posting and contributing to the forum until the admins deem me as worthy. 

At a high level, the orchestration reads a configuration "registry" of jobs (across Bronze, Silver, and Gold) that declares each item’s workspace, type (notebook or pipeline), targets, and dependencies (e.g., bronze:<id>, silver:<id>).

It resolves those dependencies into a DAG, kicks off everything it safely can in parallel, and enforces ordering where required; if a prerequisite fails, dependents are marked skipped due to dependency rather than run.

Launches use Fabric’s scheduler/REST APIs with a warm-up/poll loopto check the job status. As jobs progress, the orchestrator records start/end times, status (success/failed/skipped_dependency), identifiers, and other metadata into meta tables.

The whole thing is parameterized by environment (DEV/TEST/PROD) and layer so one notebook can drive all tiers, and it stays safe to re-run after partial failures.

If this helped, consider giving some kudos. If I answered your question or helped solve your problem, mark this post as the solution to help future forum users find it.  

Can you state as to which components within MSFT Fabric are you using? As the logging / error framework can differ based on your architectural flow and the offerings ( data pipelines, Dataflows ,notebooks etc) being used.

Hi DiKi-I ,

Thank you for reaching out to Microsoft Fabric Community.

Thank you NandanHegde tayloramy for the prompt response.

Here are few Microsoft Documentations that might help:
1.You will find topics like related to Pipeline creation,Error handling and logging,Monitoring and troubleshooting,Best practices for building and managing pipelines in the below documentation.

Azure Data Factory Documentation - Azure Data Factory | Microsoft Learn

2.You will find topics like related to Data pipeline orchestration,Error handling and logging,Pipeline monitoring and troubleshooting in the below documentation:

Azure Synapse Analytics - Azure Synapse Analytics | Microsoft Learn

3.Go to below link and search for Microsoft Fabric.

Microsoft Learn: Build skills that open doors in your career

This will pull up the latest and most relevant information for Microsoft Fabric and its components, including pipeline management, logging, and error handling.

Hi DiKi-I, 

Here's how I manage this: 

First I start a job from the API

Once the job's status is complete, the API endpoint above returns the metadata I am collecting with overall status (success, fail), start times, end times, failure reasons, etc. 

Here are the API docs: 
Job Scheduler - REST API (Core) | Microsoft Learn

If this helped, consider giving some kudos. If I answered your question or helped solve your problem, mark this post as the solution to help future forum users find it.  

Hi DiKi-I ,

Thank you for reaching out to Microsoft Fabric Community.

Thank you neetesh91 Udo_S for the prompt response.

I wanted to check if you had the opportunity to review the information provided and resolve the issue..?Please let us know if you need any further assistance.We are happy to help.

Thank you.

Hi DiKi-I ,

May I ask if you have resolved this issue? Please let us know if you have any further issues, we are happy to help.

Thank you.