Automated/Native logging of pipeline activities into OneLake

[SanthoshKumar19]

Hi Fabric team,

I’m exploring options for monitoring and governance of pipelines in Microsoft Fabric. Currently, diagnostic logging seems to capture only pipeline-level metadata (run ID, status, start/end time, error messages, item kind, item name, duration).

My requirement is to have activity-level logs (e.g., Copy Data row counts, duration, Notebook outputs, start/end time, activity name etc.,) automatically captured for each activity inside a pipeline run, and then stored centrally in OneLake for later analysis purpose.

Is there any native/automated way in Fabric to log all activities inside each pipeline run (not just pipeline metadata)?

Thanks in advance for guidance!

[Lozovskyi]

Hello @SanthoshKumar19,

Since there isn't a native solution for this yet, I use the following workaround:

Immediately after the main Copy activity completes (triggered on both the success and failure paths), I run a secondary "dummy" Copy activity. This activity captures the execution logs and saves them as a JSON file to a designated location with a custom name.

From there, I create a Shortcut to that folder. Fabric seamlessly exposes those JSON files as a Delta table, making your log data instantly available for querying or reporting in a dashboard.

Hope this helps!

[stephan_samuel]

This is not automatic but I have a working solution in production using an Eventhouse and KQL. Create the Eventhouse and a table (run_id STRING, item_name STRING), then add a KQL activity with:

.ingest inline into table MyLogTable ["@{pipeline().RunId}","@{item().table_name}"]

This has a marginally better real-time contract than an, "ordinary," row insertion. The Eventhouse can undergo ELT if that's desirable.

[v-achippa]

Hi @SanthoshKumar19,

Thank you for reaching out to Microsoft Fabric Community.

Thank you @Tamanchu and @tayloramy for the prompt response.

As we haven’t heard back from you, we wanted to kindly follow up to check if the solution provided by the user's for the issue worked? or let us know if you need any further assistance.

Thanks and regards,

Anjan Kumar Chippa

[SanthoshKumar19]

Yes @v-achippa I could able to get the solution from the responses. Thank you

[tayloramy]

Hi @SanthoshKumar19, 

Fabric doesn't have a built in way to do this, but you can build it. 

WHat I've done is I have a UDF that takes in the output string of a copy data activity, parses out the information, and writes it to a warehouse. 
I then call that UDF after every copy data activity in my pipelines to capture all the data. A little kludgy, but functional. 

[SanthoshKumar19]

Thanks Taylor for your inputs 👍🏻

[Tamanchu · Accepted solution]

Hi @SanthoshKumar19,

Note: This response was enhanced with the assistance of an AI tool. The information has been verified against the official Microsoft Fabric documentation.

There's currently no fully native/automated way to push activity-level logs directly to OneLake Fabric's built-in diagnostic settings only capture pipeline-level metadata (run ID, status, duration, errors), not individual activity outputs.

Here are the two most practical approaches:

Option 1 : Instrument your pipelines with expressions (recommended)

After each key activity, add a Script activity (or a lightweight Notebook activity) that writes a row to a Delta table in your Lakehouse using pipeline expressions :

• @pipeline().RunId → pipeline run ID
• @pipeline().Pipeline → pipeline name
• @utcNow() → timestamp
• @activity('YourCopyActivity').output.rowsCopied → rows written
• @activity('YourCopyActivity').output.dataRead → bytes read
• @activity('YourCopyActivity').output.copyDuration → duration in seconds
• @activity('YourCopyActivity').status → Succeeded / Failed

You build a reusable "log writer" child pipeline that accepts these as parameters then call it from any pipeline after each key step.

→ Docs: Pipeline expression language – system variables

Option 2 : REST API post-run harvest

After each pipeline run completes, trigger a Notebook that calls the Fabric REST API to fetch the full activity run details and append them to your log table:

GET https://api.fabric.microsoft.com/v1/workspaces/{workspaceId}/items/{pipelineId}/jobs/{jobInstanceId}

This gives you structured activity-level output for any completed run.

→ Docs: Get Item Job Instance – Fabric REST API

Option 1 is simpler to set up and gives you real-time logging per activity. Option 2 is better if you want to keep your pipelines clean and harvest logs asynchronously.

[SanthoshKumar19]

Thanks for your quick response Tamanchu ! It helps a lot.