Detailed (automated) information/logging on copy job run

[JVL2]

we are using the copy job (preview) item. Suppose the copy job copies 4 tables, and one table fails; the status of the copy job is 'succeeded'. I need to know whether all tables have been copied succesfully or not. I know opening the monitoring tab, but this is not feasable when having a lot of copy jobs. Therefore I am searching for an automated way to achieve this.

what I tried:
- setup azure log analytics workspace and couple this with the fabric workspace. I see logging appearing for notebook activity however no logging at all when running the copy job
- I tried using the fabric rest api. I am able to get information about the copy job itself (using:"/v1/workspaces/{workspace_id}/items/{itemId}/jobs/instances" ) but not about the status of the tables.

How can I get info/logging on the status of each separate table within a copy job? Preferably using the fabric rest api

thanks!

[JVL2]

Hi @nilendraFabric ,

During my search I also found this endpoint. However, I always get an empty list as response. For job_id I used 'runId' (tried all other ID's I could find too but to no availe).

In my workspace, I only have Copy Jobs. When calling this endpoint: 

f"/v1/workspaces/{workspace_id}/datapipelines" -> an empty list is returned.
When calling this endpoint: 

f"/v1/workspaces/{workspace_id}/copyjobs" -> a list with all Copy Jobs is returned.

So I guess that calling: POST https://api.fabric.microsoft.com/v1/workspaces/<your WS Id>/datapipelines/pipelineruns/<job id>/queryactivityruns -> will only work for datapipelines, not for copy jobs.

Does anybody knows what the endpoint is to get the query activity information for a copy job?

[nilendraFabric]

GET https://api.fabric.microsoft.com/v1/workspaces/{workspaceId}/items/{itemId}/jobs/instances/{jobInsta...

The endpoint /v1/workspaces/{workspace_id}/copyjobs returns a list of Copy Jobs, which is the correct endpoint for accessing Copy Job information

This endpoint will provide you with information about a specific Copy Job run, including its status, start time, end time, and other relevant details

Please give it a try once.

[JVL2]

Indeed, your endpoint provides information about the copy job itself. However, as mentioned in my first message, I want to have information on the table activities within this copy job. Unfortunately, the endpoint you provided does not return this info. 

In the UI, when opening a copy job, there the information I want is present on a per table level. So somehow it should be possible to fetch this info. Altough, I do not know how.

[nilendraFabric · Accepted solution]

@JVL2 ,

Apologies for not reading the query properly.

There isn't a direct API endpoint to retrieve the detailed per-table information for copy jobs in Microsoft Fabric. This is a limitation of the current API implementation. The copy job feature is also relatively new, having been announced in public preview in late 2024, and the API capabilities are still evolving.


As a workaround, you could implement custom logging within your pipeline. For example, you could use a Web activity or a Notebook activity to query the source and destination after each copy job run, and log the results to a custom location.

I would recommend submit a feature request to the Microsoft Fabric team. As the product is still in development, they may be open to adding this functionality to the API in future updates.

 https://ideas.fabric.microsoft.com/

Please accept this solution and give kudos if this is helpful. Accepted solution will help the community as well.

Thanks
Nilendra

[JVL2]

@nilendraFabric  thanks for the input. I will give your proposed solution a try. The main issue in fact is that when running a copy job and one table fails, the status returned for the copy job is 'completed' which should be in my opninion be 'failed'. That way there should be no need to know the status of each individual table.

[nilendraFabric]

Hi @JVL2 

there is no direct way to get detailed automated information or logging on the status of individual tables within a copy job using the Microsoft Fabric REST API. 

worth trying this:

 Use the Query Activity Runs API:
You can use the Query Activity Runs API to get more detailed information about the copy job execution. This API allows you to retrieve information about individual activities within a pipeline run, which may include details about table-level operations.

POST https://api.fabric.microsoft.com/v1/workspaces/<your WS Id>/datapipelines/pipelineruns/<job id>/queryactivityruns

if this helps please accept the solution and give kudos.

Thanks

Nilendra