topic Re: View API response in copy activity in Pipelines

View API response in copy activity

nkailasamurthy — Fri, 06 Sep 2024 19:27:27 GMT

Hi there,

So here is my situation

I'm using a copy activity(Data pipeline - Fabric) to make an api call which gives me a json response which has to be parsed and loaded to a table in lakehouse. I have tried creating a file and a table from the response but i keep getting this error message

"ErrorCode=JsonInvalidDataFormat,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Error occurred when deserializing source JSON file
''. Check if the data is in valid JSON object format"

I dont know how to view the json response from this activity to see what is wrong with it.

When I make this API call using Web activity and see the output from the run history, I see nothing wrong with it. I then created a json file in lakehouse with the json response from web activity and used this as input for copy activity and it parses it just fine.

So my questions are

1. How do I view the JSON response from copy activity?

2. What is the best approach for this task prefereably without notebooks ?

Thanks in advance

Re: View API response in copy activity

nkailasamurthy — Fri, 06 Sep 2024 19:32:23 GMT

Attaching the output from web activity

Re: View API response in copy activity

FabianSchut — Sun, 08 Sep 2024 11:16:57 GMT

Hi nkailasamurthy,

What are your source and sink settings for the copy activity? Which connection type are you using (Web or Web v2 or another)?

For the JSON response, you could use the Preview data in the source settings to view the input. You can see if anything is broken there.

Re: View API response in copy activity

Anonymous — Tue, 17 Sep 2024 06:54:19 GMT

Hi nkailasamurthy ,

Below are my thoughts.

Firstly, to view the JSON response from a copy activity in Microsoft Fabric, you can follow these steps:

1. Ensure that debugging is enabled for your pipeline. This will allow you to see detailed logs and outputs.

2. Execute your pipeline with the copy activity.

3. After the pipeline run, go to the Monitor tab in the Data Factory. Here, you can see the run history of your pipeline.

3. Click on the specific copy activity within the pipeline run. You should see detailed logs and outputs, including the JSON response if the activity was successful.

Secondly, best Approach for Parsing and Loading JSON to Lakehouse, you can follow theses steps:

1. Web Activity: Use a web activity to make the API call and capture the JSON response.
2. Store JSON in Lakehouse: Save the JSON response to a file in your lakehouse.
3. Copy Activity: Use a copy activity to read the JSON file from the lakehouse and load it into your table.

Best Regards

Yilong Zhou

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.