topic Re: Pipeline fails/loads for REST API and mapping by import schema does not work in Data Engineering

Pipeline fails/loads for REST API and mapping by import schema does not work

Maz_Nah — Thu, 02 Jan 2025 16:46:37 GMT

I have created a new data pipeline and using REST to get data from an open news website and the limit of 1000 is follwed in the url. The issue or the help I need is

1. even though Preview URL shows the data but it does not execute successfully after changing dates 10.Dec to 15.Dec works and 15.Dec to 20.Dec does not

2. Mapping never works . Upon clickig Import schema as in screenshot it fails so a unable to map the coloumn

REST API

Connection https://api.reliefweb.int/

Relative URL /v1/reports?query%5Bvalue%5D=date.created%3A%5B2024-12-20+TO+2024-12-25%7D+AND+language.id%3A267&limit=1000&offset=0&profile=list&fields[include][]=body

Destination to a data lake house

Kinldy advice

Re: Pipeline fails/loads for REST API and mapping by import schema does not work

lbendlin — Thu, 02 Jan 2025 19:09:59 GMT

seems to work fine?

What's the error message?

Re: Pipeline fails/loads for REST API and mapping by import schema does not work

Maz_Nah — Thu, 02 Jan 2025 19:52:02 GMT

Hi,

Though it works in preview data but does not execute on run and mapping does not get the schema either

Error

Operation on target Pip_relief_LH failed: ErrorCode=HttpFileFailedToRead,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Failed to read data from http server. Check the error from http server：Unable to read data from the transport connection: The connection was closed.,Source=Microsoft.DataTransfer.ClientLibrary,''Type=System.IO.IOException,Message=Unable to read data from the transport connection: The connection was closed.,Source=System,'

Re: Pipeline fails/loads for REST API and mapping by import schema does not work

v-achippa — Fri, 03 Jan 2025 14:21:40 GMT

Hi @Maz_Nah,

Thank you for reaching out to Microsoft Fabric Community.

The issue occurs because, during the execution the pipeline attempts to fetch all the data for the specified date range and the connection is closed by the server due to resource constraints or timeouts. The API limits the data request based on the range or volume, causing the connection to be interrupted, and the pipeline fails to handle it properly.
The error indicates an HTTP failure, when attempting to retrieve data from the API endpoint and it suggests that the connection between the pipeline and the API server was interrupted or terminated unexpectedly, this can be possibly due to:
- Server-side restrictions on the API.
- Network issues or timeout during the data retrieval.
- Exceeding API rate limits or query complexity
Break down the date range into smaller intervals (for example 2–3 days) to reduce the load on the API and avoid timeout issues.
The Import Schema operation fails because the schema definition is not retrieved correctly from the REST API, so define a schema manually.
Check the API documentation for rate-limiting policies and ensure your pipeline does not exceed the allowed limits.
Configure your HTTP dataset or REST linked service to include a timeout and retry policy to handle intermittent connection issues:
- Timeout: Increase the timeout period to accommodate large data retrievals.
- Retry Policy: Set the retry count to handle temporary/transient failures.

If this post helps, then please consider Accepting as solution to help the other members find it more quickly, don't forget to give a "Kudos" – I’d truly appreciate it!

Thanks and regards,

Anjan Kumar Chippa

Re: Pipeline fails/loads for REST API and mapping by import schema does not work

Maz_Nah — Sun, 05 Jan 2025 08:08:51 GMT

thanks will try it