Pipeline "Copy Data" failing after over 24 hours

[Subdivide7365]

In Power BI Data Pipeline, I have a simple pipeline with a single "Copy Data" action.  The action is copying data from a REST API to a Lakehouse table in my workspace.

If I use the "Preview data" function (within the action's "Source" tab), the data is retreived in about 10 seconds.  If I run the pipeline, it fails after over 24 hours, with the error code 500 System Error: "InternalServerError executing pipeline."

This action is a duplicate of other actions (using the same API data source) that "copy data" to the Lakehouse successfully.

I am currently on the trial, if that affects performance.

Any insights?  Thanks in advance!

[Subdivide7365]

"Could it be that there is a difference with this particular API that you are using?"
What sort of difference?  It is a different API endpoint.  The endpoint that has successful pipeline actions is from the same provider of the endpoint that has failed pipeline actions.  Also, the results show successfully via the "Preview data" function.  I'm simply wanting Pipeline to intake the data seen in the "Preview data" function.

"And what happens if you had to create a new data pipeline from scratch? Does this thing work successfully?"
The same (delayed and failed) results occur when creating a new data pipeline from scratch.

[GilbertQ]

Hi @Subdivide7365 

There must be something different with this API, because all the other pipelines are working it does appear that the difference is the API?

[Subdivide7365]

Thank you for your response @GilbertQ.

It is a different API endpoint, so there will innately be something different with the API, but no difference in they way it is called.  It uses the same source connection, authentication headers, etc.  Are there differences I should look for?  The only perceivable difference is the source's "relative url", as required.

These things make me think it's not an issue with the API:
1. Using the Pipeline's "Preview data" function accurately displays the data I would like the pipeline to intake.  I just want that data.
2. The error is "InternalServerError executing..." (see original post) after a status of "In progress" for over 24 hours would seem to indicate that it is an action queueing/intake/processing issue.

Is there a way for Microsoft support to look into the "Pipeline run ID" to get further details on the generic error?  I'd want to get this resolved before investing further in the platform via paid support.

Thanks again in advance.

[GilbertQ]

Hi @Subdivide7365 

Could it have something to do with the Resident Evil Alpa and you might have to do some pagination in order to loop through the data being received from the API.?

[Subdivide7365]

@GilbertQ 

"Could it have something to do with the Resident Evil Alpa..."
What?

"you might have to do some pagination in order to loop through the data"

How can I loop through data that has not been received?

@v-cboorla-msft@Anonymous: I am interested in your service, although server errors (above) nor support are not yet providing any usable leads. I would like to receive support on this seemingly simple service issue without first investing in the service.  Possible?

[GilbertQ]

Hi @Subdivide7365 

Could it be that there is a difference with this particular API that you are using? And what happens if you had to create a new data pipeline from scratch? Does this thing work successfully?