Data Ingestion using API

[Rafaela07]

Hi everyone,

I'm working on a new project in Fabric and I'm trying to ingest data from an external API into a Lakehouse using Data Factory Copy Activity.
The client provided the url, and a username and password for basic authentication. They also provided some parameters, eg ?limit=100
For the copy activity I created a rest connection using the url and the authentication credentials and the connectivity test is successful. In the relative url part I've put something like /api/campaigns?limit=50 and I tried to preview the respective data.
However I'm bumping into the following error:
Rest call failed with server error, status code 500 InternalServerError, please check your activity settings and remote server issues.
Request URL: XXXXXXXXXXXXXXXX/api/campaigns?limit=50.
Response: {"errors":[{"message":"Looks like I encountered an error (error #500). If I do it again, please report me to the system administrator!","code":500,"type":null}]}"

Am I missing something? I've never ingested data using apis before so I'm not sure what is going wrong.
Any suggestions regarding this situation?

Thank you in advance,

[deborshi_nag]

Hi @Rafaela07 

I would suggest trying out the REST endpoint using a desktop tool like Postman or Insomnia. 

Hope this helps - please appreciate by leaving a Kudos or accepting as a Solution! 

[tayloramy]

Hi @Rafaela07, 

A 500 error means that you successfully connected to the API, but something on the backend of the API went wrong. 

The steps I would take to troubleshoot are: 
1: create a text notebook and see if you can get data from the API using Python. That gives you far more flexibility to know exactly what URL is being hit. 

2: If the notebook encounters the same issue, contact the API vendor 

3: If the notebook works, then there is something different between the URL the pipeline is using and the URL the notebook is using.