HTTP GET request not working in Copy Data activity, although same URL works in Postman

[mrpmcmanus]

I am currently trying to submit a HTTP GET request in a Copy Data activity for a Data Pipeline in Data Factory. I have tried three different URLs from the three websites I am trying to copy data from, and am met with the same response each time: 

"Bad data is found at line 1 in source ... You can ignore bad data by setting BadDataFound to null. IReader state: ColumnCount: 27345 CurrentIndex: -1 HeaderRecord: IParser state: ByteCount: 0 CharCount: 1077997 Row: 1 RawRow: 1 Count: 27345 RawRecord: Hidden because ExceptionMessagesContainRawData is false."

I cannot edit the source data/schema as its coming from a read-only URL.

There is no setting "BadDataFound" within the Copy Data activity - rendering that option completely useless.

The "Fault tolerance" setting within the Copy Data activity is set to "Skip incompatible rows" and that has not helped.

This API GET request works completely fine within Postman.
Wondering what the issue is, if Data Factory doesn't actually support this or if I'm missing something completely.

[Anonymous · Accepted solution]

Hi @mrpmcmanus ,

Based on the error reporting information you provided, it suggests that there may be an issue with the data format or architecture. Also the data is read-only and cannot be edited, you may need to check the data format for consistency. For the error message mentioning “BadDataFound”, you may need to check other error handling options provided by Data Factory.

For configuring HTTP in copy data, you can refer to the following documentation:

Configure HTTP in a copy activity - Microsoft Fabric | Microsoft Learn

Best Regards,
Adamk Kong

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.