Forum Discussion
Solved
Get data from REST API source with iterated nested API calls (with Data Pipelines)
I have a working Dataflow that does the following: 1. Pulls a Customers list from an API call (https://api.somecrm.com/v1/customers). 2. For EACH Customer it calls another API to get his/hers purch...
- First, you need to set up a WEB Activity that pulls the customer list by calling the API/customers.
- Once you have the customer list, you have to set up a ForEach Activity.
- In ForEach Activity, go to Settings -> Items -> Add Dynamic Content. Array(activity([NAMEOFWEBACTIVITY]).output.data)
- Here, you need to access the data from JSON.
- Now, add a copy activity inside the ForEach Activity in order to save the JSON files into the lakehouse directly.
- To access the JSON objects via "customerId" go to the CopyActivity inside ForEach Activity.
- Source -> Add the connection here.
- In Realtive URL -> Add dynamic content @concat('purchaselist/',item().customerId)
- Set up the destination where you want to save the JSON objects as JSON files and run the pipeline.
You should specify the json property name (or path if its nested under a different top level property), which yields the collection of customers.
In general, one good way to debug is by providing a dummy property, and at runtime the failure message will indicate to some extent what properties were available to choose from.
Exactly my question.
What is the exact syntax for specifying the json path+property (it is nested).
I've googled but couldn't find samples.
Also, sholud I use Web activity for the first call? I saw people use Lookup activity, too.
Which one is the correct?