Forum Discussion
Solved
Get data from REST API source with iterated nested API calls (with Data Pipelines)
I have a working Dataflow that does the following: 1. Pulls a Customers list from an API call (https://api.somecrm.com/v1/customers). 2. For EACH Customer it calls another API to get his/hers purch...
- First, you need to set up a WEB Activity that pulls the customer list by calling the API/customers.
- Once you have the customer list, you have to set up a ForEach Activity.
- In ForEach Activity, go to Settings -> Items -> Add Dynamic Content. Array(activity([NAMEOFWEBACTIVITY]).output.data)
- Here, you need to access the data from JSON.
- Now, add a copy activity inside the ForEach Activity in order to save the JSON files into the lakehouse directly.
- To access the JSON objects via "customerId" go to the CopyActivity inside ForEach Activity.
- Source -> Add the connection here.
- In Realtive URL -> Add dynamic content @concat('purchaselist/',item().customerId)
- Set up the destination where you want to save the JSON objects as JSON files and run the pipeline.
Try @activity('Web1').output.value
Then, in your nested activities, you can have a parameter referencing customer id and other fields with this pipeline expression : @item().customerid
Is it possible to use the @item() properties in mapping? I tried directly use the iteration variable, but I got an error about typing. The UI does not allow manually setting type, so next I tried to create dynamic mapping and define the type there, but then I got an error about invalid JSON (even though JSON is valid).
"source": {
"path": "@item().id"
},
"JsonException,Message=Array index expected."