Forum Discussion

iBusinessBI's avatar
iBusinessBI
Kudo Collector
3 years ago
Solved

Get data from REST API source with iterated nested API calls (with Data Pipelines)

I have a working Dataflow that does the following: 1. Pulls a Customers list from an API call (https://api.somecrm.com/v1/customers). 2. For EACH Customer it calls another API to get his/hers purch...
Need Help
  • Ayybee's avatar
    Ayybee
    2 years ago
    1. First, you need to set up a WEB Activity that pulls the customer list by calling the API/customers.
    2. Once you have the customer list, you have to set up a ForEach Activity.
    3. In ForEach Activity, go to Settings -> Items -> Add Dynamic Content.   Array(activity([NAMEOFWEBACTIVITY]).output.data)
    4. Here, you need to access the data from JSON. ​
    5. Now, add a copy activity inside the ForEach Activity in order to save the JSON files into the lakehouse directly.
    6. To access the JSON objects via "customerId" go to the CopyActivity inside ForEach Activity.
    7. Source -> Add the connection here.
    8. In Realtive URL -> Add dynamic content  @concat('purchaselist/',item().customerId)​
    9. Set up the destination where you want to save the JSON objects as JSON files and run the pipeline.
HimanshuS-msft's avatar
HimanshuS-msft
Microsoft Employee
3 years ago

Hello iBusinessBI 

Can you please elaborate 
"I tried to build but got stuck in the ForEach activity with a syntax - how do I tell it how to get a specific customerID."?
I am confused as  the FOREACH will iterate all the CustimerID and not any specific one .
Thanks 
Himanshu


 

iBusinessBI's avatar
iBusinessBI
Kudo Collector
3 years ago

HimanshuS-msft 
Sure.

Here is my screenshot.

The Web1 activity calls an API that returns JSON with many customers including id and other fields.

For every customerid I need to make another API call.

What do I write in "Items"?

Thanks

 

 

  • Gregory_Pinard's avatar
    Gregory_Pinard
    New Member
    3 years ago

    Try @activity('Web1').output.value
    Then, in your nested activities, you can have a parameter referencing customer id and other fields with this pipeline expression : @item().customerid

    • PanuO's avatar
      PanuO
      Helper II
      2 years ago

      Is it possible to use the @item() properties in mapping? I tried directly use the iteration variable, but I got an error about typing. The UI does not allow manually setting type, so next I tried to create dynamic mapping and define the type there, but then I got an error about invalid JSON (even though JSON is valid). 

      "source": {
      "path": "@item().id"
      },

      "JsonException,Message=Array index expected."