Forum Discussion

jbruewer's avatar
jbruewer
Helper I
6 years ago
Solved

REST api request and loop by offset until no further records found

Hi,   i am facing the challange of requesting data from a REST service that limits the records by 250 without sharing how many records should retrieved at all. So i am not able to calculate in adva...
Need Help
  • artemus's avatar
    artemus
    6 years ago

    First, check the request metadata to see if it tells you how many records there are. 

    E.g. 

    let

       webData = Web.Contents("https://..."),

       webMetadata = Value.Metadata(webData)

    in

       webMetadata

     

    For doing a while loop in power you can instead use recursive functions like:

    let
   my_func = (startIndex) =>
      let
          webRequest = Web.Contents(...),
         ....,
         results = ...,
         numRecords = ....,
         if numRecords = page_size then
             results & my_func(startIndex + page_size)
         else
              results
in
   my_func(0)

     Note the @ used for recursion

artemus's avatar
artemus
Microsoft Employee
6 years ago

First, check the request metadata to see if it tells you how many records there are. 

E.g. 

let

   webData = Web.Contents("https://..."),

   webMetadata = Value.Metadata(webData)

in

   webMetadata

 

For doing a while loop in power you can instead use recursive functions like:

let
   my_func = (startIndex) =>
      let
          webRequest = Web.Contents(...),
         ....,
         results = ...,
         numRecords = ....,
         if numRecords = page_size then
             results & my_func(startIndex + page_size)
         else
              results
in
   my_func(0)

 Note the @ used for recursion

Anonymous's avatar
Anonymous
Not applicable
4 years ago

Thanks artemus for this, exactly what I was after.

Only one thing for the next reader, you forgot an "in" after your numRecords =

Query should be:

let
   my_func = (startIndex) =>
      let
          webRequest = Web.Contents(...),
         ....,
         results = ...,
         numRecords = ....,
      in
         if numRecords = page_size then
             results & @my_func(startIndex + page_size)
         else
              results
in
   my_func(0)