Power Query M - API Pagination

[CMS-001]

Hello,

I have the following query:

let

postedFrom = DateTime.ToText(Date.AddDays(DateTime.LocalNow(),-364),"MM/dd/yyyy"),

postedTo = DateTime.ToText(DateTime.LocalNow(),"MM/dd/yyyy"),

Source = Json.Document(Web.Contents("https://api.sam.gov/prod/opportunities/v1/search?limit=1000&api_key=[Key]&postedFrom="&postedFrom&"&postedTo="&postedTo&"&offset=")),

opportunitiesData = Source[opportunitiesData],

#"Converted to Table" = Table.FromList(opportunitiesData, Splitter.SplitByNothing(), null, null, ExtraValues.Error),

#"Expanded Column1" = Table.ExpandRecordColumn(#"Converted to Table", "Column1", {"noticeId", "title", "solicitationNumber", "department", "subTier", "office", "postedDate", "type", "baseType", "archiveType", "archiveDate", "typeOfSetAsideDescription", "typeOfSetAside", "responseDeadLine", "naicsCode", "classificationCode", "active", "award", "pointOfContact", "description", "organizationType", "officeAddress", "placeOfPerformance", "additionalInfoLink", "uiLink", "links", "resourceLinks"}, {"noticeId", "title", "solicitationNumber", "department", "subTier", "office", "postedDate", "type", "baseType", "archiveType", "archiveDate", "typeOfSetAsideDescription", "typeOfSetAside", "responseDeadLine", "naicsCode", "classificationCode", "active", "award", "pointOfContact", "description", "organizationType", "officeAddress", "placeOfPerformance", "additionalInfoLink", "uiLink", "links", "resourceLinks"}),

#"Expanded officeAddress" = Table.ExpandRecordColumn(#"Expanded Column1", "officeAddress", {"zipcode", "city", "countryCode", "state"}, {"officeAddress.zipcode", "officeAddress.city", "officeAddress.countryCode", "officeAddress.state"}),

#"Expanded placeOfPerformance" = Table.ExpandRecordColumn(#"Expanded officeAddress", "placeOfPerformance", {"state", "zip", "country"}, {"placeOfPerformance.state", "placeOfPerformance.zip", "placeOfPerformance.country"}),

#"Expanded placeOfPerformance.state" = Table.ExpandRecordColumn(#"Expanded placeOfPerformance", "placeOfPerformance.state", {"code", "name"}, {"placeOfPerformance.state.code", "placeOfPerformance.state.name"}),

#"Expanded placeOfPerformance.country" = Table.ExpandRecordColumn(#"Expanded placeOfPerformance.state", "placeOfPerformance.country", {"code", "name"}, {"placeOfPerformance.country.code", "placeOfPerformance.country.name"})

in

#"Expanded placeOfPerformance.country"

and I am trying to copy what was accomplished on this post here but I am struggling with how to implement the pagination.

• The record limit per page is 1,000
• The totalRecords returns ~ 46,000 records

See image below:

My attempts to modify the code from that other post have thus far not worked. My goal is to generate a table with all the records that looks like this:

Any suggestions? Documentation for this API can be found here.

[Anonymous]

Hi @CMS-001 ,

Please refer the following documentation to handle it.

https://datachant.com/2016/06/27/cursor-based-pagination-power-query/

Best Regards

Rena

[CMS-001]

Hello Rena,

Thank you for your reply! Unfortunately, that blog post details cursor-based pagination whereas the API I am working with provides pagination via offset.

Best Regards,

Dan