Forum Discussion
Hubspot Private Apps - Powerquery pagination
Ignore my previous comments, ive got it working!
Not sure how many rows you were expecting but ive got well over a million. for that reason i'd highly recommend using the start/end date time stamps or event type to filter. Excel will really struggle otherwise. none the less, the code below will work.
let
baseuri = "https://api.hubapi.com/email/public/v1/events?&limit=1000&properties=recipient&properties=type&properties=portalId&properties=appId&properties=appName&properties=emailCampaignId&property=subject&property=dropReason&property=dropMessage&property=duration",
headers = [Headers=[#"Content-Type"="application/json", Authorization="Bearer pat-YOURTOKENHERE"]],
initReq = Json.Document(Web.Contents( baseuri, headers)),
initTable = Record.ToTable(initReq),
initData = initReq[events],
//We want to get data = {lastNPagesData, thisPageData}, where each list has the limit # of Records,
//then we can List.Combine() the two lists on each iteration to aggregate all the records. We can then
//create a table from those records
gather = (data as list, uri) =>
let
//get new offset from active uri
newOffset = Json.Document(Web.Contents(uri, headers))[offset],
//build new uri using the original uri so we dont append offsests
newUri = baseuri & "&offset=" & newOffset,
//get new req & data
newReq = Json.Document(Web.Contents(newUri, headers)),
newTable = Record.ToTable(newReq),
newdata = newReq[events],
//add that data to rolling aggregate
data = List.Combine({data, newdata}),
//if theres no next page of data, return. if there is, call @gather again to get more data
check = if newReq[hasMore] = true then
let
nextData = @gather(newdata, newUri)
in
List.Combine({data, nextData})
else
List.Combine({data, newdata})
in
check,
//before we call gather(), we want see if its even necessary. First request returns only one page? Return.
outputList = if initReq[hasMore] = true then
@gather(initData, baseuri)
else
initData
in
outputList
Let me know how you get on
cheers
Well, when I say work, I mean it will do what is as expected but depending on how long you have been using the crm it probably wont load to the workbook as there are too many rows.
You'll get something like this
As i say, id recommend using some filters in your URL
- JackSelman3 years agoHelper I
Syndicate_Admin Thank you so much! This has worked a treat! I'm going to create one query for each calendar year then append through PowerBI to avoid this issue.