Extracting Data from a Pulse Connect API

[mcfw]

Hello everyone,

I'm trying to create a realtime APM dashboard. The data resides on a pulse connect server. I already have the URL, API-Key and API-KEY_Value to extracct the data. The issue here is the data is too many, so I have to paginate to only bring out successful transactions. Even at that, it only brings me data of the last day, instead of data from the past 60 days. 
Can someone please help out on the above code.

Thank you

[v-tejrama · Accepted solution]

Hi @mcfw ,

It appears you are working with a large dataset from the Pulse Connect API in Power Query and encountering challenges related to pagination and date range limitations. Based on your description, the issue where only the most recent day's data is returned typically indicates that the API relies on cursor-based pagination rather than traditional offset-based methods. If the cursor or token from each API response is not correctly used in subsequent requests, you will continue to receive only the latest subset of data, regardless of your date filter.

Your date range logic seems appropriate, but it will only function as intended once pagination is handled correctly. In Power Query, this usually requires implementing a loop that continues to call the API using the next cursor value from each response until all data is retrieved. Without this, the query will remain limited to the initial results.

Therefore, I recommend checking whether the API response includes a next cursor or similar indicator and updating your query to utilize it for pagination. This should enable you to access the complete dataset as expected.

Thank you.

[v-tejrama]

Hi @mcfw ,

I wanted to follow up and see if you had a chance to review the information shared. If you have any further questions or need additional assistance, feel free to reach out.

Thank you.

[v-tejrama]

Hi @mcfw ,

I wanted to check if you had the opportunity to review the information provided. Please feel free to contact us if you have any further questions.

Thank you.

[parry2k]

@mcfw there are tons of videos/blog on how to use cursor based API pagination in PQ, wondering if you did google on this topic? I'm assuming this is cursor based pagination.