Elasticsearch Scroll via Rest API

[AmyMCollins]

Hi,

I am trying to retrieve data from Elasticsearch using the REST API. I am able to pull data using the search API but I am struggling at retrieving data using the scroll api.

I have been following other forum posts (for instance https://community.powerbi.com/t5/Power-Query/Getting-Data-from-Elastic-Stack-with-REST-with-Scroll/m...and have used a recursive function to call the scroll api. However, I am getting authenication errors. 'DataSource.Error: Web.Contents with the Content option is only supported when connecting anonymously.'

I have also tried to use Basic Authentication and API Keys (source: https://www.dalesandro.net/accessing-rest-apis-with-basic-auth-and-api-key-in-power-query/ ). However, then I get the error 'Access to the resource is forbidden'

I don't understand how the search API works but the scroll API does not. Am I missing something?

Thanks in advance, 

Amy

[RyanResal]

Hello, I've worked on a solution for this issue and it used to work with me but it must be optimized, search scroll is not ideal to retrieve large data sets from elastic.