Forum Discussion
Elasticsearch Scroll via Rest API
Hi,
I am trying to retrieve data from Elasticsearch using the REST API. I am able to pull data using the search API but I am struggling at retrieving data using the scroll api.
I have been following other forum posts (for instance https://community.powerbi.com/t5/Power-Query/Getting-Data-from-Elastic-Stack-with-REST-with-Scroll/m-p/230222) and have used a recursive function to call the scroll api. However, I am getting authenication errors. 'DataSource.Error: Web.Contents with the Content option is only supported when connecting anonymously.'
I have also tried to use Basic Authentication and API Keys (source: https://www.dalesandro.net/accessing-rest-apis-with-basic-auth-and-api-key-in-power-query/ ). However, then I get the error 'Access to the resource is forbidden'
I don't understand how the search API works but the scroll API does not. Am I missing something?
Thanks in advance,
Amy
1 Reply
- RyanResalRegular Visitor
Hello, I've worked on a solution for this issue and it used to work with me but it must be optimized, search scroll is not ideal to retrieve large data sets from elastic.