Forum Discussion

AmyMCollins's avatar
AmyMCollins
Regular Visitor
3 years ago

Elasticsearch Scroll via Rest API

Hi,

 

I am trying to retrieve data from Elasticsearch using the REST API. I am able to pull data using the search API but I am struggling at retrieving data using the scroll api.

 

I have been following other forum posts (for instance https://community.powerbi.com/t5/Power-Query/Getting-Data-from-Elastic-Stack-with-REST-with-Scroll/m-p/230222) and have used a recursive function to call the scroll api. However, I am getting authenication errors. 'DataSource.Error: Web.Contents with the Content option is only supported when connecting anonymously.'

 

I have also tried to use Basic Authentication and API Keys (source: https://www.dalesandro.net/accessing-rest-apis-with-basic-auth-and-api-key-in-power-query/ ). However, then I get the error 'Access to the resource is forbidden'

 

I don't understand how the search API works but the scroll API does not. Am I missing something?

 

Thanks in advance, 

Amy

 

1 Reply

  • RyanResal's avatar
    RyanResal
    Regular Visitor

    Hello, I've worked on a solution for this issue and it used to work with me but it must be optimized, search scroll is not ideal to retrieve large data sets from elastic.