Forum Discussion

EHD's avatar
EHD
Regular Visitor
1 year ago
Solved

ElasticSearch connection not appearing in Copy Data activity

Hi all,   I'm still a bit new to Microsoft Fabric, so I'm hoping for some guidance. I'm trying to set up a connection to Elasticsearch so that I can use it as a source in a Copy Data activity with...
data engineering
nilendraFabric's avatar
nilendraFabric
Icon for Super User rankSuper User
1 year ago

Hello EHD 

Microsoft Fabric currently does not provide a native connector for Elasticsearch in its Copy Data activity within Data Factory pipelines

https://learn.microsoft.com/en-us/fabric/data-factory/pipeline-support

 

 

https://community.fabric.microsoft.com/t5/Data-Engineering/Getting-data-from-elasticsearch/m-p/3263796



Microsoft Fabric's Data Factory does not include Elasticsearch in its list of supported connectors for data pipelines

 

 

The REST connector can be used to interact with Elasticsearch by leveraging its API.

 

If this is helpful ,please accept this solution and guve kudos

 

 

  • EHD's avatar
    EHD
    Regular Visitor
    1 year ago

    Thanks nilendraFabric, very helpful!

     

    I've now succesfully managed to setup the connection via the REST connector but as my index contains more than 10000 documents, I'd like to implement pagination using the scroll API.

     

    Any idea how to do this using the pagination rules?