ElasticSearch connection not appearing in Copy Data activity

[EHD]

Hi all,

I'm still a bit new to Microsoft Fabric, so I'm hoping for some guidance.

I'm trying to set up a connection to Elasticsearch so that I can use it as a source in a Copy Data activity within my Pipeline.

I've successfully created an on-premises connection using our gateway and selected "Elasticsearch" as the connection type. When I check its status in Manage Connection and Gateways, it shows as "Online", so the connection appears to be working.

However, when I try to add a Copy Data activity in my pipeline, the Elasticsearch connection does not appear as an available data source.

Has anyone else encountered this? Am I missing a step, or is Elasticsearch not supported in Fabric Pipelines yet? Any help would be greatly appreciated!

Thanks in advance!

[v-mdharahman]

Hi @EHD,

Thanks for reaching out to the Microsoft fabric community forum.

It looks like you are trying to use "Elasticsearch" as a connection for your copy data activity in the pipelines. As @nilendraFabric gave a response to your issue, please go through the response and also the links attached to get a better understanding of why "Elasticsearch" is not showing as a connection in your copy data activity. Also if it solves your issue then please mark the helpful response as solution.

I would also take a moment to thank @nilendraFabric, for actively participating in the community forum and for the solutions you’ve been sharing in the community forum. Your contributions make a real difference.

If I misunderstand your needs or you still have problems on it, please feel free to let us know.  

Best Regards,
Hammad.
Community Support Team

If this post helps then please mark it as a solution, so that other members find it more quickly.

Thank you.

[nilendraFabric · Accepted solution]

Hello @EHD 

Microsoft Fabric currently does not provide a native connector for Elasticsearch in its Copy Data activity within Data Factory pipelines

https://learn.microsoft.com/en-us/fabric/data-factory/pipeline-support

https://community.fabric.microsoft.com/t5/Data-Engineering/Getting-data-from-elasticsearch/m-p/32637...

Microsoft Fabric's Data Factory does not include Elasticsearch in its list of supported connectors for data pipelines

The REST connector can be used to interact with Elasticsearch by leveraging its API.

If this is helpful ,please accept this solution and guve kudos

[EHD]

Thanks @nilendraFabric, very helpful!

I've now succesfully managed to setup the connection via the REST connector but as my index contains more than 10000 documents, I'd like to implement pagination using the scroll API.

Any idea how to do this using the pagination rules?