Forum Discussion

EHD's avatar
EHD
Regular Visitor
1 year ago
Solved

ElasticSearch connection not appearing in Copy Data activity

Hi all,

 

I'm still a bit new to Microsoft Fabric, so I'm hoping for some guidance.

I'm trying to set up a connection to Elasticsearch so that I can use it as a source in a Copy Data activity within my Pipeline.

 

I've successfully created an on-premises connection using our gateway and selected "Elasticsearch" as the connection type. When I check its status in Manage Connection and Gateways, it shows as "Online", so the connection appears to be working.

 

However, when I try to add a Copy Data activity in my pipeline, the Elasticsearch connection does not appear as an available data source.

 

Has anyone else encountered this? Am I missing a step, or is Elasticsearch not supported in Fabric Pipelines yet? Any help would be greatly appreciated!

 

Thanks in advance!

data engineering

3 Replies

  • nilendraFabric's avatar
    nilendraFabric
    Super User
    1 year ago

    Hello EHD 

    Microsoft Fabric currently does not provide a native connector for Elasticsearch in its Copy Data activity within Data Factory pipelines

    https://learn.microsoft.com/en-us/fabric/data-factory/pipeline-support

     

     

    https://community.fabric.microsoft.com/t5/Data-Engineering/Getting-data-from-elasticsearch/m-p/3263796



    Microsoft Fabric's Data Factory does not include Elasticsearch in its list of supported connectors for data pipelines

     

     

    The REST connector can be used to interact with Elasticsearch by leveraging its API.

     

    If this is helpful ,please accept this solution and guve kudos

     

     

    • EHD's avatar
      EHD
      Regular Visitor
      1 year ago

      Thanks nilendraFabric, very helpful!

       

      I've now succesfully managed to setup the connection via the REST connector but as my index contains more than 10000 documents, I'd like to implement pagination using the scroll API.

       

      Any idea how to do this using the pagination rules?

  • Anonymous's avatar
    Anonymous
    Not applicable
    1 year ago

    Hi EHD,

    Thanks for reaching out to the Microsoft fabric community forum.

     

    It looks like you are trying to use "Elasticsearch" as a connection for your copy data activity in the pipelines. As nilendraFabric gave a response to your issue, please go through the response and also the links attached to get a better understanding of why "Elasticsearch" is not showing as a connection in your copy data activity. Also if it solves your issue then please mark the helpful response as solution.

     

    I would also take a moment to thank nilendraFabric, for actively participating in the community forum and for the solutions you’ve been sharing in the community forum. Your contributions make a real difference.

     

    If I misunderstand your needs or you still have problems on it, please feel free to let us know.  

    Best Regards,
    Hammad.
    Community Support Team

     

    If this post helps then please mark it as a solution, so that other members find it more quickly.

    Thank you.