Forum Discussion

rockfish's avatar
rockfish
Regular Visitor
2 years ago

Eventstream from On-Premise (Azure VM) SQL Server CDC?

We have SQL Server Enterprise running on an Azure VM.  While we are able to create a gateway, there does not appear to be a connector to bring it through an Evenstream.  Our objective is to keep our (logical) bronze layer in sync with the source database.   CDC seemed like the best approach.


- Is it possible to ingest CDC data from on-prem SQL Server using Eventstream?

- If not, can anyone recommend the next best alternative?

 

Thanks!

Jeff

 

 

3 Replies

  • Anonymous's avatar
    Anonymous
    Not applicable

    Hi rockfish ,

    Currently, Eventstream primarily supports data ingestion from sources like Azure Event Hubs, Azure IoT Hub, sample data, and custom apps. Direct ingestion of CDC data from an on-premise SQL Server into Eventstream is not supported out-of-the-box.
    For more details on supported sources by Eventstream, you can refer to the documentation here:
    https://learn.microsoft.com/en-us/fabric/real-time-intelligence/event-streams/add-manage-eventstream-sources?pivots=enhanced-capabilities#supported-sources 

    If you must use EventStream, please use your data in a data source supported by EventStream.

    Best Regards,
    Dino Tao
    If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

  • xujx's avatar
    xujx
    Microsoft Employee

    rockfish Ingesting cdc data from on-prem SQL Server with Eventstream is in our roadmap of Eventstream. We are actively researching on this scenario to work out the best experience for this scenario. 

     

    When you say "Our objective is to keep our (logical) bronze layer in sync with the source database", how do you bring your initial snapshot to the target bronze layer from the source database? using certain batch copy job or you expect the cdc to bring the initial snapshot to bronze layer as well?

  • Anonymous's avatar
    Anonymous
    Not applicable

    Microsoft solution: Add SQL Server Change Data Capture as a source to eventstream - Microsoft Fabric | Microsoft Learn

     

    Non-Microsoft but open source options, both require CDC turned on on the SQL Server:

     

    - Kafke with Kafke Connect with a Debezium connector to retrieve the changing data from the SQL Server as a Source, then the Apache Kafka endpoint from Event Streaming as Sink.

    - Apache Pulsar with the Debezium connector to retrieve the changing data from the SQL Server as a Source, then the Apache Kafka endpoint from Event Streaming as Sink.

     

    Add Apache Kafka source to a Fabric eventstream - Microsoft Fabric | Microsoft Learn