topic Re: Eventstream from On-Premise (Azure VM) SQL Server CDC? in Eventstream

Eventstream from On-Premise (Azure VM) SQL Server CDC?

rockfish — Sat, 22 Jun 2024 15:45:03 GMT

We have SQL Server Enterprise running on an Azure VM. While we are able to create a gateway, there does not appear to be a connector to bring it through an Evenstream. Our objective is to keep our (logical) bronze layer in sync with the source database. CDC seemed like the best approach.

- Is it possible to ingest CDC data from on-prem SQL Server using Eventstream?

- If not, can anyone recommend the next best alternative?

Thanks!

Jeff

Re: Eventstream from On-Premise (Azure VM) SQL Server CDC?

Anonymous — Mon, 24 Jun 2024 01:03:55 GMT

Hi rockfish ,

Currently, Eventstream primarily supports data ingestion from sources like Azure Event Hubs, Azure IoT Hub, sample data, and custom apps. Direct ingestion of CDC data from an on-premise SQL Server into Eventstream is not supported out-of-the-box.
For more details on supported sources by Eventstream, you can refer to the documentation here:
https://learn.microsoft.com/en-us/fabric/real-time-intelligence/event-streams/add-manage-eventstream-sources?pivots=enhanced-capabilities#supported-sources

If you must use EventStream, please use your data in a data source supported by EventStream.

Best Regards,
Dino Tao
If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

Re: Eventstream from On-Premise (Azure VM) SQL Server CDC?

xujx — Mon, 24 Jun 2024 10:41:55 GMT

rockfish Ingesting cdc data from on-prem SQL Server with Eventstream is in our roadmap of Eventstream. We are actively researching on this scenario to work out the best experience for this scenario.

When you say "Our objective is to keep our (logical) bronze layer in sync with the source database", how do you bring your initial snapshot to the target bronze layer from the source database? using certain batch copy job or you expect the cdc to bring the initial snapshot to bronze layer as well?

Re: Eventstream from On-Premise (Azure VM) SQL Server CDC?

Anonymous — Tue, 11 Feb 2025 06:53:08 GMT

Microsoft solution: Add SQL Server Change Data Capture as a source to eventstream - Microsoft Fabric | Microsoft Learn

Non-Microsoft but open source options, both require CDC turned on on the SQL Server:

- Kafke with Kafke Connect with a Debezium connector to retrieve the changing data from the SQL Server as a Source, then the Apache Kafka endpoint from Event Streaming as Sink.

- Apache Pulsar with the Debezium connector to retrieve the changing data from the SQL Server as a Source, then the Apache Kafka endpoint from Event Streaming as Sink.

Add Apache Kafka source to a Fabric eventstream - Microsoft Fabric | Microsoft Learn