Forum Discussion
Public Preview Consideratation - Data Mirroring
Hi XhevahirMehalla ,
Thanks for your detailed follow-up. Let me clarify and summarize the solution so you have a clear picture for your diagram and next steps.
No separate Kafka setup is required Azure Event Hubs acts as a Kafka-compatible broker, so Debezium can publish directly to Event Hubs without maintaining your own Kafka cluster. You will need a small Azure VM or container to run the Debezium Oracle connector. This VM must have secure network connectivity to your OCI Oracle database, either via VPN, ExpressRoute, or a secure public endpoint, and the connector will use the CDC/log-mining user to access redo logs. On the Oracle side, ensure supplemental logging is enabled, the database is in ARCHIVELOG mode, and a dedicated user with proper privileges exists for Debezium.
Once Debezium is running, it reads changes from Oracle redo logs and streams them as JSON events (inserts, updates, deletes) to Event Hubs.
From Event Hubs, you have flexibility depending on your target. If you want to land data in Fabric Lakehouse, a lightweight consumer process (e.g., Azure Function, Synapse notebook, or Databricks job) can read events and write them into Delta tables in OneLake.
Once in the Lakehouse, your Fabric workspace and Power BI reports can directly query the data. If your goal is Azure SQL Database without Fabric, a small Stream Analytics job or consumer app can read the Event Hubs events and apply the changes to your tables, enabling Power BI or other reporting layers to access them as usual.
Regarding monitoring and correctness: the Azure VM running Debezium should be continuously available (Azure Monitor can help), and Debezium preserves event ordering per table and primary key. Downstream processes must handle idempotency to avoid applying the same change twice and ensure correct order when processing events from multiple tables or partitions. Planning for retries, restarts, and error handling is essential, but these are standard practices in any CDC pipeline.
Overall, this approach is cost-effective, avoids expensive GoldenGate licensing, and is robust enough for production workloads like your digital banking statements scenario. Starting with a proof-of-concept on one table and one Event Hub throughput unit is recommended to validate connectivity, latency, and processing before scaling to all three tables. This setup allows a hybrid approach you can keep your existing Synapse/Azure SQL solution for production stability while experimenting with Fabric Lakehouse for future migration, and cross-querying between these sources is fully supported if needed.
Thank you,Tejaswi.
Hi Tejaswi,
This is much clearer now.
I understand this better but still have few questions as I am putting this into a diagram:
- No Kafka setup/support is needed? Confirm?
- Need a azure VM where we will install Debenzium? If so what connection is needed to access OCI oracle db? Pls send me some high level steps and what should I look for?
- Referrin to this " or pushing it to Azure SQL Database via a consumer or Stream Analytics job. For digital banking scenarios like online statements, landing data in a Lakehouse with Delta tables is often preferred for scalability and ease of handling various data changes."
- What would be the consumer (Fabric or Azure Sql DB) ?
- Event hubs publishing the changes to : If in Fabric what do I need to setup (In fabric I know what to setup, Workspace, Lakehouse and all that but what else I need to do for data to reach Fabric - Event hub can land data on Lakehouse?).
- If I use the azure sql db but not on Fabric which will be most likely scenario - you mention I need to deploy Azure Stream Analytics. I want only to get the changes in a table on Azure that's all and then I can access them how I wnated. Pls confirm my understanding.
- Referring to this: "However, it is important to plan for monitoring and managing the Debezium runtime and to design downstream processes to ensure idempotency and correct event ordering."
- What do I need to look for in this as looks a bit worring that I have to:
- Azure VM has to stay up all the time - that's fine. Azure Monitor can do that
- Correct Event Ordering - what do you mean? doesnt Debezium handle that? Please can you elaborate a bit more on this?
Will solution be resbust enough in your view and has anyone has this setup?
Thanks
Xhev
Hi XhevahirMehalla ,
Thanks for your detailed follow-up. Let me clarify and summarize the solution so you have a clear picture for your diagram and next steps.
No separate Kafka setup is required Azure Event Hubs acts as a Kafka-compatible broker, so Debezium can publish directly to Event Hubs without maintaining your own Kafka cluster. You will need a small Azure VM or container to run the Debezium Oracle connector. This VM must have secure network connectivity to your OCI Oracle database, either via VPN, ExpressRoute, or a secure public endpoint, and the connector will use the CDC/log-mining user to access redo logs. On the Oracle side, ensure supplemental logging is enabled, the database is in ARCHIVELOG mode, and a dedicated user with proper privileges exists for Debezium.
Once Debezium is running, it reads changes from Oracle redo logs and streams them as JSON events (inserts, updates, deletes) to Event Hubs.
From Event Hubs, you have flexibility depending on your target. If you want to land data in Fabric Lakehouse, a lightweight consumer process (e.g., Azure Function, Synapse notebook, or Databricks job) can read events and write them into Delta tables in OneLake.
Once in the Lakehouse, your Fabric workspace and Power BI reports can directly query the data. If your goal is Azure SQL Database without Fabric, a small Stream Analytics job or consumer app can read the Event Hubs events and apply the changes to your tables, enabling Power BI or other reporting layers to access them as usual.
Regarding monitoring and correctness: the Azure VM running Debezium should be continuously available (Azure Monitor can help), and Debezium preserves event ordering per table and primary key. Downstream processes must handle idempotency to avoid applying the same change twice and ensure correct order when processing events from multiple tables or partitions. Planning for retries, restarts, and error handling is essential, but these are standard practices in any CDC pipeline.
Overall, this approach is cost-effective, avoids expensive GoldenGate licensing, and is robust enough for production workloads like your digital banking statements scenario. Starting with a proof-of-concept on one table and one Event Hub throughput unit is recommended to validate connectivity, latency, and processing before scaling to all three tables. This setup allows a hybrid approach you can keep your existing Synapse/Azure SQL solution for production stability while experimenting with Fabric Lakehouse for future migration, and cross-querying between these sources is fully supported if needed.
Thank you,
Tejaswi.