Forum Discussion
Helper IVPublic Preview Consideratation - Data Mirroring
Hi XhevahirMehalla ,
Thanks for your detailed follow-up. Let me clarify and summarize the solution so you have a clear picture for your diagram and next steps.
No separate Kafka setup is required Azure Event Hubs acts as a Kafka-compatible broker, so Debezium can publish directly to Event Hubs without maintaining your own Kafka cluster. You will need a small Azure VM or container to run the Debezium Oracle connector. This VM must have secure network connectivity to your OCI Oracle database, either via VPN, ExpressRoute, or a secure public endpoint, and the connector will use the CDC/log-mining user to access redo logs. On the Oracle side, ensure supplemental logging is enabled, the database is in ARCHIVELOG mode, and a dedicated user with proper privileges exists for Debezium.
Once Debezium is running, it reads changes from Oracle redo logs and streams them as JSON events (inserts, updates, deletes) to Event Hubs.
From Event Hubs, you have flexibility depending on your target. If you want to land data in Fabric Lakehouse, a lightweight consumer process (e.g., Azure Function, Synapse notebook, or Databricks job) can read events and write them into Delta tables in OneLake.
Once in the Lakehouse, your Fabric workspace and Power BI reports can directly query the data. If your goal is Azure SQL Database without Fabric, a small Stream Analytics job or consumer app can read the Event Hubs events and apply the changes to your tables, enabling Power BI or other reporting layers to access them as usual.
Regarding monitoring and correctness: the Azure VM running Debezium should be continuously available (Azure Monitor can help), and Debezium preserves event ordering per table and primary key. Downstream processes must handle idempotency to avoid applying the same change twice and ensure correct order when processing events from multiple tables or partitions. Planning for retries, restarts, and error handling is essential, but these are standard practices in any CDC pipeline.
Overall, this approach is cost-effective, avoids expensive GoldenGate licensing, and is robust enough for production workloads like your digital banking statements scenario. Starting with a proof-of-concept on one table and one Event Hub throughput unit is recommended to validate connectivity, latency, and processing before scaling to all three tables. This setup allows a hybrid approach you can keep your existing Synapse/Azure SQL solution for production stability while experimenting with Fabric Lakehouse for future migration, and cross-querying between these sources is fully supported if needed.
Thank you,Tejaswi.
Hi XhevahirMehalla ,
Thanks for reaching out to the Microsoft fabric community forum.
Thank you for providing the detailed scenario. As you consider Fabric, I’d like to offer some clarification. Data Mirroring in Fabric, including support for Oracle as a source, is currently in Public Preview. This status means it is fully supported for evaluation, pilots, and proof-of-concept projects, but it does not yet include SLA guarantees and may undergo changes before general availability. If your client is comfortable with preview features, you can explore Oracle mirroring into a Fabric Warehouse or Lakehouse, which enables near real-time replication without ETL and can streamline your current Synapse configuration. However, for business-critical workloads that require production-grade stability, it is advisable to maintain a traditional approach, such as using Azure Data Factory or Event Hubs/Kafka for CDC ingestion from Oracle OCI into Azure SQL Database, followed by DirectQuery with Power BI for real-time reporting.
Within Fabric, Synapse pipelines would be replaced by Dataflows Gen2 or Notebooks for transformations, utilizing OneLake/Lakehouse for central storage. Since you are already using Azure DevOps, you can leverage Fabric Git integration for version control and deployment management. Power BI’s native integration in Fabric will also keep your reporting processes straightforward.
To summarize, Fabric Mirroring is a forward-looking solution worth exploring for near real-time scenarios. For immediate production needs, maintaining the Oracle to Azure SQL setup via CDC/Event Hub is recommended, allowing you to experiment with Fabric while ensuring ongoing stability for your client.
Best Regards,
Tejaswi.
Community Support
Helper IVThanks for all this insights. Now I undestand the implications of using Fabcic Data Mirroring.
It looks that we need to continue using Azure Synapse, Data lake Gen 2 and Azure SQL db and Power BI.
I am not clear how to expand to get to existing solution without being on Fabric to fulfill :
- Near -real time data from Oracle OCI database
- or The perfect solution would be real-time solution (getting data real time from Oracle OCI db).
I was consider using Event Hubs and see if this will do the job although I don't know how complext is this to bring real data into an azure sql db.
Please can you give me some guidance on this or point me to the right team?
Thanks
Xhev
Hi XhevahirMehalla ,
Thank you for providing additional information. Exploring Event Hubs for near real-time data movement from Oracle OCI to Azure SQL Database is a solid approach. However, it is important to note that Event Hubs does not directly capture database changes. You will need to implement a CDC (Change Data Capture) mechanism or a log-based streaming tool on the Oracle side to send data to Event Hubs. Once the data is in Event Hubs, you can process it with Azure Stream Analytics or a similar service before writing to Azure SQL Database.
This solution offers low-latency or near real-time updates but requires extra configuration and ongoing monitoring. If you prefer a more managed CDC solution, consider options such as Oracle GoldenGate, Debezium, or Azure Data Factory’s CDC feature to capture changes and transfer them to Azure SQL.
For most reporting scenarios, near real-time refresh using CDC or frequent micro-batch loads is typically sufficient and easier to maintain than a fully event-driven pipeline. The Microsoft Learn documentation on Oracle CDC and Event Hubs provides comprehensive guidance for implementation.
Thank you,Tejaswi.
- XhevahirMehalla
That bits of CDC I am not clear about:
The request is simple:
- Our client is a digital bank
- They want to create a online statements which will be build on Power BI/Builder maybe and send to the digital app for consumption.
- We have three main tables which are update regularyl in Oracle OCI and want access to these table through a Data warehouse solution. Not direct access to Oracle DB.
- Using Oracle Golden Gate , Kafka and Event Hubs is one wany but Golden Gate is very expensive.
Please can you see if any of your architects would have any other ideas.
We look at:
- Costs
- Less Complex
- Manage them easily.
At the moment how yeto suggest a solution as I don't understand the whole thing.
I might consider:
- leave the existing solution as is in Synapse - don't migrate to Fabric yet.
- Have just Fabric license just to do the Data Mirroring (Oracle) into a LakeHouse and produce the statement using these data.
A question is:
I will have data on Azure SQL DB and some data on Fabric LakeHouse.
I guess I can do cross querying between these sources if I need to and at some point in the future we can migrate to Fabric from Synapse.
Let me know what's your thinking and If I can use Data Mirroring in oracle I need a sets of instrucrtions what Do I need to setup on Oracle OCI db.
We have the full control of the db although for the licensing we need to ask the client.
Please help!
Thank you
Hi XhevahirMehalla ,
Thank you for providing such a clear overview of your requirements. To recap, your main goal is to integrate Oracle (OCI) data into Fabric/Power BI for online statement generation, which necessitates a CDC or replication solution. While GoldenGate is one option, it may be cost-prohibitive. Below are several practical alternatives:
- Fabric Data Mirroring (Preview): Microsoft’s mirroring feature enables direct replication of Oracle tables to a Fabric Lakehouse, simplifying the process. Please refer to the relevant tutorials for setup instructions.
- Open-source approach (Debezium + Event Hubs): Debezium’s Oracle connector streams redo log changes to Kafka, and Azure Event Hubs supports Kafka protocols, allowing data transfer to ADLS Gen2/OneLake for use in Fabric Lakehouse. This method is cost-effective but requires DBA involvement for log configuration.
- Commercial CDC tools: Solutions like Qlik Replicate, BryteFlow, and Striim offer robust interfaces and support for replicating Oracle data to ADLS Gen2, Azure SQL DB, or Delta tables, but involve licensing costs.
- Azure Data Factory (batch mode): If real-time updates are not essential, ADF pipelines can periodically copy Oracle data, offering a straightforward and economical option.
Oracle prerequisites include:
- Enabling supplemental logging.
- Access to redo/archived logs.
- Provisioning a user with SELECT and log mining privileges.
- Establishing secure OCI-to-Azure connectivity.
Power BI and Fabric allow cross-querying between Azure SQL DB and Fabric Lakehouse, but consolidating reporting data in Fabric Lakehouse is recommended for efficiency.
To select the most suitable approach:
- Use Debezium or a commercial CDC tool for real-time needs.
- Debezium with Event Hubs is effective for slight delays.
- ADF is ideal for hourly or daily refreshes.
- Consider Oracle Data Mirroring in Fabric (Preview) for a managed, Fabric-native solution.
Since you have a Fabric license, starting with the mirroring option is recommended. The referenced Microsoft Learn tutorials can guide your DBA through the process.
Thank you,Tejaswi.