Forum Discussion
Public Preview Consideratation - Data Mirroring
I have a solution build on Synapse for a client which includes:
- Data lake gen 2 account as storage layer where we store CSV files extracted from Oracle OCI db
- Synapse analytics Pipelines which copy the data and transform then data within Data lake (directories raw/staging).
- Synapse pipelines where we transform data from staging container on DaTa lake and store them into Azure SQL database.
- Use Power BI Desktop and Service using Pro/PPU licenses.
- An Angular application which loads some cSV files into Data lake gen 2 and Synapse Transform them.
I want to deploy the same solution into Fabric now on another client.
We have full control of the code and client subscription.
The new client want to have some reports real time or happy with near-real time.
For that I am considerting data Mirroring which in Public Preview and I want to consider implications of choosing that as the solution.
All code is on Azure DevOps.
Please can some one point me how can I proceed.
Happy to give more details on this.
Thanks
Xhev
Hi XhevahirMehalla ,
Thanks for your detailed follow-up. Let me clarify and summarize the solution so you have a clear picture for your diagram and next steps.
No separate Kafka setup is required Azure Event Hubs acts as a Kafka-compatible broker, so Debezium can publish directly to Event Hubs without maintaining your own Kafka cluster. You will need a small Azure VM or container to run the Debezium Oracle connector. This VM must have secure network connectivity to your OCI Oracle database, either via VPN, ExpressRoute, or a secure public endpoint, and the connector will use the CDC/log-mining user to access redo logs. On the Oracle side, ensure supplemental logging is enabled, the database is in ARCHIVELOG mode, and a dedicated user with proper privileges exists for Debezium.
Once Debezium is running, it reads changes from Oracle redo logs and streams them as JSON events (inserts, updates, deletes) to Event Hubs.
From Event Hubs, you have flexibility depending on your target. If you want to land data in Fabric Lakehouse, a lightweight consumer process (e.g., Azure Function, Synapse notebook, or Databricks job) can read events and write them into Delta tables in OneLake.
Once in the Lakehouse, your Fabric workspace and Power BI reports can directly query the data. If your goal is Azure SQL Database without Fabric, a small Stream Analytics job or consumer app can read the Event Hubs events and apply the changes to your tables, enabling Power BI or other reporting layers to access them as usual.
Regarding monitoring and correctness: the Azure VM running Debezium should be continuously available (Azure Monitor can help), and Debezium preserves event ordering per table and primary key. Downstream processes must handle idempotency to avoid applying the same change twice and ensure correct order when processing events from multiple tables or partitions. Planning for retries, restarts, and error handling is essential, but these are standard practices in any CDC pipeline.
Overall, this approach is cost-effective, avoids expensive GoldenGate licensing, and is robust enough for production workloads like your digital banking statements scenario. Starting with a proof-of-concept on one table and one Event Hub throughput unit is recommended to validate connectivity, latency, and processing before scaling to all three tables. This setup allows a hybrid approach you can keep your existing Synapse/Azure SQL solution for production stability while experimenting with Fabric Lakehouse for future migration, and cross-querying between these sources is fully supported if needed.
Thank you,Tejaswi.
13 Replies
- You may use OneLake, Dataflow gen2, Notebook, SQL Datawarehouse(for mirroring) from Fabric to proceed with first step.
- If you are planning to process the real time data, you can choose data mirroring, with zero ETL, from the Azure SQL DB, Azure Cosmos DB and so on. Pls go through this artcicles Announcing the Public Preview of Mirroring in Microsoft Fabric | Microsoft Fabric Blog | Microsoft Fabric, Announcing Mirroring Azure SQL Database in Fabric for Public Preview | Microsoft Fabric Blog | Microsoft Fabric
- Replace your Azure Synapse pipleines with Notebooks/Datflows for transofmration, use lakehosue or warehouse for structured storage.
- Power BI you can use as it's natively available in Fabric.
- Since you are using Azure DevOps, here yiu maynuse Fabric Git inetgration to manage your workspace items.
Overview of Fabric Git integration - Microsoft Fabric | Microsoft Learn
Hope it may helps you.
Thank you!
Did I answer your question? Mark my post as a solution!
Proud to be a Super User!Thanks very useful however I am more interested on Oracle database on OCI (cloud) as the sourcer of data.
At Vienna conference it announced Data Mirroring for Oracle too.
How safe is this to use a product in Prublic preview?
Another thing is:
If I can't migrate to Fabric - how can do something similar - produce real time reports using Azure SQL database as target DB and source again Oracle DB on OCI as the source.
Can Event Hub help and how this will be used with Kafka? how much work I need to do to setup this?
Do I need to speak toi someone else withing MSFT ?
Thanks
Hi XhevahirMehalla ,
Thanks for reaching out to the Microsoft fabric community forum.Thank you for providing the detailed scenario. As you consider Fabric, I’d like to offer some clarification. Data Mirroring in Fabric, including support for Oracle as a source, is currently in Public Preview. This status means it is fully supported for evaluation, pilots, and proof-of-concept projects, but it does not yet include SLA guarantees and may undergo changes before general availability. If your client is comfortable with preview features, you can explore Oracle mirroring into a Fabric Warehouse or Lakehouse, which enables near real-time replication without ETL and can streamline your current Synapse configuration. However, for business-critical workloads that require production-grade stability, it is advisable to maintain a traditional approach, such as using Azure Data Factory or Event Hubs/Kafka for CDC ingestion from Oracle OCI into Azure SQL Database, followed by DirectQuery with Power BI for real-time reporting.
Within Fabric, Synapse pipelines would be replaced by Dataflows Gen2 or Notebooks for transformations, utilizing OneLake/Lakehouse for central storage. Since you are already using Azure DevOps, you can leverage Fabric Git integration for version control and deployment management. Power BI’s native integration in Fabric will also keep your reporting processes straightforward.
To summarize, Fabric Mirroring is a forward-looking solution worth exploring for near real-time scenarios. For immediate production needs, maintaining the Oracle to Azure SQL setup via CDC/Event Hub is recommended, allowing you to experiment with Fabric while ensuring ongoing stability for your client.
Best Regards,
Tejaswi.
Community Support
Thanks for all this insights. Now I undestand the implications of using Fabcic Data Mirroring.
It looks that we need to continue using Azure Synapse, Data lake Gen 2 and Azure SQL db and Power BI.
I am not clear how to expand to get to existing solution without being on Fabric to fulfill :
- Near -real time data from Oracle OCI database
- or The perfect solution would be real-time solution (getting data real time from Oracle OCI db).
I was consider using Event Hubs and see if this will do the job although I don't know how complext is this to bring real data into an azure sql db.
Please can you give me some guidance on this or point me to the right team?
Thanks
Xhev
Hi XhevahirMehalla ,
Thank you for providing additional information. Exploring Event Hubs for near real-time data movement from Oracle OCI to Azure SQL Database is a solid approach. However, it is important to note that Event Hubs does not directly capture database changes. You will need to implement a CDC (Change Data Capture) mechanism or a log-based streaming tool on the Oracle side to send data to Event Hubs. Once the data is in Event Hubs, you can process it with Azure Stream Analytics or a similar service before writing to Azure SQL Database.
This solution offers low-latency or near real-time updates but requires extra configuration and ongoing monitoring. If you prefer a more managed CDC solution, consider options such as Oracle GoldenGate, Debezium, or Azure Data Factory’s CDC feature to capture changes and transfer them to Azure SQL.
For most reporting scenarios, near real-time refresh using CDC or frequent micro-batch loads is typically sufficient and easier to maintain than a fully event-driven pipeline. The Microsoft Learn documentation on Oracle CDC and Event Hubs provides comprehensive guidance for implementation.
Thank you,Tejaswi.
