Forum Discussion
Deep dive on open mirroring
Hi pmscorca ,
Thank you for the clarification.
From a practical perspective, Open Mirroring in Microsoft Fabric works more like a managed CDC/replication service than a traditional ETL pipeline. The source system writes change data into a landing zone, and the Fabric mirroring engine continuously synchronizes and maintains Delta tables inside One Lake.
Compared to pipelines, Open Mirroring automatically handles incremental synchronization, change tracking, schema updates, and continuous replication, reducing the need for scheduled ETL jobs and manual orchestration.
Another key advantage is that the mirrored data becomes immediately available for Power BI, SQL analytics, Spark/Data Engineering, and AI/Data Science workloads because the data is maintained in Delta Parquet format.
In general, Open Mirroring is better suited for near real-time replication scenarios, while pipelines are better for complex ETL transformations and orchestration.
I hope this helps. Please feel free to reach out if you have any further questions.
Hi pmscorca , Thanks for the follow-up.
In Open Mirroring, data writing to the landing zone is handled externally using a file-based approach.
- The source system (or connector/custom app) writes change data files directly into the One Lake landing zone using the required folder structure.
- These files are typically in Parquet/CSV format and include insert, update, and delete markers, along with a _metadata.json file to define key columns.
Once the data is written:
- The Fabric mirroring engine continuously monitors the landing zone
- It reads incoming files, interprets the change (CDC) information, and
- Applies those changes to Delta tables automatically in One Lake
For your reference:
Open Mirroring Landing Zone Requirements and Formats - Microsoft Fabric | Microsoft Learn
I hope this clarifies .Please feel free to reach out to us if any queries.
Thank you.
The point is that the Open Mirroring however requires a pipeline or notebook to write data on the landing zone. It isn't a complete solution as a classic mirroring (f.e. SQL Server mirroring).
It seems that matching Open mirroring + landing zone pipeline vs incremental ingestion pipeline the related differencies are minimal.
Hi pmscorca,
Thank you for your observation. you are correct that Open Mirroring requires a process to write change data into the landing zone.However, as per Microsoft documentation, the key difference is in the responsibility model:
With Open Mirroring:
- External tools/processes deliver change data into the landing zone
- The Fabric mirroring engine manages replication and handles the complexity of applying changes and maintaining Delta tables in OneLake
Once data is available, Fabric continuously processes it and keeps the mirrored data up to date.
With pipeline-based incremental ingestion:
- You build and manage the ingestion workflow, including scheduling, incremental logic, and transformations
So while both approaches may use pipelines for data movement, the difference is that pipelines are customer managed ingestion and processing and where Open Mirroring is Fabric managed replication after data is landed.
Mirroring is intended for continuous replication scenarios with minimal setup, while pipelines are used for custom ETL,ELT orchestration and transformations.
Hope this helps, Thanks.