Forum Discussion
Mirroring in Microsoft Fabric
Hi Anonymous
I have checked with the internal team. The internal team has confirmed that:
Mirroring does physically copy the data from source database to OneLake delta parquet format as well as maintain any data changes in the OneCopy in Fabric OneLake.
I hope this information helps. Please do let us know if you have any further questions.
Thanks.
Hi v-cboorla-msft ,
I get the CDC part of mirroring but how is the inital snapshot of data handeled by Fabric. I just want to develop an understanding of it.
Hoping for a reply.
Microsoft EmployeeHi Anonymous
You're right, the initial snapshot in Fabric Mirroring is a crucial step to understand. While CDC handles ongoing changes, the initial snapshot brings your external data into Fabric for the first time. Here's how it's likely handled:
Prioritizing Efficiency: Fabric Mirroring is likely designed for speed and minimal resource usage during the initial data transfer. This could involve:
- Bulk Loading: Transferring large chunks of data at once instead of individual records, significantly accelerating the process.
- Partitioned Transfers: Particularly for very large datasets, dividing the data into smaller subsets allows for faster and more efficient transfer.
Potential Configurability:
- Partial Snapshots: Depending on your configuration, you might have the option to specify which tables or subsets of data to include in the initial snapshot. This allows you to focus on specific data relevant to your needs.
Leveraging Source Capabilities:
- It's possible that Fabric Mirroring utilizes existing export functionalities within your source database (like Azure Cosmos DB or SQL Server) to expedite the initial data transfer process. This can leverage existing tools optimized for your specific source.
Additional Considerations:
- Temporary Storage (Possible): The initial data transfer might be staged in temporary storage optimized for speed before being transformed and stored in OneLake, Fabric's data storage solution. This temporary storage would facilitate a faster initial transfer.
- Gradual Population (Speculative): For exceptionally large datasets, the initial data population might occur gradually to minimize any impact on the source database's performance. This would ensure a smooth initial synchronization process.
I hope this information helps. Please do let us know if you have any further questions.
Thanks.
- v-cboorla-msft
Hi Anonymous
We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others.
Otherwise, will respond back with the more details and we will try to help.
Thanks.Hi v-cboorla-msft ,
So during the inital snapshot Mirroring does copy the data from source to Fabric, correct me if I am wrong.
- v-cboorla-msft
Hi Anonymous
Yes, during the initial snapshot, mirroring establishes a synchronized copy of the source data within the Microsoft Fabric Lakehouse. This process ensures that the data in the lakehouse is synchronized with the source system. It’s like creating a mirror image of the data, capturing its current state. Once this initial snapshot is complete, the mirroring process efficiently captures and integrates incremental data updates, enabling efficient data synchronization and ongoing data management.
For details please refer : Fabric Mirroring - public preview
I hope this information helps. Please do let us know if you have any further questions.
Thanks.