Forum Discussion

Anonymous's avatar
Anonymous
Not applicable
2 years ago
Solved

Mirroring in Microsoft Fabric

Hi,

 

I am some ambiguity in understanding Fabric Mirroring. I know that it enables users to effortlessly replicate external data warehouses or databases. But I want to understand does it copy the physical data to Fabric and if no then how does it work?

 

Hoping for a reply.

General Question
Need Help
pipelines
  • v-cboorla-msft's avatar
    v-cboorla-msft
    2 years ago

    Hi Anonymous 

     

    I have checked with the internal team. The internal team has confirmed that:

    Mirroring does physically copy the data from source database to OneLake delta parquet format as well as maintain any data changes in the OneCopy in Fabric OneLake.

     

    I hope this information helps. Please do let us know if you have any further questions.

     

    Thanks.

10 Replies

  • v-cboorla-msft's avatar
    v-cboorla-msft
    Microsoft Employee
    2 years ago

    Hi Anonymous 

     

    Thanks for using Microsoft Fabric Community.

    No, Microsoft Fabric Mirroring does not physically copy the data to Microsoft Fabric. Instead, it employs a sophisticated process to replicate data without moving it physically. Here’s how it works:

     

    Change Data Capture (CDC): Fabric's Mirroring feature uses a technology called Change Data Capture (CDC) to act like a detective. It constantly monitors your external data sources (like databases) and keeps a watchful eye for any changes, such as new entries, updates, or deletions. 

    Delta Tables: Instead of copying everything, Mirroring only captures these identified changes. It then cleverly transforms them into a format specifically designed for analysis, often using Delta tables. These tables are like condensed summaries, focusing only on the differences.

    OneLake Storage Layer: The transformed changes are then placed in a secure storage area within Fabric called OneLake. OneLake is designed for efficiency, ensuring the data is organized and readily accessible for your analysis needs.

    Intelligent Logic: To optimize performance, Fabric employs intelligent logic. It can figure out when your source data has changed and only triggers synchronization for those specific updates. This minimizes unnecessary processing power usage.

    Granular Controls: Mirroring offers you granular control. You can choose which specific data (like tables or entities) you want to be reflected in Fabric. This allows you to focus on the most relevant information for your analysis.

    For more details please refer : Introducing Mirroring in Microsoft Fabric.

                                                   : What's new in Microsoft Fabric?

     

    I hope this information helps. Please do let us know if you have any further questions.

     

    Thanks.

     

     

    • Anonymous's avatar
      Anonymous
      Not applicable
      2 years ago

      Hi v-cboorla-msft ,

       

      I get the CDC part of mirroring but how is the inital snapshot of data handeled by Fabric. I just want to develop an understanding of it.

       

      Hoping for a reply.

      • v-cboorla-msft's avatar
        v-cboorla-msft
        Microsoft Employee
        2 years ago

        Hi Anonymous 

         

        You're right, the initial snapshot in Fabric Mirroring is a crucial step to understand. While CDC handles ongoing changes, the initial snapshot brings your external data into Fabric for the first time. Here's how it's likely handled:

        Prioritizing Efficiency: Fabric Mirroring is likely designed for speed and minimal resource usage during the initial data transfer. This could involve:

        • Bulk Loading: Transferring large chunks of data at once instead of individual records, significantly accelerating the process.
        • Partitioned Transfers: Particularly for very large datasets, dividing the data into smaller subsets allows for faster and more efficient transfer.

        Potential Configurability:

        • Partial Snapshots: Depending on your configuration, you might have the option to specify which tables or subsets of data to include in the initial snapshot. This allows you to focus on specific data relevant to your needs.

        Leveraging Source Capabilities:

        • It's possible that Fabric Mirroring utilizes existing export functionalities within your source database (like Azure Cosmos DB or SQL Server) to expedite the initial data transfer process. This can leverage existing tools optimized for your specific source.

        Additional Considerations:

        • Temporary Storage (Possible): The initial data transfer might be staged in temporary storage optimized for speed before being transformed and stored in OneLake, Fabric's data storage solution. This temporary storage would facilitate a faster initial transfer.
        • Gradual Population (Speculative): For exceptionally large datasets, the initial data population might occur gradually to minimize any impact on the source database's performance. This would ensure a smooth initial synchronization process.

         

        I hope this information helps. Please do let us know if you have any further questions.

         

        Thanks.

  • purushotham15's avatar
    purushotham15
    Regular Visitor
    1 year ago

    Looks like its similar to HTAP capability in synapse where the data can be ingested/copied automatically in dedicated pool whenever there is a change in source system(cosmos/azure sql). It works for all entities which has PK columns. Is it same here? any difference in handling?