Simplifying Microsoft Fabric Mirroring to Migration

[knaveen]

Microsoft Fabric offers a mirroring feature that continuously replicates data from external databases into OneLake. Mirroring is designed to bring disparate data sources into a single analytics platform with minimal latency and at low cost. Once mirrored, the data is stored as Delta tables, making it instantly queryable in OneLake and accessible across the Fabric ecosystem (Spark notebooks, data engineering, Power BI, etc.). Mirroring replicates both data and metadata into OneLake and converts them to Parquet format Fabric supports and also mirroring keeps data synchronised automatically so that it is always up‑to‑date in OneLake. 

Why move away from mirroring?

Mirroring provides near real‑time replication without the need for ETL pipelines and is fully managed by Microsoft. However, many organisations treat mirroring as an interim step toward fully adopting Fabric. Once your workload becomes comfortable in Fabric, you may want to decommission mirroring and manage the data directly within a Lakehouse or Warehouse. Reasons for moving off mirroring include:

• Reducing long‑term dependence on the source system. If you no longer need to read from the operational database, mirroring could be redundant and potentially costly (you may exceed the free replication quota). Storing data directly in a Lakehouse eliminates ongoing replication.

• Simplifying governance. A Lakehouse can be fully managed within Fabric’s security and governance tooling. When mirrored data remains connected to external systems, access policies may need to be maintained in multiple places.

• Increasing flexibility. Working with Parquet/Delta files directly in a Lakehouse allows you to restructure data, add new columns or transform formats that mirroring may not allow.

The standard approach to migrating off mirroring is to build data pipelines that copy data from each source table into a Lakehouse and then shut down the mirroring connection. These pipelines must be developed, tested and scheduled and can be time‑consuming, especially if there are hundreds of tables.

A simple workaround: copy mirrored files directly into your Lakehouse

While exploring Microsoft Fabric, I discovered a simpler way to migrate mirrored tables. Under the hood, mirroring stores each table’s data in OneLake as Parquet files (a Delta table). You can take advantage of that architecture by copying the Parquet files from the mirrored folder into your Lakehouse’s OneLake folder. Because both mirror storage and Lakehouse storage reside in OneLake, this approach requires no data transformation you simply copy the files as‑is.

How it works

Every mirrored table is stored under a path similar to the following (for a fictional Mirrored database ID 34ceda14-...):

abfss://<workspace_id>@onelake.dfs.fabric.microsoft.com/<mirrored_database_guid>/Tables/<schema>/<table_name>

 When you create a Lakehouse, Fabric assigns it its own OneLake container. You can copy files from the mirrored path into your Lakehouse path:

abfss://<workspace_id>@onelake.dfs.fabric.microsoft.com/<lakehouse_guid>/Tables/<schema>/<table_name>

Because both locations live in OneLake, copying files preserves the Delta format and schema. There are two ways to perform this copy:

1. Manual method via OneLake Explorer. OneLake Explorer is part of Microsoft Fabric. Navigate to the mirrored database’s Tables directory, right‑click the table you want to migrate and choose Copy. Then navigate to your Lakehouse’s Tables directory, right‑click and select Paste. The copy operation is recursive, preserving the partition folders and Parquet files.

2. Automated method via a notebook. Fabric notebooks support Python (synapse_pyspark) and include helper functions (mssparkutils.fs.cp) for file operations. You can use this script below to copy a mirrored table to your Lakehouse and verify the results.

from notebookutils import mssparkutils

# Define the source path for the mirrored table and the destination path for the Lakehouse table
source_path = "abfss://<workspace_id>@onelake.dfs.fabric.microsoft.com/<mirrored_database_id>/Tables/<Folder if Any>/MENU"
destination_path = "abfss://<workspace_id>@onelake.dfs.fabric.microsoft.com/<lakehouse_id>/Tables/dbo/MENU"

print(f"Copying MENU folder from:")
print(f"  Source: {source_path}")
print(f"  To: {destination_path}\n")

try:
    # Recursively copy the entire folder, including all Parquet files and subdirectories
    result = mssparkutils.fs.cp(source_path, destination_path, recurse=True)
    
    print("✓ Copy completed successfully!")
    print("\nVerifying destination folder…")
    
    # List files in destination to verify that the copy worked
    dest_files = mssparkutils.fs.ls(destination_path)
    print(f"Files/folders copied: {len(dest_files)}")
    for item in dest_files:
        item_type = '[DIR]' if item.isDir else '[FILE]'
        print(f"  {item_type} {item.name} ({item.size} bytes)")
        
except Exception as e:
    print("✗ Error during copy operation:")
    print(f"  {str(e)}")

Notes on the code:

• mssparkutils.fs.cp recursively copies a folder when recurse=True. This ensures that nested partition folders and Delta logs are preserved.

• Both source_path and destination_path use the abfss URI scheme to address containers in OneLake. Replace <workspace_id>, <mirrored_database_id> and <lakehouse_id> with your environment’s GUIDs. You can find these IDs in the Fabric portal under the Properties pane for the mirrored database and Lakehouse.

Why this works

Mirroring “manages the replication of data and metadata into OneLake and conversion to Parquet”learn.microsoft.com. Each table in a mirrored database is therefore stored in Parquet/Delta format within OneLake. By copying these files into the Lakehouse’s Tables directory, we effectively re‑create the table in the Lakehouse without needing to run a pipeline. Because Delta metadata (the _delta_log folder) is copied along with the Parquet files, the Lakehouse immediately recognises the table.

However, you should be aware of a few considerations:

• One‑time copy. This method performs a one‑time copy of the data at its current state. After you copy, any new changes in the source database will continue to be mirrored but will not be reflected in the Lakehouse. If you need to migrate incremental changes before shutting down mirroring, run the script again or consider building a pipeline for incremental loads.

• Security and governance. Ensure you have permission to access both the mirrored folder and the Lakehouse folder. Copying data may create duplicates that need to be governed appropriately.

Conclusion

Mirroring is a powerful feature of Microsoft Fabric that lets you replicate data from external databases directly into OneLake in a Delta‑format. When you decide to transition away from mirroring and fully adopt a Lakehouse, copying the underlying Parquet files from the mirrored storage into your Lakehouse’s OneLake can simplify migration. This approach leverages the fact that mirroring already stores data in Parquet/Delta format and avoids the heavy development and testing efforts of building data pipelines.