Yazdan's avatar
Yazdan
Icon for Advocate II rankAdvocate II
5 months ago
Status:
New

Microsoft Fabric Mirroring Native Multi-Source Consolidation into a Single Delta Table

Idea / Feature Request:

 

Microsoft Fabric Mirroring is an excellent feature for near real-time operational replication into OneLake, especially for current-state analytics workloads.

 

However, there is currently a major architectural limitation for many enterprise operational sync scenarios:

 

One mirror per source.

 

Today, each source database becomes its own separate mirrored database/table inside Fabric.

Example:

Store DB 1 → Mirrored Table 1
Store DB 2 → Mirrored Table 2
Store DB 3 → Mirrored Table 3

 

But many enterprise architectures require:

 

Multiple source databases → One consolidated Delta table

 

This is very common in:

  • Retail store systems
  • Franchise systems
  • Multi-tenant applications
  • Branch/store operational databases
  • Distributed POS environments
  • Regional operational systems

 

My current use case:

 

I currently ingest the same operational tables from multiple SQL Server store databases into one consolidated Delta table inside Microsoft Fabric Lakehouse.

I achieve this today using only a single dynamic Fabric Copy Activity:

→ SQL Server Sources
→ Direct Delta Upsert
→ One Consolidated Delta Table

 

No notebooks.
No Spark jobs.
No Parquet staging.
No intermediate merge layer.
No additional orchestration.

 

The Copy Activity dynamically:

  • loops through multiple source SQL databases
  • reads similar source tables
  • injects metadata columns dynamically
  • upserts directly into one cumulative Delta table

 

Example metadata columns:

  • DW_LOCATION
  • DW_SERVERNAME
  • DW_SOURCE_SYSTEM
  • DW_PACKAGE_NAME

This architecture provides:

  • very fast operational sync
  • low latency
  • minimal operational overhead
  • simplified maintenance
  • high-performance direct-to-Delta ingestion

 

The problem with current Mirroring:

If I move to Mirroring today, Fabric mirrors each source separately.

That means I would still need:

  • Notebook
  • Pipeline
  • Spark merge process
  • Dataflow
  • or another consolidation layer

to merge all mirrored sources into a single final Delta table.

This removes many of the operational and performance benefits for high-frequency sync workloads.

 

Suggested Feature:

 

It would be extremely valuable if Fabric Mirroring could support:

 

Native Multi-Source → Single Delta Table Consolidation

 

Possible configuration ideas:

  • Multiple source databases mapped into one destination Delta table
  • Automatic source metadata injection
  • Source identifier columns
  • Native merge handling
  • Optional partitioning by source/store/location
  • Dynamic schema alignment
  • Incremental synchronization handling

 

Why this would be very valuable:

 

This would allow Fabric Mirroring to become a true enterprise operational consolidation engine for:

  • high-frequency ingestion
  • large-scale operational replication
  • retail environments
  • multi-store architectures
  • distributed SQL systems
  • multi-tenant operational analytics

 

without requiring additional Spark/Notebook merge layers.

This would significantly reduce:

  • operational complexity
  • compute overhead
  • orchestration requirements
  • maintenance effort
  • end-to-end latency

 

while still preserving Mirroring’s biggest strengths:

  • near real-time replication
  • CDC/change-based sync
  • managed ingestion
  • low source impact
  • OneLake-native integration

I believe this would be a very powerful enhancement for Microsoft Fabric enterprise architectures.

 

Please vote if you find this feature valuable for enterprise operational sync and multi-source consolidation scenarios in Microsoft Fabric Mirroring.😊

No CommentsBe the first to comment

Recent ideas