User Profile
Yazdan
Advocate II
Joined 1 year ago
User Widgets
Contributions
Microsoft Fabric Mirroring Native Multi-Source Consolidation into a Single Delta Table
Idea / Feature Request: Microsoft Fabric Mirroring is an excellent feature for near real-time operational replication into OneLake, especially for current-state analytics workloads. However, there is currently a major architectural limitation for many enterprise operational sync scenarios: One mirror per source. Today, each source database becomes its own separate mirrored database/table inside Fabric. Example: Store DB 1 → Mirrored Table 1 Store DB 2 → Mirrored Table 2 Store DB 3 → Mirrored Table 3 But many enterprise architectures require: Multiple source databases → One consolidated Delta table This is very common in: Retail store systems Franchise systems Multi-tenant applications Branch/store operational databases Distributed POS environments Regional operational systems My current use case: I currently ingest the same operational tables from multiple SQL Server store databases into one consolidated Delta table inside Microsoft Fabric Lakehouse. I achieve this today using only a single dynamic Fabric Copy Activity: → SQL Server Sources → Direct Delta Upsert → One Consolidated Delta Table No notebooks. No Spark jobs. No Parquet staging. No intermediate merge layer. No additional orchestration. The Copy Activity dynamically: loops through multiple source SQL databases reads similar source tables injects metadata columns dynamically upserts directly into one cumulative Delta table Example metadata columns: DW_LOCATION DW_SERVERNAME DW_SOURCE_SYSTEM DW_PACKAGE_NAME This architecture provides: very fast operational sync low latency minimal operational overhead simplified maintenance high-performance direct-to-Delta ingestion The problem with current Mirroring: If I move to Mirroring today, Fabric mirrors each source separately. That means I would still need: Notebook Pipeline Spark merge process Dataflow or another consolidation layer to merge all mirrored sources into a single final Delta table. This removes many of the operational and performance benefits for high-frequency sync workloads. Suggested Feature: It would be extremely valuable if Fabric Mirroring could support: Native Multi-Source → Single Delta Table Consolidation Possible configuration ideas: Multiple source databases mapped into one destination Delta table Automatic source metadata injection Source identifier columns Native merge handling Optional partitioning by source/store/location Dynamic schema alignment Incremental synchronization handling Why this would be very valuable: This would allow Fabric Mirroring to become a true enterprise operational consolidation engine for: high-frequency ingestion large-scale operational replication retail environments multi-store architectures distributed SQL systems multi-tenant operational analytics without requiring additional Spark/Notebook merge layers. This would significantly reduce: operational complexity compute overhead orchestration requirements maintenance effort end-to-end latency while still preserving Mirroring’s biggest strengths: near real-time replication CDC/change-based sync managed ingestion low source impact OneLake-native integration I believe this would be a very powerful enhancement for Microsoft Fabric enterprise architectures. Please vote if you find this feature valuable for enterprise operational sync and multi-source consolidation scenarios in Microsoft Fabric Mirroring.😊Microsoft Fabric: Tracking the User Who Actually Triggered a Pipeline or Notebook
After extensive testing, I finally found a reliable way to determine who actually triggered Microsoft Fabric Pipelines and standalone Notebooks. This article explains why common approaches can be misleading and how the Fabric Admin Activity Events API can be used to accurately identify the initiating user.Fabric Copy Activity Upsert Needs Native Audit Columns
𝗠𝗶𝗰𝗿𝗼𝘀𝗼𝗳𝘁 𝗙𝗮𝗯𝗿𝗶𝗰 𝗖𝗼𝗽𝘆 𝗔𝗰𝘁𝗶𝘃𝗶𝘁𝘆 + 𝗗𝗲𝗹𝘁𝗮 𝗨𝗽𝘀𝗲𝗿𝘁 is honestly one of the most underrated features in Fabric for high-performance operational data loading. We can now: • Read directly from sources such as SQL Server / SQL DB • Write directly into Lakehouse Delta Tables • Use native Upsert mode • Avoid Spark notebooks completely • Avoid Parquet staging/intermediate layers • Achieve very fast sync performance with only ONE Copy Activity I’m currently using this approach to dynamically load and merge data from multiple SQL Server store databases into consolidated Delta tables in Microsoft Fabric Lakehouse. However, there is one very important feature still missing from Copy Activity Upsert. 𝗜 𝘄𝗼𝘂𝗹𝗱 𝗹𝗼𝘃𝗲 𝘁𝗼 𝘀𝗲𝗲 𝗻𝗮𝘁𝗶𝘃𝗲 𝘀𝘂𝗽𝗽𝗼𝗿𝘁 𝗳𝗼𝗿 𝗮𝘂𝘁𝗼-𝗺𝗮𝗻𝗮𝗴𝗲𝗱 𝗮𝘂𝗱𝗶𝘁 / 𝗗𝗪 𝗰𝗼𝗹𝘂𝗺𝗻𝘀 during Upsert operations. My requirement is: In the destination Delta table, I need these columns: • DW_DATE_INSERT → Datetime when a NEW record is inserted through Upsert • DW_DATE_UPDATE → Datetime when an EXISTING record is updated through Upsert • DW_LOAD_DATE → Maximum of DW_DATE_INSERT and DW_DATE_UPDATE (meaning the latest insert/update datetime) Expected behavior: • If INSERT occurs: DW_DATE_INSERT should be populated DW_DATE_UPDATE should remain NULL • If UPDATE occurs: DW_DATE_UPDATE should be updated DW_DATE_INSERT should remain unchanged with its original value Currently, adding these columns dynamically in the Source → Additional Columns section is NOT a solution. Why? Because these datetime values change every execution, the Upsert process detects every row as changed and re-updates all records again and again, even when business data has not changed. Of course, I know this can be handled using Spark / Notebook MERGE logic. But the whole beauty of this approach is: 𝗡𝗼 𝗻𝗼𝘁𝗲𝗯𝗼𝗼𝗸𝘀. 𝗡𝗼 𝗦𝗽𝗮𝗿𝗸 𝘀𝘁𝗮𝗿𝘁𝘂𝗽 𝗹𝗮𝘁𝗲𝗻𝗰𝘆. 𝗡𝗼 𝗲𝘅𝘁𝗿𝗮 𝗺𝗲𝗿𝗴𝗲 𝗹𝗮𝘆𝗲𝗿. 𝗝𝘂𝘀𝘁 𝗼𝗻𝗲 𝗳𝗮𝘀𝘁 𝗖𝗼𝗽𝘆 𝗔𝗰𝘁𝗶𝘃𝗶𝘁𝘆. It would be amazing if Microsoft could add native insert/update audit column handling directly inside Copy Activity Upsert for Delta Lakehouse tables. This would make Copy Activity even more powerful for very fast enterprise operational sync workloads. If you think this feature would be useful, please vote/support this idea so we can hopefully get it added to Fabric 🙌 #MicrosoftFabric #DataEngineering #DeltaLake #Lakehouse #ETL #ELT #OneLake #SQLServer #Fabric #Microsoft #AnalyticsEngineeringRe: How to programmatically access “Submitted by” (pipeline runner) at pipeline or notebook runtime
Hi v-dineshya , Thanks for reaching out. I’m not fully sure about the solution yet and was just testing at this stage. If it turns out to be the right approach, I’ll definitely share and mark it here as the solution. Appreciate your follow-up. Regards, YazdanRe: How to programmatically access “Submitted by” (pipeline runner) at pipeline or notebook runtime
Hi tayloramy Thanks for the guidance. I’ve enabled workspace monitoring and can now query the ItemJobEventLogs table. However, in this table I can only see ExecutingPrincipalId, which is the GUID of the user who triggered the pipeline. I need to retrieve the actual user details (given name and family name). Is there another table in the monitoring Eventhouse or somewhere else that contains user metadata which can be joined to ItemJobEventLogs to resolve the user name? Or any recommended approach to map ExecutingPrincipalId to a readable user identity? ThanksRe: How to programmatically access “Submitted by” (pipeline runner) at pipeline or notebook runtime
Hi Deborshi, Thanks for the reply. However, in the result of this query I can’t see the user who triggered the pipeline. I need that information to be available in the pipeline environment at runtime.
