Forum Discussion
RTI Event Metadata Table Use Cases
Hi everyone, I’m looking to expand how others are using metadata tables in Microsoft Fabric RTI architectures.
Current state
We have an IaC-driven deployment process that provisions RTI assets from source → bronze → silver. As part of this deployment, we persist configuration and lineage information into a metadata table stored in SQL DB.
At a high level, the metadata schema includes:
EventTopic
SourceObjectSettings (JSON) – bronze/silver table names, flatten function, etc.
TopicInformation (JSON) – topic-level semantics
Audit columns (create/modify user + timestamps)
The metadata is written at deploy time and represents the intended RTI topology.
Current primary use case: Bronze → Silver reprocessing
Our main use case today is controlled reprocessing from bronze to silver.
Since Fabric update policies only apply to data in motion, when we need to:
fix a flattening bug
handle schema changes
correct downstream logic
we need a way to deterministically replay historical data.
To support this, I built a parameterized pipeline that accepts:
EventTopic
StartTime
EndTime
The pipeline triggers a fully parameterized notebook that:
Uses the provided EventTopic to query the metadata table
Retrieves the associated flatten function, bronze table, and silver table
Dynamically builds the KQL query
Reprocesses bronze data for the specified time window into silver
This approach avoids hard-coded topic logic and has worked well so far.
What I’m looking for
I’m interested in learning how others are extending this metadata-driven pattern beyond reprocessing, particularly in larger RTI platforms.
What other valuable use cases have you found for metadata tables in RTI architectures?
Any lessons learned or pitfalls would also be appreciated.
Thanks in advance.
Hi matthewlopesdev ,
Thank you for the detailed overview. The pattern you’ve described is well-aligned with how metadata-driven RTI architectures are typically implemented in Microsoft Fabric, especially when deterministic replay is required outside of data-in-motion scenarios.
Beyond bronze-to-silver reprocessing, customers commonly extend metadata tables in the following ways within larger RTI platforms:
Operational monitoring and health checks
Metadata is often used to drive centralized monitoring logic, such as:Tracking active vs. inactive event topics
Identifying ingestion or processing delays per topic
Mapping pipeline or notebook failures back to the originating topicThis allows operational dashboards and alerts to remain generic and scalable.
Schema evolution and version control
Some implementations expand metadata to include:Schema versions and effective timestamps
Compatibility indicators
Topic deprecation or migration statusThis can be leveraged to route events to alternate processing logic or control replay behavior during schema transitions.
Dynamic orchestration and feature control
Metadata can be used to:Enable or disable processing steps per topic
Control enrichment or transformation logic dynamically
Apply topic-specific processing priorities or SLAsThis helps reduce hard-coded logic as the number of topics grows.
Governance and lineage support
Storing semantic, ownership, and classification information in metadata can assist with:End-to-end lineage tracking across RTI layers
Data ownership and stewardship alignment
Integration with governance or cataloging solutions
Lessons learned / considerationsClearly separate deployment-time metadata from runtime state to avoid unintended side effects during processing. Apply validation and versioning to metadata changes, as these effectively act as control inputs for pipelines and notebooks. Restrict write access to metadata tables, as incorrect updates can impact multiple RTI flows.
Overall, your approach reflects a mature RTI design and provides a strong foundation for scalability and operational consistency. We appreciate you sharing this with the community.
Thank you.
4 Replies
Hi matthewlopesdev - Your current approach is already aligned with how mature RTI platforms are evolving in Fabric, treating metadata as a first-class control plane rather than simple document.
Eg:
Typically extend this pattern in several high-value directions beyond bronze → silver reprocessing.
If you continue expanding in that direction, you’re effectively building a scalable RTI framework rather than just pipelines.
Metadata Driven Pipelines for Microsoft Fabric | Microsoft Community Hub
Real Time Intelligence | Fabric Catalyst
Hope this helps.
Hope this helps.
Hi matthewlopesdev ,
Thank you for the detailed overview. The pattern you’ve described is well-aligned with how metadata-driven RTI architectures are typically implemented in Microsoft Fabric, especially when deterministic replay is required outside of data-in-motion scenarios.
Beyond bronze-to-silver reprocessing, customers commonly extend metadata tables in the following ways within larger RTI platforms:
Operational monitoring and health checks
Metadata is often used to drive centralized monitoring logic, such as:Tracking active vs. inactive event topics
Identifying ingestion or processing delays per topic
Mapping pipeline or notebook failures back to the originating topicThis allows operational dashboards and alerts to remain generic and scalable.
Schema evolution and version control
Some implementations expand metadata to include:Schema versions and effective timestamps
Compatibility indicators
Topic deprecation or migration statusThis can be leveraged to route events to alternate processing logic or control replay behavior during schema transitions.
Dynamic orchestration and feature control
Metadata can be used to:Enable or disable processing steps per topic
Control enrichment or transformation logic dynamically
Apply topic-specific processing priorities or SLAsThis helps reduce hard-coded logic as the number of topics grows.
Governance and lineage support
Storing semantic, ownership, and classification information in metadata can assist with:End-to-end lineage tracking across RTI layers
Data ownership and stewardship alignment
Integration with governance or cataloging solutions
Lessons learned / considerationsClearly separate deployment-time metadata from runtime state to avoid unintended side effects during processing. Apply validation and versioning to metadata changes, as these effectively act as control inputs for pipelines and notebooks. Restrict write access to metadata tables, as incorrect updates can impact multiple RTI flows.
Overall, your approach reflects a mature RTI design and provides a strong foundation for scalability and operational consistency. We appreciate you sharing this with the community.
Thank you.Hi matthewlopesdev ,
Just following up to see if the Response provided was helpful in addressing the issue. if the issue still persists Feel free to reach out if you need any further clarification or assistance.
Best regards,
Chaithra E.Hi matthewlopesdev,
May I ask if you have resolved this issue? Please let us know if you have any further issues, we are happy to help.
Thank you.
