RTI Event Metadata Table Use Cases

[matthewlopesdev]

Hi everyone, I’m looking to expand how others are using metadata tables in Microsoft Fabric RTI architectures.

Current state

We have an IaC-driven deployment process that provisions RTI assets from source → bronze → silver. As part of this deployment, we persist configuration and lineage information into a metadata table stored in SQL DB.

At a high level, the metadata schema includes:

• EventTopic

• SourceObjectSettings (JSON) – bronze/silver table names, flatten function, etc.

• TopicInformation (JSON) – topic-level semantics

• Audit columns (create/modify user + timestamps)

The metadata is written at deploy time and represents the intended RTI topology.

Current primary use case: Bronze → Silver reprocessing

Our main use case today is controlled reprocessing from bronze to silver.

Since Fabric update policies only apply to data in motion, when we need to:

• fix a flattening bug

• handle schema changes

• correct downstream logic

we need a way to deterministically replay historical data.

To support this, I built a parameterized pipeline that accepts:

• EventTopic

• StartTime

• EndTime

The pipeline triggers a fully parameterized notebook that:

1. Uses the provided EventTopic to query the metadata table

2. Retrieves the associated flatten function, bronze table, and silver table

3. Dynamically builds the KQL query

4. Reprocesses bronze data for the specified time window into silver

This approach avoids hard-coded topic logic and has worked well so far.

What I’m looking for

I’m interested in learning how others are extending this metadata-driven pattern beyond reprocessing, particularly in larger RTI platforms.

What other valuable use cases have you found for metadata tables in RTI architectures?
Any lessons learned or pitfalls would also be appreciated.

Thanks in advance.

[v-echaithra]

Hi @matthewlopesdev,

May I ask if you have resolved this issue? Please let us know if you have any further issues, we are happy to help.

Thank you.

[v-echaithra]

Hi @matthewlopesdev ,

Just following up to see if the Response provided was helpful in addressing the issue. if the issue still persists Feel free to reach out if you need any further clarification or assistance.

Best regards,
Chaithra E.

[v-echaithra · Accepted solution]

Hi @matthewlopesdev ,

Thank you for the detailed overview. The pattern you’ve described is well-aligned with how metadata-driven RTI architectures are typically implemented in Microsoft Fabric, especially when deterministic replay is required outside of data-in-motion scenarios.

Beyond bronze-to-silver reprocessing, customers commonly extend metadata tables in the following ways within larger RTI platforms:

Operational monitoring and health checks

Metadata is often used to drive centralized monitoring logic, such as:

Tracking active vs. inactive event topics
Identifying ingestion or processing delays per topic
Mapping pipeline or notebook failures back to the originating topic

This allows operational dashboards and alerts to remain generic and scalable.

Schema evolution and version control

Some implementations expand metadata to include:

Schema versions and effective timestamps
Compatibility indicators
Topic deprecation or migration status

This can be leveraged to route events to alternate processing logic or control replay behavior during schema transitions.

Dynamic orchestration and feature control

Metadata can be used to:

Enable or disable processing steps per topic
Control enrichment or transformation logic dynamically
Apply topic-specific processing priorities or SLAs

This helps reduce hard-coded logic as the number of topics grows.

Governance and lineage support

Storing semantic, ownership, and classification information in metadata can assist with:

End-to-end lineage tracking across RTI layers
Data ownership and stewardship alignment
Integration with governance or cataloging solutions
Lessons learned / considerations

Clearly separate deployment-time metadata from runtime state to avoid unintended side effects during processing. Apply validation and versioning to metadata changes, as these effectively act as control inputs for pipelines and notebooks. Restrict write access to metadata tables, as incorrect updates can impact multiple RTI flows.

Overall, your approach reflects a mature RTI design and provides a strong foundation for scalability and operational consistency. We appreciate you sharing this with the community.

Thank you.

[rajendraongole1]

Hi @matthewlopesdev  - Your current approach is already aligned with how mature RTI platforms are evolving in Fabric, treating metadata as a first-class control plane rather than simple document.

Eg: 

Typically extend this pattern in several high-value directions beyond bronze → silver reprocessing.

If you continue expanding in that direction, you’re effectively building a scalable RTI framework rather than just pipelines.

Metadata Driven Pipelines for Microsoft Fabric | Microsoft Community Hub

Real Time Intelligence | Fabric Catalyst

Hope this helps.

Hope this helps.