RTI Event Metadata Table Use Cases

[matthewlopesdev]

Hi everyone, I’m looking to expand how others are using metadata tables in Microsoft Fabric RTI architectures.

Current state

We have an IaC-driven deployment process that provisions RTI assets from source → bronze → silver. As part of this deployment, we persist configuration and lineage information into a metadata table stored in SQL DB.

At a high level, the metadata schema includes:

• EventTopic

• SourceObjectSettings (JSON) – bronze/silver table names, flatten function, etc.

• TopicInformation (JSON) – topic-level semantics

• Audit columns (create/modify user + timestamps)

The metadata is written at deploy time and represents the intended RTI topology.

Current primary use case: Bronze → Silver reprocessing

Our main use case today is controlled reprocessing from bronze to silver.

Since Fabric update policies only apply to data in motion, when we need to:

• fix a flattening bug

• handle schema changes

• correct downstream logic

we need a way to deterministically replay historical data.

To support this, I built a parameterized pipeline that accepts:

• EventTopic

• StartTime

• EndTime

The pipeline triggers a fully parameterized notebook that:

1. Uses the provided EventTopic to query the metadata table

2. Retrieves the associated flatten function, bronze table, and silver table

3. Dynamically builds the KQL query

4. Reprocesses bronze data for the specified time window into silver

This approach avoids hard-coded topic logic and has worked well so far.

What I’m looking for

I’m interested in learning how others are extending this metadata-driven pattern beyond reprocessing, particularly in larger RTI platforms.

What other valuable use cases have you found for metadata tables in RTI architectures?
Any lessons learned or pitfalls would also be appreciated.

Thanks in advance.

[v-echaithra]

Hi @matthewlopesdev ,

I hope the above details help you fix the issue. If you still have any questions or need more help, feel free to reach out. We are always here to support you.

Thank you.

[rajendraongole1]

Hi @matthewlopesdev  - Your current approach is already aligned with how mature RTI platforms are evolving in Fabric, treating metadata as a first-class control plane rather than simple document.

Eg: 

Typically extend this pattern in several high-value directions beyond bronze → silver reprocessing.

If you continue expanding in that direction, you’re effectively building a scalable RTI framework rather than just pipelines.

Metadata Driven Pipelines for Microsoft Fabric | Microsoft Community Hub

Real Time Intelligence | Fabric Catalyst

Hope this helps.

Hope this helps.