tips and tricks

3 Topics

Best practice for handling schema evolution in Fabric Eventstream before data reaches Eventhouse?
I have an Eventstream receiving operational events where the schema may evolve over time. For example, the producer initially sends: DeviceId, Timestamp, Temperature, Status but later adds fields such as: Location, FirmwareVersion, ErrorCode I want the pipeline to continue ingesting events without breaking downstream KQL tables, update policies, materialized views, or Real-Time Dashboards. I am trying to understand where schema evolution should ideally be handled in a production Fabric RTI architecture. Would you: enforce the contract upstream using Schema Registry normalize changing fields inside Eventstream before Eventhouse ingestion land the raw payload first and handle schema evolution inside Eventhouse/KQL maintain separate versioned event schemas/tables How are people handling this in production when producers can add fields without notice? I am particularly interested in avoiding a design where every small upstream schema change forces updates across Eventstream, KQL tables, update policies, and downstream dashboards.
Solved
ShivekMaharaj
9 days ago Place Real-Time Intelligence
89Views
0likes
2Comments
Working with Microsoft Fabric Support - A Collaborative Approach
At Microsoft Fabric Support, our goal is simple: help you resolve issues as quickly and smoothly as possible. Just like the recommendations shared in the Microsoft Fabric Community’s guidance on getting questions answered effectively, providing clear and complete information upfront helps everyone move faster and avoid unnecessary back-and-forth. Support is most effective when it’s a collaborative process. You bring the knowledge of your environment and workloads, and we bring deep platform expertise and diagnostics to help investigate the issue together. Help Us Help You When opening a support case, please include as much of the following information as possible: Issue Details Clear description of the problem Expected behavior vs. actual behavior Is the issue intermittent or consistently reproducible? Approximate timeframe of when the issue occurred (UTC preferred) Error Information Please include: Full error message text Screenshots (highly recommended) Activity IDs / Correlation IDs if available Even small details can significantly speed up investigation. Environment Information That Helps Investigation Depending on the Fabric workload being used, please provide the relevant item IDs and environment details. Workspace Information Workspace ID / Workspace URL To find this: Open the Fabric workspace Copy the URL from your browser Example: ".. https://app.fabric.microsoft.com/groups/<WorkspaceID>/… …" The value after /groups/ is the Workspace ID. Screenshot example above showcases the WorkspaceID and EventhouseID. Fabric Item IDs Please provide the item ID related to the affected workload, such as: EventhouseID ActivatorID LakehouseID WarehouseID SemanticModelID EventstreamID PipelineID These IDs help us locate telemetry and backend diagnostics more efficiently. You can typically find these: In the browser URL while inside the Fabric item Within item settings/details pages From the Fabric portal navigation pane Additional Environment Details Please also include: "… WorkspaceID: Capacity Name: Region …" Example: "… WorkspaceID: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx Capacity Name: Fabric-Prod-EastUS Region: East US …" This information helps us identify: Capacity-level issues Regional service impact Configuration-specific behavior Why This Matters Microsoft Fabric is a distributed cloud platform, and troubleshooting often depends on correlating logs, telemetry, timestamps, and resource identifiers together. Providing complete details upfront helps: Reduce delays Avoid repeated clarification requests Accelerate root cause analysis Create a smoother support experience for everyone involved We’re here to partner with you throughout the process, and we truly appreciate your collaboration in helping us investigate issues effectively. We look forward to working with you!
krlitt
1 month ago Place Real-Time Intelligence
1.1KViews
5likes
3Comments
RTI Event Metadata Table Use Cases
Hi everyone, I’m looking to expand how others are using metadata tables in Microsoft Fabric RTI architectures. Current state We have an IaC-driven deployment process that provisions RTI assets from source → bronze → silver. As part of this deployment, we persist configuration and lineage information into a metadata table stored in SQL DB. At a high level, the metadata schema includes: EventTopic SourceObjectSettings (JSON) – bronze/silver table names, flatten function, etc. TopicInformation (JSON) – topic-level semantics Audit columns (create/modify user + timestamps) The metadata is written at deploy time and represents the intended RTI topology. Current primary use case: Bronze → Silver reprocessing Our main use case today is controlled reprocessing from bronze to silver. Since Fabric update policies only apply to data in motion, when we need to: fix a flattening bug handle schema changes correct downstream logic we need a way to deterministically replay historical data. To support this, I built a parameterized pipeline that accepts: EventTopic StartTime EndTime The pipeline triggers a fully parameterized notebook that: Uses the provided EventTopic to query the metadata table Retrieves the associated flatten function, bronze table, and silver table Dynamically builds the KQL query Reprocesses bronze data for the specified time window into silver This approach avoids hard-coded topic logic and has worked well so far. What I’m looking for I’m interested in learning how others are extending this metadata-driven pattern beyond reprocessing, particularly in larger RTI platforms. What other valuable use cases have you found for metadata tables in RTI architectures? Any lessons learned or pitfalls would also be appreciated. Thanks in advance.
Solved
matthewlopesdev
7 months ago Place Real-Time Intelligence
7.1KViews
2likes
4Comments