Forum Discussion
Solution SageBest practice for handling schema evolution in Fabric Eventstream before data reaches Eventhouse?
I have an Eventstream receiving operational events where the schema may evolve over time.
For example, the producer initially sends:
DeviceId, Timestamp, Temperature, Statusbut later adds fields such as:
Location, FirmwareVersion, ErrorCodeI want the pipeline to continue ingesting events without breaking downstream KQL tables, update policies, materialized views, or Real-Time Dashboards.
I am trying to understand where schema evolution should ideally be handled in a production Fabric RTI architecture.
Would you:
- enforce the contract upstream using Schema Registry
- normalize changing fields inside Eventstream before Eventhouse ingestion
- land the raw payload first and handle schema evolution inside Eventhouse/KQL
- maintain separate versioned event schemas/tables
How are people handling this in production when producers can add fields without notice?
I am particularly interested in avoiding a design where every small upstream schema change forces updates across Eventstream, KQL tables, update policies, and downstream dashboards.
Hello ShivekMaharaj ,
Welcome to this Microsoft Fabric forum.
I have used the Eventstream schema support a few times already.
I mainly use it to split up messages coming from multiple devices or services but arriving at the same Eventstream source.
Think of an Azure IoT Operations device having a connection to an Eventstream custom endpoint.
This device sends out multiple types of messages, so I can split them out in the Eventstream towards different destinations.
So I can reuse the same 'pipeline' from the edge.
The same goes for multiple sources, each having its own kind of message type, and again, these must be redirected to different destinations.
There are challenges.
I tried to use this on a source with messages from many LoRaWAN devices. Within one hour, I got 50+ message schemas from ~15 devices.
This was because some devices sent multiple messages, even partial messages (having a subset of the properties. This could differ. For example, when a GPS sensor had no fix, the latitude and longitude were left out.
This abundance of schemas (which keeps growing) made practical usage of schemas impossible.
Partial messages, while having schemas enabled, could lead to dropped messages, and there is no dead-letter box.
So, I only use the schemas for filtering if I'm in control of the message formats and the flow demands splitting the
stream in the Eventstream, too.
Otherwise, I advise just ingesting and passing on all (raw) messages to an Eventhouse (as a dynamic) and making use of table update policies so at least the raw messages are persisted.
Then, in case downstream functionality fails due to a changing format, the changed (raw) messages are still persisted, and you can act on the changed formats.
---
If this answer helps you, marking it as the accepted answer is appreciated. All community members with similar questions will benefit by doing so. Your contribution is highly appreciated.
2 Replies
- tayloramy
Super User
Hi ShivekMaharaj,
I have not tried this myself, but there is a preview feature that you may want to check out:
https://learn.microsoft.com/en-us/fabric/real-time-intelligence/event-streams/process-events-with-multiple-schemas - svelde
Hello ShivekMaharaj ,
Welcome to this Microsoft Fabric forum.
I have used the Eventstream schema support a few times already.
I mainly use it to split up messages coming from multiple devices or services but arriving at the same Eventstream source.
Think of an Azure IoT Operations device having a connection to an Eventstream custom endpoint.
This device sends out multiple types of messages, so I can split them out in the Eventstream towards different destinations.
So I can reuse the same 'pipeline' from the edge.
The same goes for multiple sources, each having its own kind of message type, and again, these must be redirected to different destinations.
There are challenges.
I tried to use this on a source with messages from many LoRaWAN devices. Within one hour, I got 50+ message schemas from ~15 devices.
This was because some devices sent multiple messages, even partial messages (having a subset of the properties. This could differ. For example, when a GPS sensor had no fix, the latitude and longitude were left out.
This abundance of schemas (which keeps growing) made practical usage of schemas impossible.
Partial messages, while having schemas enabled, could lead to dropped messages, and there is no dead-letter box.
So, I only use the schemas for filtering if I'm in control of the message formats and the flow demands splitting the
stream in the Eventstream, too.
Otherwise, I advise just ingesting and passing on all (raw) messages to an Eventhouse (as a dynamic) and making use of table update policies so at least the raw messages are persisted.
Then, in case downstream functionality fails due to a changing format, the changed (raw) messages are still persisted, and you can act on the changed formats.
---
If this answer helps you, marking it as the accepted answer is appreciated. All community members with similar questions will benefit by doing so. Your contribution is highly appreciated.
