Forum Discussion
Complex RTI Issue: Eventstream Failing to Enforce Schema When Ingesting High-Volume IoT Messages
As you mention, your schema and values change. It sounds like the way you have your eventstream defined today you are enforcing the schema.
But question, if you are sending the data to an event hub first why are you sending it to an eventstream? If it is not behind a private endpoint, just send it directly to eventhouse, there is no need to have the eventstream. Get data from Azure Event Hubs - Microsoft Fabric | Microsoft Learn
If the event hub is behind a private endpoit, you will need the eventstream though. Instead of sending it to 3 destinations (eventhouse, lakehouse, and real-time dashboard), send the data just to Eventhouse using Direct Ingestion. Do not put any transformations in the Eventstream. When you configure Direct Ingestion, on the screen where it asks you for mapping, change the nested json levels down to 0 and bring the entire json object into the row and then do your casting/type checking and everything through update policies. Mirror the data into Lakehouse instead of dual-writing using Eventhouse Onelake availability. This saves you the overhead of having to transform the data in two places.
https://learn.microsoft.com/en-us/fabric/real-time-intelligence/media/get-data-eventstream/inspect-data.png#lightbox
As you mention, your schema and values change. It sounds like the way you have your eventstream defined today you are enforcing the schema.
But question, if you are sending the data to an event hub first why are you sending it to an eventstream? If it is not behind a private endpoint, just send it directly to eventhouse, there is no need to have the eventstream. Get data from Azure Event Hubs - Microsoft Fabric | Microsoft Learn
If the event hub is behind a private endpoit, you will need the eventstream though. Instead of sending it to 3 destinations (eventhouse, lakehouse, and real-time dashboard), send the data just to Eventhouse using Direct Ingestion. Do not put any transformations in the Eventstream. When you configure Direct Ingestion, on the screen where it asks you for mapping, change the nested json levels down to 0 and bring the entire json object into the row and then do your casting/type checking and everything through update policies. Mirror the data into Lakehouse instead of dual-writing using Eventhouse Onelake availability. This saves you the overhead of having to transform the data in two places.
https://learn.microsoft.com/en-us/fabric/real-time-intelligence/media/get-data-eventstream/inspect-data.png#lightbox