Forum Discussion
Complex RTI Issue: Eventstream Failing to Enforce Schema When Ingesting High-Volume IoT Messages
As you mention, your schema and values change. It sounds like the way you have your eventstream defined today you are enforcing the schema.
But question, if you are sending the data to an event hub first why are you sending it to an eventstream? If it is not behind a private endpoint, just send it directly to eventhouse, there is no need to have the eventstream. Get data from Azure Event Hubs - Microsoft Fabric | Microsoft Learn
If the event hub is behind a private endpoit, you will need the eventstream though. Instead of sending it to 3 destinations (eventhouse, lakehouse, and real-time dashboard), send the data just to Eventhouse using Direct Ingestion. Do not put any transformations in the Eventstream. When you configure Direct Ingestion, on the screen where it asks you for mapping, change the nested json levels down to 0 and bring the entire json object into the row and then do your casting/type checking and everything through update policies. Mirror the data into Lakehouse instead of dual-writing using Eventhouse Onelake availability. This saves you the overhead of having to transform the data in two places.
https://learn.microsoft.com/en-us/fabric/real-time-intelligence/media/get-data-eventstream/inspect-data.png#lightbox
Hello SavioFerraz
The answer given by kustortininja is correct.
Regarding the direct 'Data connection' without Eventstream, consider the Event Hubs option:
In the past, I showed this in a blog post.
I prefer to use the bronze-silver-gold medallion architecture where we ingest the incoming (eventhub/iothub) messages as-is in a dynamic columns via a table mapping in Direct Ingest.
So it's ELT (extract, load, transform) instead of ETL.
So each message is ingested as-is in the bronze layer.
From the original message, you can also take separate values like device id, device type, timestamp, etc. if these help with table update policies towards the silver layer where you create typed columns.
Using an activator you could test for corrumt messages (eg. counting the number of incoming message and the number of transformed message per timespan).
Check out Eventhouse shortcuts in Lakehouse to make an Eventhouse table available as Lakehouse table.
If this answer helps, please upvote it.