Forum Discussion
Event Schema Inside Eventstream Never Matching
I am exploring the new Event Schemas as a way to pre-define my schemas and replace the multiple-schema inferencing. I have created a Event Schema Set, with three Event Schemas, all very simple with five or less columns each.
In the screenshot above, one schema shows two fields."TermsCode" and "Description", both string.
I've created a new Event Stream, and associated it with the event schemas:
I then pushed in data, which in the preview, shows the correct matching headers. But the data is only seen as "Unknown schema":
Has anyone gotten event schemas to work?
Edit: I am using the new Event Schemas system, rather than Multiple Schema Inferencing. These are different and exclusionary. I have event streams working with inferred multi-schema and I'm trying to switch to defined event schemas.
Gotcha. Appreciate that.
Turns out though, the issue isn't a mismatch. As I suspected, the fields match. The problem is that I am not sending the required headers, which unfortunately are not yet documented by Microsoft. The Event Schemas use the Cloud Events spec, so we need to send ce_type and a few others. My contact at MS is working on getting a list of which headers are required.
Okay so for anyone else stumbling on this. If you are trying to use Event Schemas inside event streams, you basically can't yet. There are HTTP headers that are required, and they are currently undocumented. I am working with Microsoft now to get a list of the required headers, and I'll update this post when I get that.
6 Replies
Okay so for anyone else stumbling on this. If you are trying to use Event Schemas inside event streams, you basically can't yet. There are HTTP headers that are required, and they are currently undocumented. I am working with Microsoft now to get a list of the required headers, and I'll update this post when I get that.
Hi Kelderic ,
Thanks for reaching out to Community Forum.
The issue happens because Fabric Eventstream strictly validates schemas at runtime. Even if preview looks correct, any mismatch in field names, structure, or data types causes events to be marked as Unknown schema. Also, schema support must be enabled during Eventstream creation, and for multiple schemas, explicit matching rules are required.Fix by recreating the Eventstream with schema enabled, ensuring the payload exactly matches the schema, and configuring schema mapping if multiple schemas are used. Check diagnostics logs to identify mismatches.
Refer these docs:https://learn.microsoft.com/en-us/fabric/real-time-intelligence/schema-sets/use-event-schemas
https://learn.microsoft.com/en-us/fabric/real-time-intelligence/schema-sets/schema-registry-limitations
https://learn.microsoft.com/en-us/fabric/real-time-intelligence/event-streams/add-source-azure-event-hubs?pivots=basic-features
https://learn.microsoft.com/en-us/fabric/real-time-intelligence/event-streams/process-events-with-multiple-schemas
Thank you.Hi there, thanks for the response.
The issue happens because Fabric Eventstream strictly validates schemas at runtime. Even if preview looks correct, any mismatch in field names, structure, or data types causes events to be marked as Unknown schema.
This is what I suspect as well, but as the screenshots show, the data appears to match the pre-defined schema perfectly.
Check diagnostics logs to identify mismatches.How do I see these?
Hi Kelderic ,
To view the diagnostics for schema mismatches, open your Event stream, click on the Event stream item in the canvas, and check the bottom pane. Switch to the Runtime logs tab or the Data insights tab, these show ingestion errors, deserialization failures, and runtime validation issues that usually explain why events get marked as Unknown schema despite the preview looking correct.Currently, Event stream health and error logs are available only inside the Event stream editor UI (Runtime logs / Data Insights)Monitor the Status and Performance of an Eventstream - Microsoft Fabric | Microsoft Learn
Thank you.
