Forum Discussion
Changing Ingestions from JSON
I have an EventStream receiving IoT Data out of JSONs. In my KQL table the columns timestamp, deviceId and for example x and y exist. The data from one sensor gets ingested in the columns timestamp, deviceId and y. No x since there is no data in the JSON for that. I dont work with a Mapping because it only makes it more complicated and the names in the JSON and in the KQL table are identical.
No I changed the named for data from y to x when creating the JSON. Since there is no mapping and the column already exists in the KQL table the data should be saved now in column x.
My problem is, it doesn't the data is not in x nor in y. It seems to be gone.
Is that a bug in fabric? Can I solve this?
6 Replies
- v-csrikanth
Community Support
Hi MoFabricIoT
We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. And, if you have any further query do let us know.
Thank you.- MoFabricIoTRegular Visitor
Hi v-csrikanth,
i contacted the MS support since fabric does weird things when ingesting the data in the Event House. I hope they can help me.
Thank you for your help!
- v-csrikanth
Community Support
Hi MoFabricIoT
Thanks for testing that. The field not returning even after you changed the JSON back to y is the key clue — the blocking state isn't in your JSON.
Start with one test: check the eventstream's data preview / Live view. If y appears correctly there but never reaches the KQL table, the issue is at the ingestion stage. If it's already missing in the eventstream, the issue is upstream in the inferred schema review it, republish, and send a new event.
If the value reaches the eventstream but not the table, try these in order:
Clear the streaming schema cache. This best fits your symptom. Eventhouse nodes cache the schema of databases that get data through streaming ingestion, and when the cache falls out of sync you can see "incomplete and distorted data in the table". It refreshes automatically, but that "can take several minutes" force it with:.clear table <TableName> cache streamingingestion schema
Check for a database-scoped mapping. Your table-scoped query wouldn't have shown one — .show works for "a database or table":.show database <DatabaseName> ingestion mappingsReview the destination's captured configuration. Direct ingestion creates a Data connection and captures the column setup at that moment. In Live view, select Configure on the Eventhouse destination node, then check Edit columns and the Table_mapping dropdown.
I wouldn't call this a bug until the schema cache and the eventstream schema are ruled out. If all three come back clean and rows still don't land, raise a support ticket via Help (?) with your data connection name, table name and a UTC time window.
Thanks,
Srikanth Cheri - MoFabricIoTRegular Visitor
Thank you for your help. There is no mapping in .show table Table ingestion mappings. I changed it back so the JSON now sends y again and not x anymore (reminder: both columns always existed in the table) and now it doesn't even put it back to column y. It did that before my changes so it should work when it comes to case sensitivity.
It feels like there is an invisible mapping in the background that is initialized in the beginning when using no mapping and cannot be changed afterwards since it doesnt really "exist" in fabric. For me this looks like a bug.
- v-csrikanth
Community Support
Hi MoFabricIoT
The value appearing in neither x nor y is the key clue.
With no explicit mapping, Kusto uses identity mapping for JSON, matching fields to columns by name and case-sensitively. So first confirm the payload contains exactly "x" — not "X", " x", or a nested variant. That alone would reproduce this.
Then check whether a mapping already exists (the Eventstream destination can create one even if you didn't):
.show table <TableName> ingestion mappingsIf it still points to $.y, renaming the producer field won't update it — "Editing the schema doesn't update the mapping of incoming data to table columns during ingestion." Note that an empty .show ingestion failures result doesn't rule this out, as that command doesn't cover every ingestion stage.
To fix, re-issue the mapping with $.x included:
.alter table <TableName> ingestion json mapping "<MappingName>".alter is a full mapping replace, so include every column with its existing path and data type — not just x.
Finally, confirm your destination mode. Under Event processing before ingestion, that layer also shapes which fields reach the table, so the field may be dropped before mapping. And check soon: Eventstream's default retention is one day, and replay is available for Event processing but not Eventhouse (Direct Ingestion).
Quickest path: JSON casing → ingestion mappings → ingestion mode.
Reference links:
- https://learn.microsoft.com/kusto/management/mappings
- https://learn.microsoft.com/kusto/management/alter-ingestion-mapping-command
- https://learn.microsoft.com/fabric/real-time-intelligence/edit-table-schema
- https://learn.microsoft.com/fabric/real-time-intelligence/event-streams/pause-resume-data-streams
Thanks,
C Srikanth
Community Support Team- v-csrikanth
Community Support
Hi MoFabricIoT
We would like to inquire whether have you got the chance to check the solutions provided by other users in community to resolve the issue. We hope the information provided helps to clear the query. Should you have any further queries, kindly feel free to contact the Microsoft Fabric community.