Changing Ingestions from JSON

Hi svelde​ , 

I tried everything with Mapping and a strict schema and it never worked. Now I ingest the raw JSON Data in one "payload" column, this produces new problems for the MLV and the Incremental Refresh later, but at least I am sure I save the data this way.

The problem is that I have different devices sending for different columns and Fabric only takes the first incoming message when letting Fabric generate a Schema and Mapping automatically.

And when I declare a Schema and Mapping weird things like doubled columns are generated by Fabric, which looks more like a bug

Hi MoFabricIoT​ 

The value appearing in neither x nor y is the key clue.

With no explicit mapping, Kusto uses identity mapping for JSON, matching fields to columns by name and case-sensitively. So first confirm the payload contains exactly "x" — not "X", " x", or a nested variant. That alone would reproduce this.

Then check whether a mapping already exists (the Eventstream destination can create one even if you didn't):

.show table <TableName> ingestion mappings

If it still points to $.y, renaming the producer field won't update it — "Editing the schema doesn't update the mapping of incoming data to table columns during ingestion." Note that an empty .show ingestion failures result doesn't rule this out, as that command doesn't cover every ingestion stage.

To fix, re-issue the mapping with $.x included:

.alter table <TableName> ingestion json mapping "<MappingName>"

.alter is a full mapping replace, so include every column with its existing path and data type — not just x.

Finally, confirm your destination mode. Under Event processing before ingestion, that layer also shapes which fields reach the table, so the field may be dropped before mapping. And check soon: Eventstream's default retention is one day, and replay is available for Event processing but not Eventhouse (Direct Ingestion).

Quickest path: JSON casing → ingestion mappings → ingestion mode.

Reference links:

• https://learn.microsoft.com/kusto/management/mappings
• https://learn.microsoft.com/kusto/management/alter-ingestion-mapping-command
• https://learn.microsoft.com/fabric/real-time-intelligence/edit-table-schema
• https://learn.microsoft.com/fabric/real-time-intelligence/event-streams/pause-resume-data-streams

Thanks,
C Srikanth
Community Support Team

Hi MoFabricIoT​ 

Thanks for testing that. The field not returning even after you changed the JSON back to y is the key clue — the blocking state isn't in your JSON.

Start with one test: check the eventstream's data preview / Live view. If y appears correctly there but never reaches the KQL table, the issue is at the ingestion stage. If it's already missing in the eventstream, the issue is upstream in the inferred schema review it, republish, and send a new event.

If the value reaches the eventstream but not the table, try these in order:

Clear the streaming schema cache. This best fits your symptom. Eventhouse nodes cache the schema of databases that get data through streaming ingestion, and when the cache falls out of sync you can see "incomplete and distorted data in the table". It refreshes automatically, but that "can take several minutes" force it with:

.clear table <TableName> cache streamingingestion schema 

Check for a database-scoped mapping. Your table-scoped query wouldn't have shown one — .show works for "a database or table":

.show database <DatabaseName> ingestion mappings

Review the destination's captured configuration. Direct ingestion creates a Data connection and captures the column setup at that moment. In Live view, select Configure on the Eventhouse destination node, then check Edit columns and the Table_mapping dropdown.

I wouldn't call this a bug until the schema cache and the eventstream schema are ruled out. If all three come back clean and rows still don't land, raise a support ticket via Help (?) with your data connection name, table name and a UTC time window.

Thanks,
Srikanth Cheri

Thank you for your help. There is no mapping in .show table Table ingestion mappings. I changed it back so the JSON now sends y again and not x anymore (reminder: both columns always existed in the table) and now it doesn't even put it back to column y. It did that before my changes so it should work when it comes to case sensitivity.

It feels like there is an invisible mapping in the background that is initialized in the beginning when using no mapping and cannot be changed afterwards since it doesnt really "exist" in fabric. For me this looks like a bug.

Hi MoFabricIoT​ 
We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. And, if you have any further query do let us know.

Thank you.

Hello MoFabricIoT​ ,

welcome to Fabric Community portal.

I recommend working with the KQL table mapping, especially in cases like this where the automatic mapping is not clear or when the message format changes.

Yes, there are two steps in the Fabric Eventstream direct ingest flow (first making the connection, then creating the mapping), but this gives you full control over ingestion.

Actually, the Fabric Eventstream comes with a visual editor to build the mapping in a UI way.

Please give it another try and experience how you can ingest with confidence that you are in control over this step of the flow.

---

If this answer helps you, marking it as the accepted answer is appreciated. All community members with similar questions will benefit by doing so. Your contribution is highly appreciated.

Hi MoFabricIoT​ 

Thanks for the update. The behaviors you're seeing are consistent with documented Kusto schema and ingestion behavior.

The duplicated columns

.create-merge table extends an existing table schema by adding columns that don't already exist, while retaining existing columns. Kusto identifiers, including column names, are case-sensitive. Therefore, differences in column naming can result in separate identifiers rather than replacing an existing column.

The missing values

When mapped fields are missing from the incoming data, Kusto fills the corresponding value with NULL. This can occur without an ingestion failure, which explains why nothing appeared in .show ingestion failures.

This also allows a single table to accommodate different payloads. If one device sends x and another sends y, the column not present in a particular payload can remain NULL.

Using a raw JSON column:

Your current approach of landing the raw JSON in a payload column can be used as the source for a structured target table. An update policy can transform the JSON during ingestion and write the extracted fields into the typed table.

KQL

.create-merge table SensorData ( timestamp:datetime, deviceId:string, x:real, y:real ) .create-or-alter function ParseSensorPayload() { RawSensorData | extend p = todynamic(payload) | project timestamp = todatetime(p.timestamp), deviceId = tostring(p.deviceId), x = toreal(p.x), y = toreal(p.y) } .alter table SensorData policy update

JSON

[ { "IsEnabled": true, "Source": "RawSensorData", "Query": "ParseSensorPayload()", "IsTransactional": false, "PropagateIngestionProperties": false } ]

The source and target tables must be in the same database, and the update policy function schema and target table schema must match in column names, types, and order.

For the materialized view, build it over SensorData rather than the raw payload table and keep the transformation schema deterministic. Microsoft documents that bag_unpack and pivot can produce a nondeterministic schema, which can cause a materialized view to be automatically disabled.

Since you already have a Microsoft Support case open, the observations about automatic schema generation and duplicated columns should also be included in that investigation.

References:

• .create-merge table command
• Entity names
• How invalid data affects ingestion
• .alter table policy update command
• Materialized views limitations

Thanks,
C Srikanth
Community Support Team

Hi MoFabricIoT​ ,

One thing that might be behind this is the Eventstream schema rather than the KQL table. When Eventstream infers a schema, it's based on the messages it sees early on, and fields that weren't in that schema can get dropped before they ever reach the Eventhouse. That would explain why the value ends up in neither column, and why switching back didn't fix it once the schema had changed.

A simple fix on the producer side is to have every device send the full set of fields, with null for anything it doesn't measure. So a device that only reports y still sends x as null. That way the inferred schema always includes every column and different devices stop overwriting each other's shape.

Quick test to confirm, send one message with both x and y populated and check whether both land. If they do, the schema is the culprit.

Your raw payload approach with an update policy is a solid fallback too, and it keeps your materialized views working off a clean typed table.