Preserve IoT Hub system properties and user properties

[uriel-kluk]

Description
Events from the IoT Hub source drop key properties. We need full pass-through of system and user properties for routing and lineage.

User Example
Diego runs a fleet of smart vending machines. Each machine sends telemetry with a user property message-type=timeSeries. When stock is low, the device sends message-type=alert. Diego needs to route alerts directly to a monitoring system. Without user properties, he cannot build a routing rule and must instead parse message bodies, adding complexity and cost.

Expected Result

• Map system properties to columns: deviceId, moduleId, enqueuedTime, messageId, correlationId, contentType, contentEncoding, connectionAuthMethod.

• Preserve all user properties as a bag.

• Expose user and system property keys for transformations.

• DeviceId is always present.

• IoTHub Enriched metadata is present.

Actual Result

• DeviceId and other system properties are missing.

• User properties do not arrive.

Impact

• Cannot route by user property like message-type.

• Cannot trace messages to devices without joins.

References
IoT Hub system properties: https://learn.microsoft.com/en-us/azure/iot-hub/iot-hub-devguide-messages-construct#system-propertie...

Repro Steps

1. Send a D2C message with user property message-type=alarms.

2. Inspect the record in Eventstream.

3. Observe missing deviceId and missing user properties.

Acceptance Criteria

• Given any D2C message, deviceId is a column.

• Given user properties, all appear in a properties bag.

• Given message-type=alarms, routing rules can use it directly.

• Given contentType and contentEncoding, values match the source message.

[uriel-kluk]

@tayloramy thanks for the detailed explanation.

I was able to split the message-type as you suggested using IoT Hub routing.

After enabling Multiple Schemas (Preview), I started receiving the System Properties as expected.

However, unless I'm missing something, it looks like the Application Properties and IoT Hub Enrichments are still not coming through. I can continue working now that the data is split, but this does feel like a limitation.

Giving you Kudos and accepting the solution, but I am going to add a feature request to have the full message projected.

[tayloramy]

Hi @uriel-kluk, 

I agree that this is a limitation. Once you create the idea for the feature request, post the link here and I will vote for it. 

You can create the idea here: https://community.fabric.microsoft.com/t5/Fabric-Ideas/idb-p/fbc_ideas

If you found this helpful, consider giving some Kudos. If I answered your question or solved your problem, mark this post as the solution.

[v-sgandrathi]

Hi @uriel-kluk,

Thank you @tayloramy for your response to thq query.

Has your issue been resolved?
If the response provided by the community member addressed your query, could you please confirm? It helps us ensure that the solutions provided are effective and beneficial for everyone.

Thank you for your understanding!

[tayloramy · Accepted solution]

Hi @uriel-kluk, 

You’re not missing anything. Today, the Eventstream IoT Hub source doesn’t surface IoT Hub SystemProperties or Application (user) properties into the canvas/operators, so fields like deviceId, moduleId, enqueuedTime, messageId, contentType/contentEncoding, and user property bags don’t show up. That’s why routing on a user property like message-type isn’t possible inside Eventstream yet. For reference, the system/user properties you’re expecting are defined by IoT Hub itself here: IoT Hub message system properties, and the Eventstream IoT Hub connector setup is here: Add IoT Hub source to Eventstream.

Here are options you can use right now:

1) Route at IoT Hub (fastest and keeps costs down)
Define routes on IoT Hub using the Application property you already send, e.g. message-type='alert', so “alerts” and “timeSeries” get split at the source. Then point Fabric at the appropriate endpoints. IoT Hub routing query syntax is documented here: Routing query syntax.
• Common pattern: IoT Hub -> route “alerts” to a dedicated Event Hub (or Storage), and “timeSeries” to another endpoint. In Fabric, add the Event Hub(s) as Eventstream sources: Add Event Hubs source.

2) Preserve system properties via Event Hubs -> KQL mapping
If your goal is lineage and analytics in KQL/Eventhouse, connect to Event Hubs and map system properties into columns when landing the data. Fabric’s Event Hubs “Get Data” flow supports embedding system properties (device id, enqueued time, etc.) into the table schema: Get data from Azure Event Hubs (system properties mapping).

3) Enrich upstream, then send to Fabric
Use Azure Stream Analytics or an Azure Function to read from IoT Hub, project SystemProperties and ApplicationProperties into the JSON body, and forward into Fabric via a Custom App/endpoint source. ASA exposes these values with functions like GetMetadataPropertyValue: ASA GetMetadataPropertyValue. Fabric Custom App/endpoint source: Add a custom endpoint/app source.

A few extra notes that help avoid surprises:
• If you plan to query the body at IoT Hub (for body-based routing), set contentType=application/json and a UTF contentEncoding; IoT Hub only evaluates body queries when those are set: Routing query requirements.
• Device identity info normally lives in IoT Hub system properties (e.g., iothub-connection-device-id): System properties list.

If you want this supported natively in Eventstream (ideal: “preserve all user properties as a bag” and “project IoT Hub system properties into columns”), please add/vote an idea under Real-Time Intelligence | Eventstream so the product group can track it: Fabric Ideas (Eventstream).

If you found this helpful, consider giving some Kudos. If I answered your question or solved your problem, mark this post as the solution.

[uriel-kluk]

Hi @tayloramy

GetMetadataPropertyValue still fails to return values in IoTHub or the built-in EventHub.
The new schema registry cannot use Application and System properties.

I am adding all the properties directly to the body, but for deployed fleets, this is not acceptable.

[tayloramy]

Hi @uriel-kluk, 

All I can suggest is you create an idea and hope that the product team sees it. 

I do not work for Microsoft nor do I have access to the product teams to directly give feature requests.