Event Stream fails miserably at reading IoT data from IoT Hub end-point...

[tobiaspe71]

I have worked with Azure IoT solutions for quite some time, previously deploying IoT Hubs, Stream Analytics Queries and down-stream Event Hubs or Blob Storages for lambda architectures. Today I used one of my real-time data generator set-ups, in which I use live sensor data from my iPhone and post that data to a cloud-deployed Node-Red instance. The Node-Red flow does a lot, including sending the parsed data to an IoT Hub. 

I then use the IoT Hub as a data source in an Event Stream. I tried to do some basic filtering logic on the incoming data but fell short. The data seems to be read correctly from the Event Streams first "node" (the IoT Hub), but often it doesn't show any data at all or just fails to read the latests streaming updates. It seems burdened with a lot of intermittent strange behaviours.

From experience I know that Stream Analytics Queries were somewhat cumbersome to use, yet they were stable. Judging from my experience of MS Fabric Real-Time Inteligence today, it is not even ready for use. 

[tobiaspe71]

Well, I have tried a few different demo setups, both professionally and personally.

One of the main crietria for an IoT demo work-wise for us, was to not be dependent on any HW that need to be acquired, configured, distributed and supported by someone.

After some serious solution searching I now use the iPhone app SensorLog that sends telemetry sensor message from any iPhone it is installed on. It sends it over HTTP to an REST publish-data end-point provided in Node-Red.

Node-Red then handles some data transformation and framing, provides a demo UI where one can change the modelled asset to one of eight available ones (Conveyor Belt Motor, Wheel Loader, Wind Mill...). The data can then be be pushed to MQTT, to an IoT Hub or even to IoT Central.

One cool thing is that the app has accelerometer and mic support, meaning that you can shout at and/or shake your phone and the spike shows up in the Node-Red UI, in the data and possibly in the IoT Central Ui. Boom!

[omarecd]

Hello @tobiaspe71 ,

This case sounds actually very interesting.
May I ask, what is the real-time data generator that you used ?
I also have some few for testing and I am actually interested in knowing what other people uses...

Thanks in advance,

Omar C.

[Anonymous]

Hi @tobiaspe71 ,

I'm sorry that you encountered such a problem. But I tested it in my environment and there was no problem:

You did not explain what operations you performed, so I cannot find the problem directly from the existing information. Please provide more complete steps.

The following are the specific steps to connect to the Azure IoT Hub source. You can check whether there is any problem with your operation:
Add Azure IoT Hub source to an eventstream - Microsoft Fabric | Microsoft Learn

You can also check the various logs in Eventhouse monitoring and the Runtime logs in Eventstream. There may be some records that can tell you whether the problem is in the data source or in the eventstream, and whether your data flows from the data source to the eventstream.
Eventhouse monitoring overview - Microsoft Fabric | Microsoft Learn
Monitoring status and performance of an Eventstream item - Microsoft Fabric | Microsoft Learn

often it doesn't show any data at all or just fails to read the latests streaming updates

Based on the sentence you mentioned, I have another suggestion, please ensure that the data ingestion rate into the IoT Hub isn't overwhelming the Eventstream. If the rate is too high, the Eventstream might struggle to keep up with the incoming data. You might want to implement rate limiting or batch processing.
https://learn.microsoft.com/en-us/fabric/real-time-intelligence/event-streams/configure-settings#eve... 

Best Regards,
Dino Tao
If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.