MQTT Data Into Fabric

[Kirikugo]

Hi all, 

We have been workshopping this idea for a while now and are struggling to come up with a solution.

My boss has a local script running on his computer (We will be migrating this to a dedicated server in the long run) that is getting the real time Temp / Pressure information from a pi we have. We want to be able to store this in Fabric under a lakehouse to have historical data. 

We have tried installing a data gateway on the computer that is hosting this MQTT and tried subscribing to the topic along with the port address but it always seems to error out. Included are some screenshots of my testing on the Fabric side. But my big question is, do I need to have this on a dedicated server to start with? Or are we just doing something wrong and we can get this temporary solution to work for now?

[v-nmadadi-msft]

Hi @Kirikugo 

May I check if this issue has been resolved? If not, Please feel free to contact us if you have any further questions.

Thank you

[v-nmadadi-msft]

Hi @Kirikugo ,

I wanted to check if you had the opportunity to review the information provided. Please feel free to contact us if you have any further questions.

Thank you.

[svelde]

Hello Kirikugo,

You want to send telemetry from an sensor running somewhere in the world and ingest it into Fabric, especially in an Eventstream for real-time insights.

You have a Raspberry PI it seems you installed some MQTT broker (eg. Mosquitto?) on the RPI to make the data available. 

This can be challlenging if you want to connect from the Eventstream to your device. Your RPI needs to be accessible fot the Eventstream so this means inbound access from the cloud to your device. This is not a secure approach and makes your device and local network vulnerable.

It's recommended to forward the telemetry from the RPI directly to the Eventstream by using eg. the custom source Kafka or EventHub compatible endpoint. You only need to write some custom code on the RPI to make that outbound call. 

A more enterprise is adding a cloud IoT Gateway service in between like the Azure Evengrid MQTT Broker (check this post on how to connect devices to that gateway) or the Azure IoT Hub (how to ingest to the IoT Hub, check this post; how to connect a device, see this blog post).

[lbendlin · Accepted solution]

You can abuse Logstash for that, even with eventhouses.

Ingest Logs using Logstash into Real-Time Intelligence | Microsoft Fabric Blog | Microsoft Fabric

[Kirikugo]

Awesome ok, so If we have an MQTT datastream, we can hook Logstash to grab that data and send it directly to an eventhouse in Fabric? 

[lbendlin]

I think so, Logstash should have connectors for that (in addition to the Kusto connector)