Forum Discussion
Missing data between Azure Stream Analytics and Power BI
Hi,
I have the following results (just an extract, the whole result table has more columns) of a test output in my Azure Stream Analytics job (one event every second):
time | conductivity |
"2020-04-16T04:52:36.0000000Z" | 977,898418 |
"2020-04-16T04:52:37.0000000Z" | 977,998421 |
"2020-04-16T04:52:38.0000000Z" | 978,098423 |
"2020-04-16T04:52:39.0000000Z" | 978,198425 |
"2020-04-16T04:52:40.0000000Z" | 978,298428 |
"2020-04-16T04:52:41.0000000Z" | 978,398430 |
"2020-04-16T04:52:42.0000000Z" | 978,498432 |
"2020-04-16T04:52:43.0000000Z" | 978,598434 |
"2020-04-16T04:52:44.0000000Z" | 978,698437 |
"2020-04-16T04:52:45.0000000Z" | 978,798439 |
"2020-04-16T04:52:46.0000000Z" | 978,898441 |
"2020-04-16T04:52:47.0000000Z" | 978,998444 |
"2020-04-16T04:52:48.0000000Z" | 979,098446 |
"2020-04-16T04:52:49.0000000Z" | 979,198448 |
"2020-04-16T04:52:50.0000000Z" | 979,298450 |
"2020-04-16T04:52:51.0000000Z" | 979,398453 |
"2020-04-16T04:52:52.0000000Z" | 979,498455 |
"2020-04-16T04:52:53.0000000Z" | 979,598457 |
"2020-04-16T04:52:54.0000000Z" | 979,698460 |
"2020-04-16T04:52:55.0000000Z" | 979,798462 |
"2020-04-16T04:52:56.0000000Z" | 979,898464 |
"2020-04-16T04:52:57.0000000Z" | 979,998466 |
"2020-04-16T04:52:58.0000000Z" | 980,098469 |
"2020-04-16T04:52:59.0000000Z" | 980,198471 |
"2020-04-16T04:53:00.0000000Z" | 980,298473 |
"2020-04-16T04:53:01.0000000Z" | 980,398476 |
"2020-04-16T04:53:02.0000000Z" | 980,498478 |
"2020-04-16T04:53:03.0000000Z" | 980,598480 |
"2020-04-16T04:53:04.0000000Z- | 980,698483 |
The output is configured as Power BI. In Power BI I have set up a simple table to show the events. As you can see there is some data missing between 16.04.2020 04:52:38 and 16.04.2020 04:52:57.
time | Conductivity |
16.04.2020 04:53:08 | 981,10 |
16.04.2020 04:53:07 | 981,00 |
16.04.2020 04:53:06 | 980,90 |
16.04.2020 04:53:05 | 980,80 |
16.04.2020 04:53:04 | 980,70 |
16.04.2020 04:53:03 | 980,60 |
16.04.2020 04:53:02 | 980,50 |
16.04.2020 04:53:01 | 980,40 |
16.04.2020 04:53:00 | 980,30 |
16.04.2020 04:52:59 | 980,20 |
16.04.2020 04:52:58 | 980,10 |
16.04.2020 04:52:57 | 980,00 |
16.04.2020 04:52:38 | 978,00 |
16.04.2020 04:52:37 | 978,00 |
16.04.2020 04:52:36 | 977,90 |
16.04.2020 04:52:35 | 977,80 |
16.04.2020 04:52:34 | 977,70 |
16.04.2020 04:52:33 | 977,60 |
This is happening nearly every minute.
- Is there a way to debug the connection between Azure Stream Analytics Output and Power BI?
- Is there a limit of data which I can transfer per second?
Thank you for your help. If any information is missing please do not hesitate to ask for it!
Hello Anonymous,
thank you for you reply.
After a further investigation I came to the result that my IoT device collected data until it has reached 20KB or 60s have expired. I decreased the collected data to 4KB so the sending interval is shorter. Now, there is no data lost anymore.
Best Wishes,
HelgeFP
4 Replies
So are you using push or streaming? I am guessing push?
https://docs.microsoft.com/en-us/power-bi/service-real-time-streaming
I am using Azure Stream Analytics to transfer the data to Power BI, so I think I use "push" then (Using Azure Stream Analytics to push data).
I have configured "Power BI" as an output for my Stream Analytics job. In Power BI I have created a report from the dataset which was generated by the Stream Analytics job.
I also have created a dashboard which shows an equal behaviour: The mesaurements refresh every second and at a certain point everything stands still for 20s (like I mentioned in my first post). And so on (measurements do refresh for ~40s and stop for ~20s...).
fr1337ddy
There isn't many details in the Power BI document, I would recommend you to follow the Azure Steram Analytic document: https://docs.microsoft.com/en-us/azure/stream-analytics/stream-analytics-power-bi-dashboardThere are 2 types of logs for debugging:
https://docs.microsoft.com/en-us/azure/stream-analytics/stream-analytics-job-diagnostic-logs
For further assistance, you will get more help and confirmed answers here: Azure Stream Analytics forum.
Paul Zheng _ Community Support Team
If this post helps, then please consider Accept it as the solution to help the other members find it more quickly
