Forum Discussion
Real Time log analytics data in PowerBI
Hi Michal,
Thanks for taking the time to reply!
I have thought of the same thing, but I do not know if querying log analytics every 5 mintues and pushing all that data to powerBI streaming is a good option.
What do you mean when you say streaming datasets are limited? I haven't worked enough with powerBI, sorry if this is a stupid question. I know that streaming datasets only holds the data in a temporary cache, but apparenly a push dataset can also be used for real time datasets.
We have various resources running in Azure, such as a Service Fabric cluster, Kubernetes cluster, hive cluster, spark streaming, ... We want to gather data from these resources into powerBI to show their activity and also to troubleshoot problems. The people who would use this do not have access to the portal so we can't use Azure's built in monitoring solutions.
No problem, always happy to help.
You have mentioned the limitations that I was thinking of. There is no underlying database, hence no filtering possibilites. As mentioned by Microsoft: "With a streaming dataset, there is no underlying database, so you cannot build report visuals using the data that flows in from the stream. As such, you cannot make use of report functionality such as filtering, custom visuals, and other report functions." Real-time streaming will be an overkill here.
In your case simplest Direct Query will do the job. Try setting up a small Azure SQL Database. Then schedule a job of pulling log data and pushing it into the DB - I do not know how to do it at the moment, but there are probably multiple ways. Later connect to the DB using Direct Query in PowerBI. This way the data will always be up-to-date, as PowerBI constantly queries the DB for the data. The only limitation here is the volume of the data - a report might not be as responsive, as we would like. In your case with 1000s of rows, this should not be a problem.
Let me know, what you think of this. In 2-3 posts, we might have a whole architecture ready :D
You can use a "push dataset" and the Power BI back-end will keep up to 5 million rows, and yes, you can filter on it, too, and create visuals.
I recently wrote an article with source code to a sample Azure Function and submitted to my company's Blog Keepers. When it gets posted I will let you know.
I might be able to send you a copy of the Word doc if you're interested. PM me.
Thanks,
== Ross ==
Hi Ross,
I'm interested for the sample Azure Function.
I don't really understand how to pull the log analytics data using Functions, since it's quite a new thing to me.
I came across this thread when looking for pushing real time data of log analytics to PowerBI and interested to try it with Functions and Stream Analytics.
Is the article available now?
