Forum Discussion

fvl's avatar
fvl
New Member
8 years ago

Real Time log analytics data in PowerBI

Hi,   I am trying to build a real time connection between log analytics and powerBI. A couple minutes delay at most.  I know there is an option to export your log analytics query and import it in ...
Michal_cwiok's avatar
Michal_cwiok
Resolver II
8 years ago

It is feasible, but will hugely depend on your infrastructure.

 

I have never worked with log analitycs, but my bet would be to go with Azure Functions to schedule data retrieval and then push this data into Power BI using Streaming Analytics. However, it can be a simple python script running on your server.

 

I have played with streaming datasets last week - it is super straight forward, but also quite limited. Perhaps if you provide more details, I will be of help.

 

Thanks

  • fvl's avatar
    fvl
    New Member
    8 years ago

    Hi Michal, 

    Thanks for taking the time to reply!

    I have thought of the same thing, but I do not know if querying log analytics every 5 mintues and pushing all that data to powerBI streaming is a good option.

    What do you mean when you say streaming datasets are limited? I haven't worked enough with powerBI, sorry if this is a stupid question. I know that streaming datasets only holds the data in a temporary cache, but apparenly a push dataset can also be used for real time datasets. 


    We have various resources running in Azure, such as a Service Fabric cluster, Kubernetes cluster, hive cluster, spark streaming, ... We want to gather data from these resources into powerBI to show their activity and also to troubleshoot problems. The people who would use this do not have access to the portal so we can't use Azure's built in monitoring solutions. 

    • Michal_cwiok's avatar
      Michal_cwiok
      Resolver II
      8 years ago

      No problem, always happy to help.

       

      You have mentioned the limitations that I was thinking of. There is no underlying database, hence no filtering possibilites. As mentioned by Microsoft: "With a streaming dataset, there is no underlying database, so you cannot build report visuals using the data that flows in from the stream. As such, you cannot make use of report functionality such as filtering, custom visuals, and other report functions." Real-time streaming will be an overkill here.

       

      In your case simplest Direct Query will do the job. Try setting up a small Azure SQL Database. Then schedule a job of pulling log data and pushing it into the DB - I do not know how to do it at the moment, but there are probably multiple ways. Later connect to the DB using Direct Query in PowerBI. This way the data will always be up-to-date, as PowerBI constantly queries the DB for the data. The only limitation here is the volume of the data - a report might not be as responsive, as we would like. In your case with 1000s of rows, this should not be a problem.

       

      Let me know, what you think of this. In 2-3 posts, we might have a whole architecture ready :D

      • ross_lambert's avatar
        ross_lambert
        Regular Visitor
        8 years ago

        You can use a "push dataset" and the Power BI back-end will keep up to 5 million rows, and yes, you can filter on it, too, and create visuals.

         

        I recently wrote an article with source code to a sample Azure Function and submitted to my company's Blog Keepers. When it gets posted I will let you know.

         

        I might be able to send you a copy of the Word doc if you're interested. PM me.

         

        Thanks,

         

        == Ross ==