Stream data from Azure App Insights to KQL in Microsoft Fabric

[karthikkeyank]

Hello, I need to send (stream) log data in real time from Azure App Insights to KQL in Microsoft Fabric.
Can someone assist?

Thanks
Karthik

[v-junyant-msft · Accepted solution]

Hi @karthikkeyank ,

Direct streaming from Application Insights to Microsoft Fabric isn't natively supported, Eventstream is only supported by the following data sources:

I suggest you to use Azure Event Hubs as an intermediary. Configure an Event Hub to receive the exported data from Application Insights.
Azure Quickstart - Create an event hub using the Azure portal - Azure Event Hubs | Microsoft Learn

Best Regards,
Dino Tao
If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

[sawtoothHman]

First off, I would absolutely love this kind of capability to be built-in. If not real-time, then at least a direct, schedulable integration between the App Insights tables in Log Analytics (KQL) and the tables in a SQL analytics endpoint in a Fabric workspace Lakehouse. Esteemed Microsoft people, please make this easier!

What I came up with is not real-time data streaming and is probably very hacky/bassackwards/ill-advised, but it does get App Insights data queried in/with Log Analytics KQL into a SQL analytics endpoint table in my Lakehouse in my Fabric workspace. Take this for what you will; I'm sure someone vastly more intelligent than I has a much more elegant solution. I'm just excited to share what I've done.

1. I exported the query results from an App Insights table (requests) in Log Analytics (KQL) to a PowerBI dataset. That created a new semantic model in my Fabric workspace.
2. For some reason, I could not add the new semantic model from PowerBI to my existing Lakehouse. So in my existing Fabric workspace Lakehouse, I created a notebook to do it for me. 
3. The notebook runs some python code that reads the data from that new semantic model in my workspace. 
4. After the data from that semantic model is read into a dataframe, the notebook writes the dataframe to a table in the SQL analytics endpoint  lakehouse.
   1. EDIT: It does not appear in the SQL Analytics Endpoint as a queryable table immediately. I had to write it as a lakehouse table first, then load into a spark dataframe, then write it out to a table that was available in the SQL Analytics Endpoint view of my Lakehouse. Kinda weird, but I wanted to clarify that.

Et voila, App Insights data into Fabric. I imagine it could be automated with scheduled refreshes and pipelines but I haven't done that yet. Again, not real-time.

Python module is sempy, Copilot spat something out in two seconds for me. I found the following help doc after I did the above: Read data from semantic models and write data that semantic models can consume using python - Micros...

I did this because I've had trouble understanding how to do particular, specific things I want with the cohort workbooks and reports and whatnot in App Insights, plus I just wanted to put data from App Insights into my Fabric workspace so I can use it where I am already analyzing data I extract from CosmosDB for MongoAPI accounts through other equally crazy methods. I might be trying to wash dishes with a firehose given my choices of tooling, but it's all working for me. Your mileage may vary! Best wishes for your data endeavors.

[karthikkeyank · Accepted solution]

Hi there, @sawtoothHman 
I appreciate you providing a thorough solution and sharing your beneficial experience.

I've finally figured out how to stream data in real time from Azure Services to Microsoft Fabric. But, May have better approach too. I have tried many before this!

The telemetric data is sent from the device to the app gateway at the first level. built an internal API and methods. The EventHub keys are passed in logger service, which are available in EventStream, were utilized in this approach. Thus, EventHub is now receiving the telemetric data and using the EventStream that is kept in KQL as destinaton.

Because, The existing environment is setup is, Device --> Azure App Gateway --> Azure App Serives --> Azure SQL Server

The current setup is, Device --> Azure App Gateway --> APIM --> Azure App Serives & Microsoft Fabric --> Azure SQL Server

I hope this will help someone!


Thanks
Karthik

[karthikkeyank]

@v-junyant-msft  How to setup a exporting data in Application Insights?
I tried using Diagnostic Settings. But, It feels like a fewer amount of logs are being streamed over Azure Event Hubs.
The Diagnostic Settings are attached. I'm just bringing up the requests

 

[v-junyant-msft · Accepted solution]

Hi @karthikkeyank ,

Direct streaming from Application Insights to Microsoft Fabric isn't natively supported, Eventstream is only supported by the following data sources:

I suggest you to use Azure Event Hubs as an intermediary. Configure an Event Hub to receive the exported data from Application Insights.
Azure Quickstart - Create an event hub using the Azure portal - Azure Event Hubs | Microsoft Learn

Best Regards,
Dino Tao
If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.