topic Re: Exporting Data from AppInsights to Fabric Lakehouse in Pipelines

Exporting Data from AppInsights to Fabric Lakehouse

slimbnsalah — Thu, 18 Apr 2024 16:13:45 GMT

Hello,

Im working on Transferring data from AppInsights to a Lakehouse in a Fabric Capacity on daily bases.

I want to know what is the best way to do it.

The solution I can think of is:

Consume the AppInsights API in a DataPipeline or DataFlow in Fabric that exports to a lakehouse.

My Problem is that data is so huge that it exceeds the limits of what the API can handle, and I want to avoid backing up the data on many times of the day. I want to do it only once a day.

I would appreciate if you can help me with this.

Re: Exporting Data from AppInsights to Fabric Lakehouse

Anonymous — Fri, 19 Apr 2024 10:49:03 GMT

Hi slimbnsalah ,

Thanks for using Fabric Community.
At this time, we are reaching out to the internal team to get some help on this .
We will update you once we hear back from them.

Re: Exporting Data from AppInsights to Fabric Lakehouse

Anonymous — Mon, 22 Apr 2024 18:37:34 GMT

Hi slimbnsalah ,

We have an update from internal team -

"My Problem is that data is so huge that it exceeds the limits of what the API can handle" - they would need to work with their data source provider (AppInsights) if they are exceeding their API limits at their tier and/or look to see if there is ways to to control the timeframe / supply parameters within their request to stay within the APIs limits.

Hope this is helpful. Please let me know incase of further queries.

Re: Exporting Data from AppInsights to Fabric Lakehouse

Anonymous — Tue, 23 Apr 2024 10:35:27 GMT

HI slimbnsalah ,

You can also use pipelines rest connector with pagination settings to iterate over all the data returned by AppInsights through APIs, and use Lakehouse as destination. You shouldnt run into any API limits in this case. Let me know incase if you have any further queries.

Re: Exporting Data from AppInsights to Fabric Lakehouse

slimbnsalah — Tue, 23 Apr 2024 10:44:30 GMT

use pipelines rest connector with pagination settings

Rest connector is not yet supported in Dataflow Gen2, Source: https://learn.microsoft.com/en-us/fabric/data-factory/connector-rest-overview

Even with that, how will the API handle data more than it can return?

Re: Exporting Data from AppInsights to Fabric Lakehouse

Anonymous — Thu, 25 Apr 2024 07:45:57 GMT

Hi slimbnsalah ,

You should be able to use rest connector activity in data factory pipelines. With the help of pagination settings to iterate over all the data returned by AppInsights through APIs, and use Lakehouse as destination. You shouldnt run into any API limits in this case.

Links to refer -
Pagination Rules in Rest Connector

I hope this is helpful.

Re: Exporting Data from AppInsights to Fabric Lakehouse

Jerome22 — Thu, 25 Apr 2024 10:33:10 GMT

I know that the team is working on a direct connection to the appinsights logs from wihtin Fabric (like we can connect snowflake now)

so "soon" we will be able to consume these logs without the need of copying them.

certainly something like a KDL database available.

but for now a dataflow is required

Re: Exporting Data from AppInsights to Fabric Lakehouse

Anonymous — Fri, 26 Apr 2024 10:24:49 GMT

Hi slimbnsalah ,

We haven’t heard from you on the last response and was just checking back to see if you got some insights.
Otherwise, will respond back with the more details and we will try to help .

Thanks

Re: Exporting Data from AppInsights to Fabric Lakehouse

Anonymous — Mon, 29 Apr 2024 11:44:04 GMT