Exporting Data from AppInsights to Fabric Lakehouse

[slimbnsalah]

Hello,

Im working on Transferring data from AppInsights to a Lakehouse in a Fabric Capacity on daily bases.

I want to know what is the best way to do it.

The solution I can think of is:

Consume the AppInsights API in a DataPipeline or DataFlow in Fabric that exports to a lakehouse.

My Problem is that data is so huge that it exceeds the limits of what the API can handle, and I want to avoid backing up the data on many times of the day. I want to do it only once a day.

I would appreciate if you can help me with this.

[Jerome22]

I know that the team is working on a direct connection to the appinsights logs from wihtin Fabric (like we can connect snowflake now)

so "soon" we will be able to consume these logs without the need of copying them.

certainly something like a KDL database available.

but for now a dataflow is required 

[v-gchenna-msft]

Hi @slimbnsalah ,

Thanks for using Fabric Community.
At this time, we are reaching out to the internal team to get some help on this .
We will update you once we hear back from them.

[v-gchenna-msft]

Hi @slimbnsalah ,

We have an update from internal team -

"My Problem is that data is so huge that it exceeds the limits of what the API can handle" - they would need to work with their data source provider (AppInsights) if they are exceeding their API limits at their tier and/or look to see if there is ways to to control the timeframe / supply parameters within their request to stay within the APIs limits.

---

Hope this is helpful. Please let me know incase of further queries.

[v-gchenna-msft]

HI @slimbnsalah ,

You can also use pipelines rest connector with pagination settings to iterate over all the data returned by AppInsights through APIs, and use Lakehouse as destination. You shouldnt run into any API limits in this case. Let me know incase if you have any further queries.

[slimbnsalah]

use pipelines rest connector with pagination settings

---

Rest connector is not yet supported in Dataflow Gen2, Source: https://learn.microsoft.com/en-us/fabric/data-factory/connector-rest-overview

Even with that, how will the API handle data more than it can return?

[v-gchenna-msft]

Hi @slimbnsalah ,

You should be able to use rest connector activity in data factory pipelines. With the help of pagination settings to iterate over all the data returned by AppInsights through APIs, and use Lakehouse as destination. You shouldnt run into any API limits in this case. 

Links to refer - 
Pagination Rules in Rest Connector 

I hope this is helpful. 

[v-gchenna-msft]

Hi @slimbnsalah ,

We haven’t heard from you on the last response and was just checking back to see if you got some insights.
Otherwise, will respond back with the more details and we will try to help .

Thanks

[v-gchenna-msft]

Hi @slimbnsalah ,

We haven’t heard from you on the last response and was just checking back to see if you got some insights.
Otherwise, will respond back with the more details and we will try to help .

Thanks