Skip to main content
cancel
Showing results for 
Search instead for 
Did you mean: 

Get certified in Microsoft Fabric—for free! For a limited time, get a free DP-600 exam voucher to use by the end of 2024. Register now

Reply
slimbnsalah
Regular Visitor

Exporting Data from AppInsights to Fabric Lakehouse

Hello,

 

Im working on Transferring data from AppInsights to a Lakehouse in a Fabric Capacity on daily bases.

I want to know what is the best way to do it.

 

The solution I can think of is:

Consume the AppInsights API in a DataPipeline or DataFlow in Fabric that exports to a lakehouse.

 

 

My Problem is that data is so huge that it exceeds the limits of what the API can handle, and I want to avoid backing up the data on many times of the day. I want to do it only once a day.

 

I would appreciate if you can help me with this.

8 REPLIES 8
Jerome22
Advocate II
Advocate II

I know that the team is working on a direct connection to the appinsights logs from wihtin Fabric (like we can connect snowflake now)

so "soon" we will be able to consume these logs without the need of copying them.

certainly something like a KDL database available.

 

but for now a dataflow is required 

 

Anonymous
Not applicable

Hi @slimbnsalah ,

Thanks for using Fabric Community.
At this time, we are reaching out to the internal team to get some help on this .
We will update you once we hear back from them.

Anonymous
Not applicable

Hi @slimbnsalah ,

We have an update from internal team -

"My Problem is that data is so huge that it exceeds the limits of what the API can handle" - they would need to work with their data source provider (AppInsights) if they are exceeding their API limits at their tier and/or look to see if there is ways to to control the timeframe / supply parameters within their request to stay within the APIs limits.


Hope this is helpful. Please let me know incase of further queries.

Anonymous
Not applicable

HI @slimbnsalah ,

You can also use pipelines rest connector with pagination settings to iterate over all the data returned by AppInsights through APIs, and use Lakehouse as destination. You shouldnt run into any API limits in this case. Let me know incase if you have any further queries.

use pipelines rest connector with pagination settings

Rest connector is not yet supported in Dataflow Gen2, Source: https://learn.microsoft.com/en-us/fabric/data-factory/connector-rest-overview

Even with that, how will the API handle data more than it can return?

Anonymous
Not applicable

Hi @slimbnsalah ,

You should be able to use rest connector activity in data factory pipelines. With the help of pagination settings to iterate over all the data returned by AppInsights through APIs, and use Lakehouse as destination. You shouldnt run into any API limits in this case. 

vgchennamsft_0-1714031142982.png

 

Links to refer - 
Pagination Rules in Rest Connector 

I hope this is helpful. 

Anonymous
Not applicable

Hi @slimbnsalah ,

We haven’t heard from you on the last response and was just checking back to see if you got some insights.
Otherwise, will respond back with the more details and we will try to help .

Thanks

Anonymous
Not applicable

Hi @slimbnsalah ,

We haven’t heard from you on the last response and was just checking back to see if you got some insights.
Otherwise, will respond back with the more details and we will try to help .

Thanks

Helpful resources

Announcements
November Carousel

Fabric Community Update - November 2024

Find out what's new and trending in the Fabric Community.

Live Sessions with Fabric DB

Be one of the first to start using Fabric Databases

Starting December 3, join live sessions with database experts and the Fabric product team to learn just how easy it is to get started.

November Update

Fabric Monthly Update - November 2024

Check out the November 2024 Fabric update to learn about new features.

Las Vegas 2025

Join us at the Microsoft Fabric Community Conference

March 31 - April 2, 2025, in Las Vegas, Nevada. Use code MSCUST for a $150 discount! Early Bird pricing ends December 9th.