Skip to main content
cancel
Showing results for 
Search instead for 
Did you mean: 

Find everything you need to get certified on Fabric—skills challenges, live sessions, exam prep, role guidance, and more. Get started

Reply
sarnendude
Frequent Visitor

High latency in Eventstream under 'Synapse Real-Time Analytics' in Microsoft Fabric

We created an Eventstream under 'synapse real-time analytics' in Microsoft Fabric to ingest real time events from Azure Event Hub to Fabric Lakehouse without any event processing by eventstream engine.
Multiple times we pushed events from Azure Event Hub using its 'Generate Data (Preview)' feature & it takes time between 30 seconds to 2 minutes to load into Fabric lakehouse.
Official documentaion says "....Eventstream with a latency of a few seconds".
In my scenario, variation of time 30 seconds to 2 minutes, so is it expected time to load or do we have any optimization steps to reduce this time? Please help.

Official documentation: https://learn.microsoft.com/en-us/fabric/real-time-analytics/overview

 

Thanks,

Sarnendu De

1 ACCEPTED SOLUTION
ajetasi
Microsoft Employee
Microsoft Employee

Currently, its a limitation that Eventstream to Lakehouse takes upto 2 minutes of latency. We are working on optimizing the latency by compacting small files that are generated in destination table. Please refer to lakehouse documentation under "Eventstreams"

 ajetasi_0-1691168381850.jpeg 

If you want a low latency (~2 seconds) destination from Eventstream, please feel free to use KQL DB destination. That is what is mentioned in the documentation you are referring to:

"You can stream large volumes of data into your KQL database through Eventstream with a latency of a few seconds, then use a KQL queryset to analyze"

 

View solution in original post

3 REPLIES 3
Tzvia
Microsoft Employee
Microsoft Employee

It sounds that If you need the lower latency from Ingestion to query, you should consider to use Synapse Real-Time Analytics at Microsoft Fabric that was built to support these scenarios 

https://learn.microsoft.com/en-us/fabric/real-time-analytics/

 

 

ajetasi
Microsoft Employee
Microsoft Employee

Currently, its a limitation that Eventstream to Lakehouse takes upto 2 minutes of latency. We are working on optimizing the latency by compacting small files that are generated in destination table. Please refer to lakehouse documentation under "Eventstreams"

 ajetasi_0-1691168381850.jpeg 

If you want a low latency (~2 seconds) destination from Eventstream, please feel free to use KQL DB destination. That is what is mentioned in the documentation you are referring to:

"You can stream large volumes of data into your KQL database through Eventstream with a latency of a few seconds, then use a KQL queryset to analyze"

 

Thank you for clarification & letting us know in details. It helps 😊

Helpful resources

Announcements
FabricCarousel_June2024

Fabric Monthly Update - June 2024

Check out the June 2024 Fabric update to learn about new features.

PBI_Carousel_NL_June

Fabric Community Update - June 2024

Get the latest Fabric updates from Build 2024, key Skills Challenge voucher deadlines, top blogs, forum posts, and product ideas.

Top Solution Authors
Top Kudoed Authors