topic Re: High latency in Eventstream under 'Synapse Real-Time Analytics' in Microsoft Fabric in Eventstream

High latency in Eventstream under 'Synapse Real-Time Analytics' in Microsoft Fabric

sarnendude — Mon, 20 May 2024 21:04:16 GMT

We created an Eventstream under 'synapse real-time analytics' in Microsoft Fabric to ingest real time events from Azure Event Hub to Fabric Lakehouse without any event processing by eventstream engine.
Multiple times we pushed events from Azure Event Hub using its 'Generate Data (Preview)' feature & it takes time between 30 seconds to 2 minutes to load into Fabric lakehouse.
Official documentaion says "....Eventstream with a latency of a few seconds".
In my scenario, variation of time 30 seconds to 2 minutes, so is it expected time to load or do we have any optimization steps to reduce this time? Please help.

Official documentation: https://learn.microsoft.com/en-us/fabric/real-time-analytics/overview

Thanks,

Sarnendu De

Re: High latency in Eventstream under 'Synapse Real-Time Analytics' in Microsoft Fabric

ajetasi — Fri, 04 Aug 2023 17:01:08 GMT

Currently, its a limitation that Eventstream to Lakehouse takes upto 2 minutes of latency. We are working on optimizing the latency by compacting small files that are generated in destination table. Please refer to lakehouse documentation under "Eventstreams"

If you want a low latency (~2 seconds) destination from Eventstream, please feel free to use KQL DB destination. That is what is mentioned in the documentation you are referring to:

"You can stream large volumes of data into your KQL database through Eventstream with a latency of a few seconds, then use a KQL queryset to analyze"

Re: High latency in Eventstream under 'Synapse Real-Time Analytics' in Microsoft Fabric

sarnendude — Sat, 05 Aug 2023 05:45:36 GMT

Thank you for clarification & letting us know in details. It helps 😊

Re: High latency in Eventstream under 'Synapse Real-Time Analytics' in Microsoft Fabric

Tzvia — Sat, 05 Aug 2023 19:41:17 GMT

It sounds that If you need the lower latency from Ingestion to query, you should consider to use Synapse Real-Time Analytics at Microsoft Fabric that was built to support these scenarios

https://learn.microsoft.com/en-us/fabric/real-time-analytics/