Forum Discussion
High latency in Eventstream under 'Synapse Real-Time Analytics' in Microsoft Fabric
We created an Eventstream under 'synapse real-time analytics' in Microsoft Fabric to ingest real time events from Azure Event Hub to Fabric Lakehouse without any event processing by eventstream engine.
Multiple times we pushed events from Azure Event Hub using its 'Generate Data (Preview)' feature & it takes time between 30 seconds to 2 minutes to load into Fabric lakehouse.
Official documentaion says "....Eventstream with a latency of a few seconds".
In my scenario, variation of time 30 seconds to 2 minutes, so is it expected time to load or do we have any optimization steps to reduce this time? Please help.
Official documentation: https://learn.microsoft.com/en-us/fabric/real-time-analytics/overview
Thanks,
Sarnendu De
Currently, its a limitation that Eventstream to Lakehouse takes upto 2 minutes of latency. We are working on optimizing the latency by compacting small files that are generated in destination table. Please refer to lakehouse documentation under "Eventstreams"
If you want a low latency (~2 seconds) destination from Eventstream, please feel free to use KQL DB destination. That is what is mentioned in the documentation you are referring to:
"You can stream large volumes of data into your KQL database through Eventstream with a latency of a few seconds, then use a KQL queryset to analyze"
3 Replies
Currently, its a limitation that Eventstream to Lakehouse takes upto 2 minutes of latency. We are working on optimizing the latency by compacting small files that are generated in destination table. Please refer to lakehouse documentation under "Eventstreams"
If you want a low latency (~2 seconds) destination from Eventstream, please feel free to use KQL DB destination. That is what is mentioned in the documentation you are referring to:
"You can stream large volumes of data into your KQL database through Eventstream with a latency of a few seconds, then use a KQL queryset to analyze"
Thank you for clarification & letting us know in details. It helps 😊
It sounds that If you need the lower latency from Ingestion to query, you should consider to use Synapse Real-Time Analytics at Microsoft Fabric that was built to support these scenarios
https://learn.microsoft.com/en-us/fabric/real-time-analytics/
