Microsoft Fabric Community Conference 2025, March 31 - April 2, Las Vegas, Nevada. Use code MSCUST for a $150 discount.
Register nowGet certified as a Fabric Data Engineer: Check your eligibility for a 50% exam voucher offer and join us for free live learning sessions to get prepared for Exam DP-700. Get started
We created an Eventstream under 'synapse real-time analytics' in Microsoft Fabric to ingest real time events from Azure Event Hub to Fabric Lakehouse without any event processing by eventstream engine.
Multiple times we pushed events from Azure Event Hub using its 'Generate Data (Preview)' feature & it takes time between 30 seconds to 2 minutes to load into Fabric lakehouse.
Official documentaion says "....Eventstream with a latency of a few seconds".
In my scenario, variation of time 30 seconds to 2 minutes, so is it expected time to load or do we have any optimization steps to reduce this time? Please help.
Official documentation: https://learn.microsoft.com/en-us/fabric/real-time-analytics/overview
Thanks,
Sarnendu De
Solved! Go to Solution.
Currently, its a limitation that Eventstream to Lakehouse takes upto 2 minutes of latency. We are working on optimizing the latency by compacting small files that are generated in destination table. Please refer to lakehouse documentation under "Eventstreams"
If you want a low latency (~2 seconds) destination from Eventstream, please feel free to use KQL DB destination. That is what is mentioned in the documentation you are referring to:
"You can stream large volumes of data into your KQL database through Eventstream with a latency of a few seconds, then use a KQL queryset to analyze"
It sounds that If you need the lower latency from Ingestion to query, you should consider to use Synapse Real-Time Analytics at Microsoft Fabric that was built to support these scenarios
https://learn.microsoft.com/en-us/fabric/real-time-analytics/
Currently, its a limitation that Eventstream to Lakehouse takes upto 2 minutes of latency. We are working on optimizing the latency by compacting small files that are generated in destination table. Please refer to lakehouse documentation under "Eventstreams"
If you want a low latency (~2 seconds) destination from Eventstream, please feel free to use KQL DB destination. That is what is mentioned in the documentation you are referring to:
"You can stream large volumes of data into your KQL database through Eventstream with a latency of a few seconds, then use a KQL queryset to analyze"
Thank you for clarification & letting us know in details. It helps 😊
March 31 - April 2, 2025, in Las Vegas, Nevada. Use code MSCUST for a $150 discount!
If you love stickers, then you will definitely want to check out our Community Sticker Challenge!