New records inserted to KQL db using Fabric eventstream taking time to reflect records in OneLake

[supriyapandey19]

Hi,

I am using Fabric eventstream to load the streaming data to KQL db and have activated the OneLake availability on the KQL db. The new records inserted in KQL db through eventstream are taking time to reflect the new insertion in OneLake, it is not instantaneous. I have created a Direct Lake semantic model on top of OneLake to create Power BI reports and hence, the power bi reports are taking time to show the new records inserted through evenstream even though Direct Lake mode is expected to be instant.

Any solution, articles, references will be appreciated.

Thanks.

[Anonymous]

Hi @supriyapandey19 ,

While Direct Lake mode in Power BI provides a near real-time analysis capability by directly loading parquet-formatted files from OneLake, there can be inherent delays in the data pipeline, especially when dealing with streaming data. The process involves capturing the streaming data, transforming it if necessary, loading it into the KQL database, and then making it available in OneLake for Direct Lake consumption. Each of these steps, while optimized for performance, can introduce slight delays.

But this delay is usually not too long, and depending on the number and complexity of the conversion steps you set up, it's generally a few minutes:

Introducing Real-Time Intelligence in Microsoft Fabric | Microsoft Fabric Blog | Microsoft Fabric

What is the approximate length of the delay you are facing?

Best Regards,
Dino Tao
If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

[supriyapandey19]

Hi,

The delay is happening when the data is loaded to KQL db, but to reflect that data in OneLake it is taking time. The delay is sometimes even around an hour.

What is causing this delay and how it can be optimized?

Thanks.

[Anonymous]

Hi @supriyapandey19 ,

There are many reasons for the delay, but to find out the exact cause you may need to go to the backend and grab the data to know.

If you wish, you can open a support ticket here:
Microsoft Fabric Support and Status | Microsoft Fabric

A dedicated Microsoft engineer will come to solve the problem for you.For how to create a support ticket, please refer to https://community.powerbi.com/t5/Community-Blog/How-to-create-a-support-ticket-in-Power-BI/ba-p/6830...

It would be great if you continue to share in this issue to help others with similar problems after you know the root cause or solution.

Best Regards,
Dino Tao
If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

[supriyapandey19]

Hi,

In our case, we are using KQL db as the destination to store streaming data coming from a csv file of around 100 Kb using Eventstream, and once the data is loaded into KQL db we want to utilize this data fron OneLake to create direct lake semantic model and that's why we have activated the OneLake availability option on our KQL db. 

But, after loading the data into KQL db it is sometimes taking upto few hours to show up that data in OneLake and we are not sure if this is expected behaviour or not and in this case what should we do as we need the data fron OneLake in our reports in real time.

Thanks.