Near real time

[Kar_c]

Hi,

I have a requirement of pulling the data from an API in near real time(5 min interval) and load it into lakehouse or warehouse? How do I do it in fabric? The entire pipeline has to complete within secs so that the pipeline can be triggered again after 5 mins. I have minimal transformations. 

What components should I use to achieve this? I am currently using a copy activity and two notebooks(one to convert the json data to parquet format and the other one to load the data- can combine these if needed), it takes around 4 mins for the pipeline to complete. 

Need suggestions to handle this 

[Kar_c]

Thanks everyone, will try and get back 🙂

[datacoffee]

You can use the Notebook as a streaming notebook (with a wait command for 5 mins) and send the data to an Eventstream.

Here you can transform the data (unfold the json) quite easily and land it in a Lakehouse and/or Eventhouse to fit your needs for further processing of the data.

The latency mentioned above is only for the mirror between Eventhouse and OneLake - but you can land your data directly in a Lakehouse from the Eventstream with extremely low latency.

[PrachiJain_2025]

Hello @ilbe ,Is there any limitation in EventHouse? 

We are encountering the following error:

"This incident has been reported. Error code: KustoWebV2; 21b26176-53a5-446b-832e-87ea443543de."

Could you please advise .

Thank you

[lbendlin]

I would have said "Use Eventhouses instead"  but these Kusto databases seem to have a horrible latency (8 minutes plus) - at least that's what the Capacity Metrics App exposes. So not sure if it would work in your scenario.

[omarecd]

Hello @lbendlin ,

I was reading and I find interesting your feedback. I have been doing some test from my side and the Kusto Database performs actually very correct, a couple of seconds of delay is more than enough for me. Would it be possible to share a screenshot or extra details about the latency that you mention ? Unfortunately I can't use/install the Capacity Metrics App because I am missing some rights...

Thanks and regards,

Omar C.

[lbendlin]

I believe the difference is in the configuration - batch mode vs streaming mode.  The Capacity Metrics app source seems to be running in batch mode.