topic Re: Near real time in Eventstream

Near real time

Kar_c — Fri, 28 Feb 2025 21:05:57 GMT

Hi,

I have a requirement of pulling the data from an API in near real time(5 min interval) and load it into lakehouse or warehouse? How do I do it in fabric? The entire pipeline has to complete within secs so that the pipeline can be triggered again after 5 mins. I have minimal transformations.

What components should I use to achieve this? I am currently using a copy activity and two notebooks(one to convert the json data to parquet format and the other one to load the data- can combine these if needed), it takes around 4 mins for the pipeline to complete.

Need suggestions to handle this

Re: Near real time

lbendlin — Fri, 28 Feb 2025 23:37:25 GMT

I would have said "Use Eventhouses instead" but these Kusto databases seem to have a horrible latency (8 minutes plus) - at least that's what the Capacity Metrics App exposes. So not sure if it would work in your scenario.

Re: Near real time

PrachiJain_2025 — Sat, 01 Mar 2025 10:35:14 GMT

Hello ilbe ,Is there any limitation in EventHouse?

We are encountering the following error:

"This incident has been reported. Error code: KustoWebV2; 21b26176-53a5-446b-832e-87ea443543de."

Could you please advise .

Thank you

Re: Near real time

datacoffee — Sat, 01 Mar 2025 10:55:29 GMT

You can use the Notebook as a streaming notebook (with a wait command for 5 mins) and send the data to an Eventstream.

Here you can transform the data (unfold the json) quite easily and land it in a Lakehouse and/or Eventhouse to fit your needs for further processing of the data.

The latency mentioned above is only for the mirror between Eventhouse and OneLake - but you can land your data directly in a Lakehouse from the Eventstream with extremely low latency.

Re: Near real time

Kar_c — Thu, 06 Mar 2025 10:03:10 GMT

Thanks everyone, will try and get back 🙂

Re: Near real time

omarecd — Sun, 09 Mar 2025 21:18:16 GMT

Hello lbendlin ,

I was reading and I find interesting your feedback. I have been doing some test from my side and the Kusto Database performs actually very correct, a couple of seconds of delay is more than enough for me. Would it be possible to share a screenshot or extra details about the latency that you mention ? Unfortunately I can't use/install the Capacity Metrics App because I am missing some rights...

Thanks and regards,

Omar C.

Re: Near real time

lbendlin — Sun, 09 Mar 2025 21:25:41 GMT

I believe the difference is in the configuration - batch mode vs streaming mode. The Capacity Metrics app source seems to be running in batch mode.

Re: Near real time

v-dineshya — Mon, 28 Apr 2025 11:54:38 GMT

Hi Kar_c ,

As we haven’t heard back from you, we wanted to kindly follow up to check if the solution provided for the issue worked? or Let us know if you need any further assistance?
If our response addressed, please mark it as Accept as solution and consider giving a KUDOS. Feel free to reach out if you need further assistance.

Regards,

Dinesh

Re: Near real time

v-dineshya — Fri, 02 May 2025 17:59:08 GMT

Hi Kar_c ,

Regards,

Dinesh

Re: Near real time

v-dineshya — Tue, 06 May 2025 05:33:57 GMT

Hi @Kar_c ,

Regards,

Dinesh