Forum Discussion
Near real time
Hi,
I have a requirement of pulling the data from an API in near real time(5 min interval) and load it into lakehouse or warehouse? How do I do it in fabric? The entire pipeline has to complete within secs so that the pipeline can be triggered again after 5 mins. I have minimal transformations.
What components should I use to achieve this? I am currently using a copy activity and two notebooks(one to convert the json data to parquet format and the other one to load the data- can combine these if needed), it takes around 4 mins for the pipeline to complete.
Need suggestions to handle this
9 Replies
I would have said "Use Eventhouses instead" but these Kusto databases seem to have a horrible latency (8 minutes plus) - at least that's what the Capacity Metrics App exposes. So not sure if it would work in your scenario.
Hello lbendlin ,
I was reading and I find interesting your feedback. I have been doing some test from my side and the Kusto Database performs actually very correct, a couple of seconds of delay is more than enough for me. Would it be possible to share a screenshot or extra details about the latency that you mention ? Unfortunately I can't use/install the Capacity Metrics App because I am missing some rights...
Thanks and regards,
Omar C.
I believe the difference is in the configuration - batch mode vs streaming mode. The Capacity Metrics app source seems to be running in batch mode.
You can use the Notebook as a streaming notebook (with a wait command for 5 mins) and send the data to an Eventstream.
Here you can transform the data (unfold the json) quite easily and land it in a Lakehouse and/or Eventhouse to fit your needs for further processing of the data.
The latency mentioned above is only for the mirror between Eventhouse and OneLake - but you can land your data directly in a Lakehouse from the Eventstream with extremely low latency.
Hello ilbe ,Is there any limitation in EventHouse?
We are encountering the following error:"This incident has been reported. Error code: KustoWebV2; 21b26176-53a5-446b-832e-87ea443543de."Could you please advise .
Thank you
Thanks everyone, will try and get back 🙂
Hi Kar_c ,
As we haven’t heard back from you, we wanted to kindly follow up to check if the solution provided for the issue worked? or Let us know if you need any further assistance?
If our response addressed, please mark it as Accept as solution and consider giving a KUDOS. Feel free to reach out if you need further assistance.Regards,
Dinesh
Hi Kar_c ,
As we haven’t heard back from you, we wanted to kindly follow up to check if the solution provided for the issue worked? or Let us know if you need any further assistance?
If our response addressed, please mark it as Accept as solution and consider giving a KUDOS. Feel free to reach out if you need further assistance.Regards,
Dinesh
