Options for High‑Frequency, Low‑Latency API Ingestion

Hello prathijp 

welcome to this moderated Fabric community forum.

You want to set up ingestion of messages at a ~2 minute interval where dynamic parameters (like last successful ingestion timestamp) helps you to ingest without data loss.

You have a solution running on notebooks but the capacity consumption is larger than expected.

In general, working with real-time data means having comput running day and night for ingesting, transforming and acting on data. So, smaller sized capacities (eg. lower than F4-F8) will consume most of the capacity units for these tasks.

Fabric offers a real-time solution which is generally based on the Eventstream, Eventhouse, Activator and real-time dashboard. This solution is very efficient in processing raw data via its own medallion architecture and acting on (lack of) data.

The most simple solution for you should be the newly available HTTP connector for Eventstream. 

Unfortnately, this (public preview) endpoint cannot work with dynamic parameters yet (please create a Fabric wish)!

So, there are two options:

1. If the endpoint can offer the data of the last X minutes without the need of a dynamic parameter. You can use it. This could result in duplicate data on ingestion but this is very easily fixed in the Eventhouse medallion architecture

2. If you need to use the dynamic ingestion, look at this solution where a notebook ingests the data and offers it to th Eventhouse for further processing. The Eventstream is optional, it is very easy to add an Activator testing for missing data but you could ingest into the Eventhouse directly too via its API.

Using Eventstream and Eventhouse is the way most enterprise select when working with real-time data.

If you are in control of the source API, you could also think of a 'push' solution instead of a 'pull' solution.  

I hope this answer will provide more options for you to choose from.

---

If this answer helps you, a thumbs-up or marking it as accepted answer is appreaciated. All community members with similar questions will benefit by doing so. Your contribution is highly appreciated.