Options for High‑Frequency, Low‑Latency API Ingestion

Hello prathijp 

welcome to this moderated Fabric community forum.

You want to set up ingestion of messages at a ~2 minute interval where dynamic parameters (like last successful ingestion timestamp) helps you to ingest without data loss.

You have a solution running on notebooks but the capacity consumption is larger than expected.

In general, working with real-time data means having comput running day and night for ingesting, transforming and acting on data. So, smaller sized capacities (eg. lower than F4-F8) will consume most of the capacity units for these tasks.

Fabric offers a real-time solution which is generally based on the Eventstream, Eventhouse, Activator and real-time dashboard. This solution is very efficient in processing raw data via its own medallion architecture and acting on (lack of) data.

The most simple solution for you should be the newly available HTTP connector for Eventstream. 

Unfortnately, this (public preview) endpoint cannot work with dynamic parameters yet (please create a Fabric wish)!

So, there are two options:

1. If the endpoint can offer the data of the last X minutes without the need of a dynamic parameter. You can use it. This could result in duplicate data on ingestion but this is very easily fixed in the Eventhouse medallion architecture

2. If you need to use the dynamic ingestion, look at this solution where a notebook ingests the data and offers it to th Eventhouse for further processing. The Eventstream is optional, it is very easy to add an Activator testing for missing data but you could ingest into the Eventhouse directly too via its API.

Using Eventstream and Eventhouse is the way most enterprise select when working with real-time data.

If you are in control of the source API, you could also think of a 'push' solution instead of a 'pull' solution.  

I hope this answer will provide more options for you to choose from.

---

If this answer helps you, a thumbs-up or marking it as accepted answer is appreaciated. All community members with similar questions will benefit by doing so. Your contribution is highly appreciated.

Hi prathijp, 

What kind of API is this? 

The first thing that comes to mind is Real Time Intelligence, provided that you can get a data stream out of the source system. 

Spark Structured Streaming might also be helpful.  

Get started with streaming data in lakehouse - Microsoft Fabric | Microsoft Learn

Hi prathijp 

Have you had a chance to look through the responses shared earlier? If anything is still unclear, we’ll be happy to provide additional support.

The above approaches did not work since my API requires dynamic paramters like start date and end dates which can not be handled in other ways as we dont have control of API. 

If I use Eventhouse, how to get the data back in my medallion data layers?

Hi prathijp if you could elaborate what this "external API" data source is that would be helpful.

I can see that if you’re running 2‑minute (or sub‑2‑minute) via Pipelines/Notebooks will almost always burn a lot of background compute CUs and these won't be sustainable.

My recommendation is to use Fabric RTI Eventstream + Custom App/Endpoint (push into Fabric).

• Fabric Eventstream is meant to ingest/route streaming events and land them into destinations like Eventhouse (KQL Database) and/or Lakehouse.
• Microsoft provides a tutorial that streams real‑time events from a custom application into a KQL database via Eventstream, then builds a near‑real‑time Power BI report on that KQL database. 
• RTI “push” patterns include writing to an Azure Event Hub/Kafka/AMQP endpoint via Eventstream Custom Endpoint, and Eventstream also supports many CDC/streaming sources.

Hopefully this would help you:

• Replace scheduling Pipelines/Notebooks every 2 minutes (the thing causing your 70% capacity alerts), and
• Get the data visualised in RTI dashboards fast by querying the KQL Database (Eventhouse) instead of waiting for SQL endpoint refresh cycles

Appreciate if you can 'Kudos' and/or 'Accept as Solution' if this answered your query.

Hi,

For a polling interval of 2 minutes or less, I'd avoid orchestrating the workload with Fabric Pipelines or scheduled notebooks, as they introduce startup overhead and can quickly consume capacity when running continuously.

A few approaches you could consider:

• Microsoft Fabric Eventstream / Real-Time Intelligence if the source can publish events or if you can introduce an intermediary that converts API responses into a stream. This is generally the best option for low-latency ingestion.

• Azure Functions or Azure Container Apps on a timer trigger to call the external API every 2 minutes, parse the XML/JSON, and write the results directly to a Fabric Eventhouse, Lakehouse, or Warehouse. This offloads the frequent polling from your Fabric capacity.

• If the API supports webhooks or push notifications, prefer that over polling whenever possible, as it reduces both latency and infrastructure costs.

For reporting:

• If near real-time visibility is required, consider Eventhouse/KQL Database with Real-Time Intelligence dashboards instead of relying on SQL Endpoint refreshes.

• If the data must ultimately reside in your medallion architecture, you can ingest it into Eventhouse first and periodically persist it to your Lakehouse for downstream analytics.

Since you've already observed 70% capacity alerts from frequent pipeline/notebook executions, moving the polling logic outside Fabric and using Fabric primarily for ingestion and analytics is typically a more scalable and cost-effective architecture.

A couple of questions that would help refine the recommendation:

• Is the external API pull-only, or does it support webhooks/event notifications?

• Approximately how much data is returned every 2 minutes (records or MB)?

• Do your Streamlit dashboards require sub-minute freshness, or is a 2–5 minute end-to-end latency acceptable?

If you've found another approach or Microsoft has recommended a best practice for this scenario, please consider sharing it here—it would be valuable for others implementing high-frequency API ingestion in Fabric.

If this reply helps, please consider giving it a Kudos. If it answers your question, marking it as the Accepted Solution will help other community members with similar requirements.