Forum Discussion

prathijp's avatar
prathijp
Helper I
5 months ago

Options for High‑Frequency, Low‑Latency API Ingestion

I’m looking for guidance on options for invoking external APIs(with dynamic date parameters in specific format like '202602152359') at very high frequency (every 2 minutes or less) and making the dat...
Eventstream
rizalard0684's avatar
rizalard0684
Resolver III
4 months ago

Hi prathijp if you could elaborate what this "external API" data source is that would be helpful.

 

I can see that if you’re running 2‑minute (or sub‑2‑minute) via Pipelines/Notebooks will almost always burn a lot of background compute CUs and these won't be sustainable.

 

My recommendation is to use Fabric RTI Eventstream + Custom App/Endpoint (push into Fabric).

  • Fabric Eventstream is meant to ingest/route streaming events and land them into destinations like Eventhouse (KQL Database) and/or Lakehouse.
  • Microsoft provides a tutorial that streams real‑time events from a custom application into a KQL database via Eventstream, then builds a near‑real‑time Power BI report on that KQL database. 
  • RTI “push” patterns include writing to an Azure Event Hub/Kafka/AMQP endpoint via Eventstream Custom Endpoint, and Eventstream also supports many CDC/streaming sources.

Hopefully this would help you:

  • Replace scheduling Pipelines/Notebooks every 2 minutes (the thing causing your 70% capacity alerts), and
  • Get the data visualised in RTI dashboards fast by querying the KQL Database (Eventhouse) instead of waiting for SQL endpoint refresh cycles

Appreciate if you can 'Kudos' and/or 'Accept as Solution' if this answered your query.