topic Re: Ingest Data from Streaming API in Eventstream

Ingest Data from Streaming API

BW_RFA — Mon, 20 May 2024 20:59:22 GMT

I am trying to ingest data into my Lakehouse from a streaming API (https://developer-specs.company-information.service.gov.uk/streaming-api/guides/overview)

I currently do this using a Function App that connects to the API and packages and stores files every 8 minutes. The Function then reconnects using the latest timestamp processed. This is the recommended method in the above link.

How can I recreate this in Fabric pipelines/eventstream to ingest the streaming data into OneLake?

Thanks

Re: Ingest Data from Streaming API

Anonymous — Thu, 01 Feb 2024 10:36:04 GMT

Hi BW_RFA ,

Thanks for using Fabric Community.
Did you got a chance to look into this Real Time Analytics Fabric?

Microsoft Fabric event streams overview - Microsoft Fabric | Microsoft Learn

Re: Ingest Data from Streaming API

BW_RFA — Thu, 01 Feb 2024 10:52:55 GMT

Hi,

I have but it doesn't explain how I can connect to a streaming API. Should I do this through a custom app, eventHubs or IoT Hub?

I don't think it's a good use case for IoT Hub, and EventHubs needs data to be pushed to it instead of pulling the data.

How would writing a custom app for this scenario work? Would it be a case of using the same/similar code to the one in the current ingestion Function App I have?

Re: Ingest Data from Streaming API

Anonymous — Thu, 01 Feb 2024 11:31:29 GMT

Hi BW_RFA ,

You can use a python script to regularly poll the API and send it to Eventstreams which can then send to Lakehouse or KQL DB or even easier way if velocity is not a problem is to setup a Logic App which reads from the API and send to Eventstreams.

Re: Ingest Data from Streaming API

Anonymous — Fri, 02 Feb 2024 10:54:40 GMT

Hello BW_RFA ,

We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet .
In case if you have any resolution please do share that same with the community as it can be helpful to others .
Otherwise, will respond back with the more details and we will try to help .

Re: Ingest Data from Streaming API

BW_RFA — Fri, 02 Feb 2024 15:46:10 GMT

Hi,

I am building a custom app to ingest this data. Thank you for your help

Re: Ingest Data from Streaming API

jerrycaleb — Fri, 28 Feb 2025 14:44:37 GMT

How did you end up doing it?