Ingest Data from Streaming API

[BW_RFA]

I am trying to ingest data into my Lakehouse from a streaming API (https://developer-specs.company-information.service.gov.uk/streaming-api/guides/overview)

I currently do this using a Function App that connects to the API and packages and stores files every 8 minutes. The Function then reconnects using the latest timestamp processed. This is the recommended method in the above link.

How can I recreate this in Fabric pipelines/eventstream to ingest the streaming data into OneLake?

Thanks

[Anonymous]

Hi @BW_RFA ,

Thanks for using Fabric Community.
Did you got a chance to look into this Real Time Analytics Fabric?

Microsoft Fabric event streams overview - Microsoft Fabric | Microsoft Learn

[BW_RFA]

Hi,

I have but it doesn't explain how I can connect to a streaming API. Should I do this through a custom app, eventHubs or IoT Hub?

I don't think it's a good use case for IoT Hub, and EventHubs needs data to be pushed to it instead of pulling the data.

How would writing a custom app for this scenario work? Would it be a case of using the same/similar code to the one in the current ingestion Function App I have? 

[Anonymous · Accepted solution]

Hi @BW_RFA ,

You can use a python script to regularly poll the API and send it to Eventstreams which can then send to Lakehouse or KQL DB or even easier way if velocity is not a problem is to setup a Logic App which reads from the API and send to Eventstreams.

[Anonymous]

Hello @BW_RFA ,

We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet .
In case if you have any resolution please do share that same with the community as it can be helpful to others .
Otherwise, will respond back with the more details and we will try to help .

[BW_RFA]

Hi,

I am building a custom app to ingest this data. Thank you for your help

[jerrycaleb]

How did you end up doing it?