Forum Discussion
Ingest Data from Streaming API
I am trying to ingest data into my Lakehouse from a streaming API (https://developer-specs.company-information.service.gov.uk/streaming-api/guides/overview)
I currently do this using a Function App that connects to the API and packages and stores files every 8 minutes. The Function then reconnects using the latest timestamp processed. This is the recommended method in the above link.
How can I recreate this in Fabric pipelines/eventstream to ingest the streaming data into OneLake?
Thanks
Hi BW_RFA ,
You can use a python script to regularly poll the API and send it to Eventstreams which can then send to Lakehouse or KQL DB or even easier way if velocity is not a problem is to setup a Logic App which reads from the API and send to Eventstreams.
6 Replies
Hi BW_RFA ,
Thanks for using Fabric Community.
Did you got a chance to look into this Real Time Analytics Fabric?
Microsoft Fabric event streams overview - Microsoft Fabric | Microsoft LearnHi,
I have but it doesn't explain how I can connect to a streaming API. Should I do this through a custom app, eventHubs or IoT Hub?
I don't think it's a good use case for IoT Hub, and EventHubs needs data to be pushed to it instead of pulling the data.
How would writing a custom app for this scenario work? Would it be a case of using the same/similar code to the one in the current ingestion Function App I have?
Hi BW_RFA ,
You can use a python script to regularly poll the API and send it to Eventstreams which can then send to Lakehouse or KQL DB or even easier way if velocity is not a problem is to setup a Logic App which reads from the API and send to Eventstreams.
