The ultimate Microsoft Fabric, Power BI, Azure AI, and SQL learning event: Join us in Stockholm, September 24-27, 2024.
Save €200 with code MSCUST on top of early bird pricing!
Find everything you need to get certified on Fabric—skills challenges, live sessions, exam prep, role guidance, and more. Get started
I am trying to ingest data into my Lakehouse from a streaming API (https://developer-specs.company-information.service.gov.uk/streaming-api/guides/overview)
I currently do this using a Function App that connects to the API and packages and stores files every 8 minutes. The Function then reconnects using the latest timestamp processed. This is the recommended method in the above link.
How can I recreate this in Fabric pipelines/eventstream to ingest the streaming data into OneLake?
Thanks
Solved! Go to Solution.
Hi @BW_RFA ,
You can use a python script to regularly poll the API and send it to Eventstreams which can then send to Lakehouse or KQL DB or even easier way if velocity is not a problem is to setup a Logic App which reads from the API and send to Eventstreams.
Hi @BW_RFA ,
Thanks for using Fabric Community.
Did you got a chance to look into this Real Time Analytics Fabric?
Microsoft Fabric event streams overview - Microsoft Fabric | Microsoft Learn
Hi,
I have but it doesn't explain how I can connect to a streaming API. Should I do this through a custom app, eventHubs or IoT Hub?
I don't think it's a good use case for IoT Hub, and EventHubs needs data to be pushed to it instead of pulling the data.
How would writing a custom app for this scenario work? Would it be a case of using the same/similar code to the one in the current ingestion Function App I have?
Hi @BW_RFA ,
You can use a python script to regularly poll the API and send it to Eventstreams which can then send to Lakehouse or KQL DB or even easier way if velocity is not a problem is to setup a Logic App which reads from the API and send to Eventstreams.
Hello @BW_RFA ,
We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet .
In case if you have any resolution please do share that same with the community as it can be helpful to others .
Otherwise, will respond back with the more details and we will try to help .
Hi,
I am building a custom app to ingest this data. Thank you for your help
Join the community in Stockholm for expert Microsoft Fabric learning including a very exciting keynote from Arun Ulag, Corporate Vice President, Azure Data.
Get the latest Fabric updates from Build 2024, key Skills Challenge voucher deadlines, top blogs, forum posts, and product ideas.