Skip to main content
cancel
Showing results for 
Search instead for 
Did you mean: 

Find everything you need to get certified on Fabric—skills challenges, live sessions, exam prep, role guidance, and more. Get started

Reply
BW_RFA
Helper I
Helper I

Ingest Data from Streaming API

I am trying to ingest data into my Lakehouse from a streaming API (https://developer-specs.company-information.service.gov.uk/streaming-api/guides/overview)

 

I currently do this using a Function App that connects to the API and packages and stores files every 8 minutes. The Function then reconnects using the latest timestamp processed. This is the recommended method in the above link.

 

How can I recreate this in Fabric pipelines/eventstream to ingest the streaming data into OneLake?

 

Thanks

1 ACCEPTED SOLUTION

Hi @BW_RFA ,

You can use a python script to regularly poll the API and send it to Eventstreams which can then send to Lakehouse or KQL DB or even easier way if velocity is not a problem is to setup a Logic App which reads from the API and send to Eventstreams.



View solution in original post

5 REPLIES 5
v-gchenna-msft
Community Support
Community Support

Hi @BW_RFA ,

Thanks for using Fabric Community.
Did you got a chance to look into this Real Time Analytics Fabric?

Microsoft Fabric event streams overview - Microsoft Fabric | Microsoft Learn

Hi,

 

I have but it doesn't explain how I can connect to a streaming API. Should I do this through a custom app, eventHubs or IoT Hub?

 

I don't think it's a good use case for IoT Hub, and EventHubs needs data to be pushed to it instead of pulling the data.

 

How would writing a custom app for this scenario work? Would it be a case of using the same/similar code to the one in the current ingestion Function App I have? 

Hi @BW_RFA ,

You can use a python script to regularly poll the API and send it to Eventstreams which can then send to Lakehouse or KQL DB or even easier way if velocity is not a problem is to setup a Logic App which reads from the API and send to Eventstreams.



Hello @BW_RFA ,

We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet .
In case if you have any resolution please do share that same with the community as it can be helpful to others .
Otherwise, will respond back with the more details and we will try to help .

Hi,

 

I am building a custom app to ingest this data. Thank you for your help

Helpful resources

Announcements
FabricCarousel_June2024

Fabric Monthly Update - June 2024

Check out the June 2024 Fabric update to learn about new features.

PBI_Carousel_NL_June

Fabric Community Update - June 2024

Get the latest Fabric updates from Build 2024, key Skills Challenge voucher deadlines, top blogs, forum posts, and product ideas.

Top Solution Authors
Top Kudoed Authors