data engineering
4 TopicsIngest Data from Streaming API
I am trying to ingest data into my Lakehouse from a streaming API (https://developer-specs.company-information.service.gov.uk/streaming-api/guides/overview) I currently do this using a Function App that connects to the API and packages and stores files every 8 minutes. The Function then reconnects using the latest timestamp processed. This is the recommended method in the above link. How can I recreate this in Fabric pipelines/eventstream to ingest the streaming data into OneLake? ThanksSolved20KViews0likes6CommentsEvent Stream unable to load the data into Lakehouse\ KQL
Hi, Looking some advice and help. I have a scenario. IOT devices messages and events data pointed to event hub (having 3-4 different message types). Without using the eventprocessing able to load the data into either lakehouse\ KQL by using event stream. But partial data loaded. When tried to use the eventprocessing with Manage Fields and Filter operations then the data is loading into both lakeshouse and KQL. Created the tables seperately and also tried to load but no luck. I am using filter operation as " column1 equals abc". Appreciate any quick help..8.6KViews0likes2CommentsHow to create event stream, reflex and connect it to pipeline.
Hi All, We have pipeline which reads multiple zip files and then execute these multiple zip files parallely to generate output. Right now this pipeline is triggerd manually after placing multiple files in landing folder. We want to automate this process i.e. the moment the files are placed in landing folder the pipeline should be triggered automatically. To achieve this result, we have created event stream and reflex. But the thing is when the file is placed in landing folder, we can see this file in event stream as well as in reflex but the pipeline is not triggered. Can somebody guide us in how to achieve this automatic triggering of pipeline using event stream and reflex in Fabric and what could be the reason that we are not able to achieve the result. Thanks, DeepmalaSolved8.5KViews0likes1CommentGroup by on eventstream's lakehouse target doesn't aggregate numeric fields
Hi, The problem is: The fields are not available to apply an aggregation function. I can make a count, but I can't make a sum, avg or other. This was on the NYTaxi sample data, close the the existing real time walkthrough on Microsoft portal, but I included a lakehouse target (kusto and lakehouse, two targets) and this happened on the transformation for the lakehouse. Is this a bug, or did I miss something? The image below illustrate the problem. The only thing which appears is "partitionId": Kind Regards, DennesSolved23KViews0likes9Comments