real-time analytics

8 Topics

No Start/Stop on event stream ?
Hi, I built an eventstream with two targets. The problem is: Once I completed the configuration of the first target, the ingestion started, leaving the 2nd target behind. Should the eventstream have a start/stop button to ensure both targets will be in sync? Am I missing something, or is this a feature request? Kind Regards, Dennes
DennesTorres
1 year ago Place Eventstream
27KViews
0likes
10Comments
Ingest Data from Streaming API
I am trying to ingest data into my Lakehouse from a streaming API (https://developer-specs.company-information.service.gov.uk/streaming-api/guides/overview) I currently do this using a Function App that connects to the API and packages and stores files every 8 minutes. The Function then reconnects using the latest timestamp processed. This is the recommended method in the above link. How can I recreate this in Fabric pipelines/eventstream to ingest the streaming data into OneLake? Thanks
Solved
BW_RFA
1 year ago Place Eventstream
20KViews
0likes
6Comments
Where is Sample data coming from for using Sample data as a streaming source in Fabric eventstream
Hi, We want to mimic how the Sample data as a source in Eventstream under Real time Anlytics in Fabric is pulling the Sample data from (like: Sample db placed in some cloud platform, github, etc.) for it to allow streaming data. Basically, we want to understand what is the source for Sample data source in eventstream and how is it able to have streaming data behaviour. We want to replicate this behaviour for implementation of our Realtime analytics use case which needs a source for streaming the data and we want some other sources except Azure IOT hub and Azure Event hub. Please help with any links on this or any information. Thanks.
Solved
Anonymous
2 years ago Place Eventstream
6.5KViews
0likes
1Comment
Problems with eventstream's lakehouse transformation
Hi, When applying an aggregate to a lakehouse transformation in the eventstream, we can't change the name of the aggregated column: We can't change that "COUNT_*" Probably as a result, when we choose to create the table in the lakehouse, the table is not created and the process fails. The message should not be telling me the table was not found, because I asked the target to create it. Is this a bug or am I doing something wrong? Kind Regards, Dennes
Solved
DennesTorres
2 years ago Place Eventstream
8.6KViews
0likes
2Comments
High latency in Eventstream under 'Synapse Real-Time Analytics' in Microsoft Fabric
We created an Eventstream under 'synapse real-time analytics' in Microsoft Fabric to ingest real time events from Azure Event Hub to Fabric Lakehouse without any event processing by eventstream engine. Multiple times we pushed events from Azure Event Hub using its 'Generate Data (Preview)' feature & it takes time between 30 seconds to 2 minutes to load into Fabric lakehouse. Official documentaion says "....Eventstream with a latency of a few seconds". In my scenario, variation of time 30 seconds to 2 minutes, so is it expected time to load or do we have any optimization steps to reduce this time? Please help. Official documentation: https://learn.microsoft.com/en-us/fabric/real-time-analytics/overview Thanks, Sarnendu De
Solved
sarnendude
2 years ago Place Eventstream
12KViews
0likes
3Comments
Group by on eventstream's lakehouse target doesn't aggregate numeric fields
Hi, The problem is: The fields are not available to apply an aggregation function. I can make a count, but I can't make a sum, avg or other. This was on the NYTaxi sample data, close the the existing real time walkthrough on Microsoft portal, but I included a lakehouse target (kusto and lakehouse, two targets) and this happened on the transformation for the lakehouse. Is this a bug, or did I miss something? The image below illustrate the problem. The only thing which appears is "partitionId": Kind Regards, Dennes
Solved
DennesTorres
2 years ago Place Eventstream
23KViews
0likes
9Comments
How to stream messages from EventStream to an existing Azure EventHub
Hello all. I'm trying to stream messages from a Fabric EventStream to an existing Azure EventHub. From what I see here, only these four types are supported as EventStream destinations: custom app, KQL database, lakehouse, and reflex. From what I see, custom apps support EventHubs, in the sense that they provide three "protocols": Eventhub, AMQP and Kafka. This would enable callers to make use of the EventHub associated with the custom app. What I want to do is different. I want the messages sent to the custom app to be streamed to an existing Azure EventHub. Our friend GPT 4 suggests creating an Azure EventGrid and using that as a pub sub message distribution service between the EventHub in the custom app and the existing Azure EventHub. Not having used EventGrid before, I wanted to reach out and see if anyone has any opinions on this approach (or it there are better options / best practices). BTW, I suppose another approach is to write my own code (e.g. Azure function) to read from the custom app's EventHub and write messages to the Azure EventHub, but I'm pretty sure I'd be way better off using a more "off the shelf" solution. Will report back on my findings one way or another.
Solved
roalexan
2 years ago Place Eventstream
9.1KViews
0likes
2Comments
Flatten nested JSON in KQL DB
Hi All, Is there a way to flatten and ingest nested JSON into KQL DB using Real time analytics. Consider it as a one time load from Onelake.
Solved
Debasish_p
2 years ago Place Eventstream
28KViews
0likes
10Comments