eventstream
150 TopicsEventstream connection with Event Hub
Hello, I have an existing event hub which I'm planning to use with an eventstream in Fabric. I see two connection options: 1. SAS keys 2. Organisational account I'm in a microsoft internal team and we're not allowed to use SAS keys. When I try to use my own account via organisational account option I'm getting following error Sign in failed with the following error code: "invalid_request". Error description given: "AADSTS65002: Consent between first party application 'b52893c8-bc2e-47fc-918b-77022b299bbc' and first party resource '80369ed6-5f11-4dd9-bef3-692475845e77' must be configured via preauthorization - applications owned and operated by Microsoft must get approval from the API owner before requesting tokens for that API. Trace ID: 3e344c2e-4ab0-442d-af25-23f3e1c15500 Correlation ID: 18583d2a-007b-4b7a-a7d0-5485087e55c7 Timestamp: 2025-07-29 16:05:32Z" (Session ID: f67ac96c-b700-4aed-8514-2b9acd5b00c8, Region: usfr) I'm wondering how to solve this and also is there any support to use Managed Identities for auth? This is normally possible but I don't see how via Microsoft Fabric.10KViews1like4CommentsOptions for High‑Frequency, Low‑Latency API Ingestion
I’m looking for guidance on options for invoking external APIs(with dynamic date parameters in specific format like '202602152359') at very high frequency (every 2 minutes or less) and making the data available with minimal end‑to‑end latency. Requirements: Invoke external APIs at 2‑minute intervals (or less). Parse incoming XML/JSON data into structured tables. Refresh user reports/streamlit dashboards quickly, alongside other enterprise data, to support near‑real‑time decision‑making wihtout the delay of sql endpoint refresh. We currently use a Fabric medallion architecture with payload based ingestions. However, this approach is not well‑suited for this use case, as running pipelines/notebooks on 2–5 minute schedules is triggering 70% Fabric capacity alerts.2.4KViews1like7CommentsHeartbeat Action Query Not Sent to Source
I created an Eventstream with Postgre SQL CDC as source. I want to use heartbeat action query to ensure that Eventstream to keep the replication slot size low in source. There is no issue/error in the Eventstream but the heartbeat query is never been sent to source. Is there a configuration I missed? This is the table that I created and the initial value: CREATE TABLE public.debezium_heartbeat ( id SERIAL PRIMARY KEY, timestamp TIMESTAMP WITH TIME ZONE ); INSERT INTO public.debezium_heartbeat (id, timestamp) VALUES (1, now()); This is the heartbeat action query that I put on the Eventstream source: UPDATE public.debezium_heartbeat SET timestamp = now() WHERE id = 1Solved1.2KViews1like7CommentsPreserve IoT Hub system properties and user properties
Description Events from the IoT Hub source drop key properties. We need full pass-through of system and user properties for routing and lineage. User Example Diego runs a fleet of smart vending machines. Each machine sends telemetry with a user property message-type=timeSeries. When stock is low, the device sends message-type=alert. Diego needs to route alerts directly to a monitoring system. Without user properties, he cannot build a routing rule and must instead parse message bodies, adding complexity and cost. Expected Result Map system properties to columns: deviceId, moduleId, enqueuedTime, messageId, correlationId, contentType, contentEncoding, connectionAuthMethod. Preserve all user properties as a bag. Expose user and system property keys for transformations. DeviceId is always present. IoTHub Enriched metadata is present. Actual Result DeviceId and other system properties are missing. User properties do not arrive. Impact Cannot route by user property like message-type. Cannot trace messages to devices without joins. References IoT Hub system properties: https://learn.microsoft.com/en-us/azure/iot-hub/iot-hub-devguide-messages-construct#system-properties-of-device-to-cloud-messages Repro Steps Send a D2C message with user property message-type=alarms. Inspect the record in Eventstream. Observe missing deviceId and missing user properties. Acceptance Criteria Given any D2C message, deviceId is a column. Given user properties, all appear in a properties bag. Given message-type=alarms, routing rules can use it directly. Given contentType and contentEncoding, values match the source message.Solved4.5KViews0likes6CommentsDeserialization error when publishing events to Eventstream via Event Grid after initial success
Hi, I’m trying to ingest events into Microsoft Fabric Eventstream using Azure Event Grid and ran into an issue that I’m unable to understand. Here’s what I did step by step: I created an Event Grid Namespace, followed by a custom Topic. In Fabric Eventstream, I configured a Custom Source by selecting the Event Grid namespace and topic. From the Azure portal, I noted down the topic endpoint and access key. Using Azure Cloud Shell, I started publishing events to the Event Grid topic. Initially, my plan was to ingest 10 events at once, but that didn’t work. So I changed the approach and started publishing the events one by one. This is where the issue started, the first two events were published and consumed by Eventstream successfully.When I tried to publish the third event, Eventstream threw a deserialization error.Despite that, only the third event (and subsequent ones) fail with a deserialization issue. What’s confusing is that the schema is exactly the same for all events and I followed the same steps, used the same format, and didn’t change anything between events. Has anyone faced a similar issue when publishing events from Event Grid to Fabric Eventstream? What could cause deserialization? Any known limitations or best practices for publishing events via Event Grid to Eventstream? Thank YouSolved1.1KViews0likes3CommentsSQL Operator with multiple destinations
Hi all, a few weeks back I was happy to read the SQL Operator for Eventstream is GA AND supports multiple destinations. I tried to add a second destination to an existing SQL Operator. That does not work (for me). I keep getting thr error: "Failed to create data source: <output-alias> due to the following error: Long running operation failed with status 'Failed'. Additional Info:'Stream Analytics job has validation errors: The output <output-alias> used in the query was not defined.'" The error occurs after publishing, when activating source (works fine) and destinations (fail). And I have run out of ideas what it could be. I tried: * Create a new Eventstream * Use different kinds of destinations (EH, LH) * Naming things: with and wihtout "-", "_", camelCase, lowercase * The order when and where and how to define the destinations (in the SQL editor first, in the Eventstream "main" view first, ....) * End each of the "Select into. ..." with and without ";" * There are no problems shown in the SQL Editor, Test query didn't complain either. One destination works perfectly fine. Two not at all (for me). What I observed: If I change the order of the "select into ..."s the other output-alias is the one failing. ==> It's always the first "select into ..." the error is coming from. (Probably trying to connect to this first, second probably not even tried after first fails). Maybe important: I use several CTEs to prepare the events before the final Selects. Not both destinations use the same CTE as basis for their tables. The fallback might be to have two SQL Operators, duplicating some code and stuff. I'm not looking forward to this, esp. since in ASA it (as I read) works. Any help, hints, fixes, education what I made wrong are highly appreciated. thx MartinSolved1.4KViews0likes6CommentsDeleted workspace still showing CU usage in Capacity Metrics (Eventstream)
Hi all, We deleted a workspace, but the Fabric Capacity Metrics app still shows CU usage for that workspace, and the top consumer listed is an Eventstream. What’s confusing is that Eventstream doesn’t appear in the supported item types for Item Recovery, so it seems like a non‑recoverable item. https://learn.microsoft.com/en-us/fabric/admin/retention-recovery#supported-item-types So why would it still show up as consuming CUs after the workspace was deleted? Is this expected because a deleted workspace stays in workspace retention for a period before it’s permanently removed? If yes, what’s the correct way to stop the CU usage, do we need to permanently delete the workspace from the admin portal during the retention window? Thanks, ShreyaSolved1.3KViews0likes4CommentsSpark Notebook termination after ~14 days (Eventstream Destination)
I have a realtime ingestion process as the following : In an eventstream ingesting events from an eventhub Doing some operations (filter events , ...) Set a notebook as destination of the eventstream In the notebook : apply upsert operation and append or updata data into a Lakehouse table when I add the notebook as destination an instance of notebook starts to run (I can see the activity in monitor) and working fine (the events are ingested and in Lakehouse table, evrything looks great) but after almost 14 days the notebook suddenly stops and I need to manually delete the notebook from eventstrteam and add it again as destination. Does it seem the native behaviour of Fabric to stop spending resources or there is a setting I can set it to let it run forever? ThanksSolved1.3KViews0likes3Comments