Blog Post

Fabric Updates Blog
6 MIN READ

Intelligent streaming: Real-time game-day operations with Fabric Eventstreams, Spark and AI functions

arindamc's avatar
arindamc
Icon for Microsoft Employee rankMicrosoft Employee
3 months ago

On game day, every second counts

We're just days away from one of the biggest sporting events of the year — the FIFA World Cup Final. Ever wondered how the venue behind such an event keeps 80,000 fans safe, fed, entertained, and moving? A few thousand staff on handhelds. A single operational database catching every guest request as it happens. A concession stand on the North concourse running low on hot dog buns 20 minutes before halftime, a merch kiosk mobbed after a jumbotron warmup replay, a spill on the main concourse. Each row/event matters. Each row can have a mix of structured data & free text. And each row is useful only if the right team can act within seconds.

This is an example of a problem every operations team knows: an operational database changing constantly, and a small set of teams that must act on those changes now. This is the same row-to-action pattern we have discussed before in with one twist. The choice of action to take depends on understanding what the row means rather than checking a threshold.

This is the row-to-intelligent-action pattern. This needs an AI step that interprets the row, takes context into account, and then recommends or takes an action.

Fabric provides three building blocks to implement this pattern:

  • Native CDC source connectors that capture every insert, update, and delete from the operational database as it happens.
  • DeltaFlow (Preview) feature that reshapes those raw Debezium CDC envelopes into source-table-shaped, analytics-ready rows.
  • A Spark Structured Streaming Notebook destination that reads the flattened stream of events and uses AI Functions for PySpark to reason (classify, extract, summarize, or embed) over any combination of columns on every micro-batch before writing enriched output onward.

Figure: Row-to-Intelligent-Action in Real-Time

For the rest of this post, we'll stay with the World Cup–scale stadium as our running example. The staff handhelds write into an operational database — one row per guest request, with a free-text request_description (a concession restock or a merchandise surge). The streaming pipeline captures each row change, classifies the request, and routes actionable categories to the right team in seconds — triggering on meaning ("almost halftime, hot dog buns running low near the North stand"), not on a threshold.

If you are new to the CDC + DeltaFlow half of this pattern, our earlier post Building real-time, event-driven applications with Database CDC and DeltaFlow walks through it end-to-end. This post picks up from there and adds the AI-enrichment branch.

From change data to AI-driven routing

The first thing the ops team needs is for every incoming request to be categorized. The routing decision depends on it. This is easily done using a Spark Structured Streaming Notebook that reads the DeltaFlow-shaped change feed from an Eventstream. The requests inside each micro-batch are then classified using AI Functions for PySpark.

# Inside foreachBatch — batch_df is one micro-batch from the DeltaFlow-flattened stream.
actionable = batch_df.filter(
    "_change_type IN ('insert','update_postimage') AND length(trim(request_description)) > 0"
)
classified = actionable.ai.classify(
    labels=["ConcessionRestock", "MerchRestock", "Accessibility",
            "LostAndFound", "Complaint", "Medical", "Security"],
    input_col="request_description",
    output_col="predicted_category",
)
# Fan out: full AI-enriched history to Silver, action categories to Gold,
# and publish a Business Event for each actionable row so downstream teams can act.
classified.write.format("delta").mode("append").saveAsTable("Silver.GuestRequests_enriched")
action = classified.filter(
"predicted_category IN ('ConcessionRestock','MerchRestock','Medical','Security','Accessibility')"
)
action.write.format("delta").mode("append").saveAsTable("Gold.ActionableRequests")
notebookutils.businessEvents.publish(
    eventSchemaSetWorkspace="<workspace_name>",
    eventSchemaSet="<schemaset_name>",
    eventTypeName="ActionableGuestRequest",
    eventData=action, dataVersion="v1"
)

For more details about connecting to an Eventstream from Spark Notebook, follow the steps in the documentation Query Microsoft Fabric Eventstream documentation.

The labels define your routing taxonomy; ai.classify returns predicted_category on the DataFrame; each actionable row is emitted as a Business Event carrying the AI-classified category and the source-row context. Concretely:

Input row: request_description = "hot dog buns down to 40, beer racks halfway, line 15 deep, 20 min to halftime"

Output column: predicted_category = "ConcessionRestock"

Fabric manages the underlying inference — no model server to operate. The AI step runs in the same micro-batch as the rest of the Spark logic, so end-to-end latency stays predictable. And the output is just another column — joinable, aggregatable, alertable. See the AI Functions for PySpark reference for the full method surface (extract, summarize, similarity, embed, sentiment).

From AI insight to real-time action

With that classification in hand, the ops team wants the right downstream systems to know about actionable requests the moment they land. In the same micro-batch, the notebook publishes each actionable row as a Business Event — a first-class, discoverable signal in the that carries the AI-enriched category and the source-row and any additional context. From there, the concessions restock team sees a Teams post, the ops system opens a ticket via Power Automate and a partner logistics platform receives a webhook.

Because the alert carries whatever the AI step produced, the concessions team knows what to do, not just that something changed. That's the shift this whole pattern exists for.

And once intelligent routing is in place, the same enriched stream unlocks a second step: when many stands across a concourse report shortages within the same window, a downstream notebook (or another Eventstream operator) spots the cluster and triggers a mass-replenishment order and a temporary staffing bump — the same pattern, one step up.

Live views for the ops center

Alongside the alerting path, the same enriched output powers the live operational picture. The Operations Manager monitors Concession restock requests cluster on the north concourse while a Merchandise spike forms at gate B — all on a Real-Time Dashboard that is refreshed within seconds.

Once the match ends, Eventhouse still has every AI-classified row and its source-row context in one place, so analysts can rewind and replay the concession spike quarter by quarter, spot the misclassifications the model made, and tune the label taxonomy before the next event.

End-to-end: from a staff handheld to event handled

Figure: End-to-end game day operations loop

From the moment a stand manager logs "hot dog buns down to 40, beer racks halfway, line 15 deep" on their handheld, the loop runs itself:

  1. The row lands in Azure SQL as the handheld submits the update.
  2. The CDC connector with DeltaFlow enabled captures and reshapes the change onto the Eventstream.
  3. The Spark Streaming Notebook with AI Functions classifies the request inside foreachBatch.
  4. A Business Event fires and Teams pages the concessions restock team, with the AI-classified category already on the alert.

In parallel, the enriched stream is captured in Eventhouse for retrospective analysis (step 5), and operations center can monitor the concourse heatmap on the Real-Time Dashboard (step 6).

The result: what used to be a walkie-talkie call to a central dispatcher, manually triaged, becomes a live operational loop. The restock team gets the north-concourse alert in seconds with the routing decision already made. And by the time the second-half whistle blows, the stands are fully stocked and the World Cup Final rolls on: unremarked-upon operations, exactly as they should be

Beyond game day

The same row-to-intelligent-action pattern is applicable is many scenarios including:

  • Airports and transit: Route accessibility, medical, and security requests logged at gates to the right desk in seconds.
  • Hospital operations: Classify housekeeping requests, pharmacy stock-out notes, bed-management updates into routing categories.
  • Field service and utilities: Summarize technician job notes and route urgent items (gas leak, downed line, customer hazard etc.) to the on-call dispatcher the moment they are logged.
  • Retail and warehouse operations: Classify store-associate or floor-supervisor incident reports so leadership sees urgent items immediately.

In each case, the building blocks are the same — only the table, the context, and the thresholds change.

Get started today

  1. Pick a CDC source, create an Eventstream and add an CDC source connector to your operational database (Azure SQL Database CDC Source in Fabric Eventstream).
  2. Enable DeltaFlow and route output to an Eventhouse destination. Build a Real-Time Dashboard.
  3. Develop a Spark Structured Streaming Notebook, use the auto-generated PySpark snippet to read the change feed via the Kafka adapter, and enrich the feed using AI Functions for PySpark
  4. Close the loop, write enriched output to Silver/Gold, publish a Business Event from the notebook for each actionable row, and build a Real-Time Dashboard over the enriched Eventhouse stream for the live ops picture.

Explore the following resources:

We'd love your feedback!

If you find this blog helpful, please give it a thumbs-up! Have ideas for what you'd like to see next? Drop us a comment or reach out to [email protected] — we'd love to hear what real-time scenarios you're building and what topics you'd like us to cover in future posts.

Have new feature ideas? Submit them at Fabric Ideas - Microsoft Fabric Community

Updated 3 months ago
Version 1.0
No CommentsBe the first to comment