Lessons learned: A Real-Time Intelligence pipeline for Germany’s power grid
I built a real-time pipeline in Microsoft Fabric for live data from Germany’s power grid: generation by source, renewable share, cross-border flows, day-ahead prices and grid frequency per second. The data comes from the public Energy-Charts API of Fraunhofer ISE. On top of the Eventhouse sit a Real-Time Dashboard, seven Activator alerts and a Data Agent. The pipeline in short Source: The Energy-Charts API v2 of Fraunhofer ISE with five endpoints: public net generation, traffic light signal with renewable share, cross-border physical flows, day-ahead prices and grid frequency. No API key is required. Ingestion: Four Python Azure Functions (Python 3.13, Flex Consumption) call the API: grid frequency every minute, generation every 5 minutes, prices every 15 minutes, and a backfill every 3 hours for late corrections. They send JSON events via the Event Hubs protocol. Eventstream: One custom endpoint. Two filters with Manage fields route frequency data into frequency_raw and everything else into energy_raw. he Eventstream splits the data into two tables Eventhouse: Append-only raw tables with permanent retention. The materialized view energy_latest uses arg_max(generated_at) to always return the latest version of each value, so late corrections are simply sent again. For the grid frequency, frequency_1s removes duplicate seconds, frequency_1min aggregates per minute, and the function GridFrequency() adds the deviation from 50 Hz. Real-Time Dashboard: Three pages: Grid Live, Electricity Prices and Grid Frequency. KQL Queryset and Activator: A queryset holds the seven alert queries. Activator turns them into email rules, for example for negative prices, high export, grid congestion, frequency deviations or a data outage. Data Agent: Answers questions in plain English via eight read-only KQL functions and shows the generated query for every answer. It can optionally be published to Microsoft 365 Copilot. Git integration: The function code, all KQL scripts and every Fabric item (Eventstream, Eventhouse, Queryset, Dashboard, Activator and Data Agent) live in one GitHub repository. Design decisions that paid off One event per value (long format). New series need no schema change. A backfill function instead of a longer look-back. The source corrects values for many hours after the first publication. The 5-minute runs stay small, and the backfill catches the rest. Timer schedules offset by seconds. The API allows 2 requests per minute per endpoint, so the functions never call it at the same moment. Lessons learned from the Data Agent Give it functions, not tables. With access to a materialized view, the agent bypassed my functions, returned a forecast value as “now” and showed UTC as local time. Now only eight KQL functions are selected as the data source. Encapsulate analyses. For “How does the price depend on the renewable share?” the agent computed something different on every run. A dedicated function, PriceVsRenewableShare(), fixed that. Return every figure once. When totals were repeated in every row, the model summed them up. The function now returns totals as a separate row. Questions in English. Other languages are not officially supported yet. Full walkthrough and code Blog: From API to Data Agent GitHub: ms-fabric / real-time-intelligence-project-energy Data: Energy-Charts.info (Fraunhofer ISE), Bundesnetzagentur | SMARD.de, CC BY 4.0 About the author I’m Rene Fürstenberg. I’ve worked with Microsoft technologies since 2004, with Azure analytics since 2016, and I’ve been a Microsoft Trainer since 2019. Today my focus is Microsoft Fabric: data engineering, Real-Time Intelligence and Power BI. I offer consulting and training in English and German. If you want to dig deeper or have questions about this project, you can find me here: Blog: renefuerstenberg.de, with more hands-on guides on Fabric and Azure LinkedIn: Rene Fürstenberg. Feel free to connect or send me a message. YouTube: Cloud Junkies, videos on Microsoft Fabric (in German) GitHub: ms-fabric, the code for this and other Fabric projects Medium: @renefuerstenbergGlobal Aircraft✈️ Live Tracking with Microsoft Fabric Real-Time Intelligence
In this blog, I’ll walk you through, build a real-time global flight tracking system. We will ingest live flight data from the public OpenSky Network API using a Python polling script, stream it through Microsoft Fabric Eventstream into an Eventhouse (KQL Database), transform the dense raw arrays using KQL update policies and visualize the results on a Real-Time Dashboard complete with maps, KPIs and analytical charts. Prerequisites: Valid Fabric Capacity / Trail License Knowledge on Python Knowledge on KQL Step1: Setup Workspace & Eventhouse (KQL Database) Created a workspace “FlightTracking-[WS]” Created an eventhouse “FLightTracking-EH” Create a raw ingestion table “RawFlightbatc” in KQL Databse Step 2: Setup a Fabric Eventstream Created a Eventstream “GlobalFlightStream” and select ‘Use custom endpoint’ Click ‘Add’ Click on ‘Publish’ Copy the ‘Event hub name’ and ‘connection string-primary key’ into notepad Step 3: Notebook Creation and Setup Python script Created a notebook. Make sure select Python Install azure eventhub package Let’s go back to eventstream and add destination by selecting ‘Eventhouse’ Configure all details and click on save Comeback to Notebook and insert the Python script which is having all the connection strings / passwords etc. Run the notebook Now notebook started running and sending the data to eventstream Data is loaded into eventstream and Click on Publish Now eventstream is ‘Live’ Data is loading into KQL Database Step 4: Regularizing Data with KQL & Update Policies Create a cleaned table “FlightStates” Create the parsing function and update policy Alter table with updated policy Step 5: Building Real-Time Dashboard Click on Realtime dashboard and give a name “FlightOperationsDashboard” Now click on edit Run the below code to get total active flights count Change the chart to Stat and rename, click on Apply KPI added and click on Add visual and take new Stat visual Insert the code and run to get India Origin Flights and Format it and click Apply In the same way, I built other KPIs Select Map Chart Run the below code and Fill all details and Click on Apply. Here we’re calculating Flights trend We can see chart added to Dashboard Select a Bar chart Run the below code, fill all details and click on apply to add Bar chart to Dashboard. Here we’re getting the top 10 countries by aircrafts Run the below code, fill all details and click on apply to add Column chart to Dashboard. Here we’re categorizing the baro-altitude which is critical for airport delay predection Run the below code, fill all details and click on apply to add Pie chart to Dashboard. It splits the aircraft parked versus those actively flying Together, these visuals transform raw flight telemetry into an operational monitoring experience. Key takeaways: Fabric Eventstream provides a streamlined way to ingest external real-time feeds. Eventhouse/KQL Database provides a real-time analytical environment for flight telemetry. KQL mv-expand simplifies the processing of nested flight-state arrays. Update Policies automate transformation from raw streaming data into structured analytical data. KQL geospatial functions enable location-based flight analysis. Real-Time Dashboards transform streaming telemetry into actionable operational insights. Conclusion: This project demonstrates how Microsoft Fabric Real-Time Intelligence can be used to build an end-to-end real-time aircraft tracking solution. we can transform continuously arriving aircraft telemetry into meaningful real-time insights. Do you want to replicate? Get Code file from my GitHub link You can find all the KQL queries, update policy functions, and the complete Python polling script in the official GitHub repository below: [Download from here] Acknowledgements I would like to express my sincere gratitude to @SuryaTejaJosyul , @minniwalia and @rajendraongole1 for their continuous guidance and support throughout this Real-Time Intelligence (RTI) implementation. Their insights and encouragement played a key role in helping me complete this solution successfull Happy learning! — Inturi Suparna Babu [LinkedIn]
Handling Multiple Schemas in Eventstream for Azure Event Hub and Azure IOT Hub data sources
One of the challenges with streaming data is that data comes in a variety of schemas, which can be dynamic and are not always predictable. As applications and data structures change, schema values can sometimes be wildly different across devices or event inputs. Recently a customer reached out with several questions on this common problem. With the capabilities in Fabric Real Time Intelligence we can flexibly ingest this data. Enough of the background let’s get to the problem!Eventhouse vs Lakehouse - When to Use Which in Microsoft Fabric?
Microsoft Fabric offers multiple storage and analytical engines. This blog helps you make the right architectural choice early on and save them massive migration efforts later. In this blog, I’ll walk you through the differences between these two powerhouses, look at real-world challenges, map out concrete scenarios, and give you a definitive comparison framework so you can choose the right tool for the job.🚦From Signals to Insights: Bengaluru Smart City Traffic Updates with Fabric Real-Time Intelligence
In this blog, we will build a Bengaluru Smart City Traffic Updates Dashboard using Microsoft Fabric Real-Time Intelligence. We'll simulate live traffic signals from major Bengaluru junctions, stream events into Fabric, store them in a KQL database, and create an interactive real-time dashboard that provides actionable traffic insights. By streaming events into Fabric and leveraging powerful analytics, you’ll see how raw signals can be transformed into actionable insights for smarter urban mobility.
Optimizing Real-Time Analytics in Microsoft Fabric RTI with Follower and Read-Optimized Databases
This article discusses about an architectural pattern to achieve better performance when trying to work with traditional relational database schema in Microsoft Fabric Real Time Intelligence (RTI). A common scenario is that you have Fabric as the enterprise data platform that ingests data from a live system running at scale. This article addresses two common requirements: You have streaming ingestion with variety of consumers for the data. You have use cases that demand fast response time even when the query is joining huge tables. The pattern discussed here uses a database shortcut in Fabric RTI, also known as a follower database, and a read-write database. With the database shortcut, each downstream system gets dedicated capacity and SLAs. With a read-write database, data can be denormalized and stored in a read-optimized format to improve real time query performance.Building a Real-Time Weather Dashboard in Microsoft Fabric: Step-by-Step Guide
In this blog, I’ll walk you through how to build a Real-Time Weather Analytics Dashboard in Microsoft Fabric by streaming public weather feeds into Eventstream, storing them in Eventhouse, and finally visualizing them in a Real-Time Dashboard. For this example, I’m using Bengaluru, India and several of its suburban locations.
Eventhouse and its best practices
As data volumes grow and real‑time insights become critical, choosing the right analytics platform matters. Eventhouse simplifies large‑scale data ingestion and querying while offering smart controls for cost, performance, and reliability. In this article, we dive into how Eventhouse works and the best practices that help you get the most value from it.Comparing JSON in Eventhouse Without Defining Schema
When working with raw JSON data in Eventhouse, one common challenge is comparing two records to identify what changed—especially when the structure isn’t fixed. Fields may appear, disappear, or shift in type, making traditional column-based comparisons brittle or outright impossible. This post will walk you through how to compare two JSON arrays and find what changed without needing to define the schema first.
