Lessons learned: A Real-Time Intelligence pipeline for Germany’s power grid
I built a real-time pipeline in Microsoft Fabric for live data from Germany’s power grid: generation by source, renewable share, cross-border flows, day-ahead prices and grid frequency per second. The data comes from the public Energy-Charts API of Fraunhofer ISE. On top of the Eventhouse sit a Real-Time Dashboard, seven Activator alerts and a Data Agent. The pipeline in short Source: The Energy-Charts API v2 of Fraunhofer ISE with five endpoints: public net generation, traffic light signal with renewable share, cross-border physical flows, day-ahead prices and grid frequency. No API key is required. Ingestion: Four Python Azure Functions (Python 3.13, Flex Consumption) call the API: grid frequency every minute, generation every 5 minutes, prices every 15 minutes, and a backfill every 3 hours for late corrections. They send JSON events via the Event Hubs protocol. Eventstream: One custom endpoint. Two filters with Manage fields route frequency data into frequency_raw and everything else into energy_raw. he Eventstream splits the data into two tables Eventhouse: Append-only raw tables with permanent retention. The materialized view energy_latest uses arg_max(generated_at) to always return the latest version of each value, so late corrections are simply sent again. For the grid frequency, frequency_1s removes duplicate seconds, frequency_1min aggregates per minute, and the function GridFrequency() adds the deviation from 50 Hz. Real-Time Dashboard: Three pages: Grid Live, Electricity Prices and Grid Frequency. KQL Queryset and Activator: A queryset holds the seven alert queries. Activator turns them into email rules, for example for negative prices, high export, grid congestion, frequency deviations or a data outage. Data Agent: Answers questions in plain English via eight read-only KQL functions and shows the generated query for every answer. It can optionally be published to Microsoft 365 Copilot. Git integration: The function code, all KQL scripts and every Fabric item (Eventstream, Eventhouse, Queryset, Dashboard, Activator and Data Agent) live in one GitHub repository. Design decisions that paid off One event per value (long format). New series need no schema change. A backfill function instead of a longer look-back. The source corrects values for many hours after the first publication. The 5-minute runs stay small, and the backfill catches the rest. Timer schedules offset by seconds. The API allows 2 requests per minute per endpoint, so the functions never call it at the same moment. Lessons learned from the Data Agent Give it functions, not tables. With access to a materialized view, the agent bypassed my functions, returned a forecast value as “now” and showed UTC as local time. Now only eight KQL functions are selected as the data source. Encapsulate analyses. For “How does the price depend on the renewable share?” the agent computed something different on every run. A dedicated function, PriceVsRenewableShare(), fixed that. Return every figure once. When totals were repeated in every row, the model summed them up. The function now returns totals as a separate row. Questions in English. Other languages are not officially supported yet. Full walkthrough and code Blog: From API to Data Agent GitHub: ms-fabric / real-time-intelligence-project-energy Data: Energy-Charts.info (Fraunhofer ISE), Bundesnetzagentur | SMARD.de, CC BY 4.0 About the author I’m Rene Fürstenberg. I’ve worked with Microsoft technologies since 2004, with Azure analytics since 2016, and I’ve been a Microsoft Trainer since 2019. Today my focus is Microsoft Fabric: data engineering, Real-Time Intelligence and Power BI. I offer consulting and training in English and German. If you want to dig deeper or have questions about this project, you can find me here: Blog: renefuerstenberg.de, with more hands-on guides on Fabric and Azure LinkedIn: Rene Fürstenberg. Feel free to connect or send me a message. YouTube: Cloud Junkies, videos on Microsoft Fabric (in German) GitHub: ms-fabric, the code for this and other Fabric projects Medium: @renefuerstenbergGlobal Aircraft✈️ Live Tracking with Microsoft Fabric Real-Time Intelligence
In this blog, I’ll walk you through, build a real-time global flight tracking system. We will ingest live flight data from the public OpenSky Network API using a Python polling script, stream it through Microsoft Fabric Eventstream into an Eventhouse (KQL Database), transform the dense raw arrays using KQL update policies and visualize the results on a Real-Time Dashboard complete with maps, KPIs and analytical charts. Prerequisites: Valid Fabric Capacity / Trail License Knowledge on Python Knowledge on KQL Step1: Setup Workspace & Eventhouse (KQL Database) Created a workspace “FlightTracking-[WS]” Created an eventhouse “FLightTracking-EH” Create a raw ingestion table “RawFlightbatc” in KQL Databse Step 2: Setup a Fabric Eventstream Created a Eventstream “GlobalFlightStream” and select ‘Use custom endpoint’ Click ‘Add’ Click on ‘Publish’ Copy the ‘Event hub name’ and ‘connection string-primary key’ into notepad Step 3: Notebook Creation and Setup Python script Created a notebook. Make sure select Python Install azure eventhub package Let’s go back to eventstream and add destination by selecting ‘Eventhouse’ Configure all details and click on save Comeback to Notebook and insert the Python script which is having all the connection strings / passwords etc. Run the notebook Now notebook started running and sending the data to eventstream Data is loaded into eventstream and Click on Publish Now eventstream is ‘Live’ Data is loading into KQL Database Step 4: Regularizing Data with KQL & Update Policies Create a cleaned table “FlightStates” Create the parsing function and update policy Alter table with updated policy Step 5: Building Real-Time Dashboard Click on Realtime dashboard and give a name “FlightOperationsDashboard” Now click on edit Run the below code to get total active flights count Change the chart to Stat and rename, click on Apply KPI added and click on Add visual and take new Stat visual Insert the code and run to get India Origin Flights and Format it and click Apply In the same way, I built other KPIs Select Map Chart Run the below code and Fill all details and Click on Apply. Here we’re calculating Flights trend We can see chart added to Dashboard Select a Bar chart Run the below code, fill all details and click on apply to add Bar chart to Dashboard. Here we’re getting the top 10 countries by aircrafts Run the below code, fill all details and click on apply to add Column chart to Dashboard. Here we’re categorizing the baro-altitude which is critical for airport delay predection Run the below code, fill all details and click on apply to add Pie chart to Dashboard. It splits the aircraft parked versus those actively flying Together, these visuals transform raw flight telemetry into an operational monitoring experience. Key takeaways: Fabric Eventstream provides a streamlined way to ingest external real-time feeds. Eventhouse/KQL Database provides a real-time analytical environment for flight telemetry. KQL mv-expand simplifies the processing of nested flight-state arrays. Update Policies automate transformation from raw streaming data into structured analytical data. KQL geospatial functions enable location-based flight analysis. Real-Time Dashboards transform streaming telemetry into actionable operational insights. Conclusion: This project demonstrates how Microsoft Fabric Real-Time Intelligence can be used to build an end-to-end real-time aircraft tracking solution. we can transform continuously arriving aircraft telemetry into meaningful real-time insights. Do you want to replicate? Get Code file from my GitHub link You can find all the KQL queries, update policy functions, and the complete Python polling script in the official GitHub repository below: [Download from here] Acknowledgements I would like to express my sincere gratitude to @SuryaTejaJosyul , @minniwalia and @rajendraongole1 for their continuous guidance and support throughout this Real-Time Intelligence (RTI) implementation. Their insights and encouragement played a key role in helping me complete this solution successfull Happy learning! — Inturi Suparna Babu [LinkedIn]From Rain Drops to Playability: Scoring Real-Time Weather Insights for FIFA World Cup 2026⚽
In this blog, I will walk you through how we can support the FIFA World Cup 2026 by organizing teams and players by delivering real-time, actionable weather insights. By predicting critical weather patterns - such as high humidity, extreme heat, and sudden rainfall - teams can proactively adjust their game-day strategies, and organizers can ensure optimal player safety. To bring this vision to life, we will build a comprehensive, real-time weather monitoring dashboard covering all 16 FIFA World Cup stadiums, powered entirely by the real-time analytics capabilities of Microsoft FabricEventhouse vs Lakehouse - When to Use Which in Microsoft Fabric?
Microsoft Fabric offers multiple storage and analytical engines. This blog helps you make the right architectural choice early on and save them massive migration efforts later. In this blog, I’ll walk you through the differences between these two powerhouses, look at real-world challenges, map out concrete scenarios, and give you a definitive comparison framework so you can choose the right tool for the job.Building Real-Time Indian Railways Dashboard in Microsoft Fabric – A step-by-step guide
In this blog, I’ll walk you through how to build a Real-Time Indian Railways Train Running Status Dashboard using Microsoft Fabric. We’ll stream live railway data from a public API into Eventstream, store it in Eventhouse, transform it using KQL, and finally visualize it in a real-time dashboard.Building a Real-Time Weather Dashboard in Microsoft Fabric: Step-by-Step Guide
In this blog, I’ll walk you through how to build a Real-Time Weather Analytics Dashboard in Microsoft Fabric by streaming public weather feeds into Eventstream, storing them in Eventhouse, and finally visualizing them in a Real-Time Dashboard. For this example, I’m using Bengaluru, India and several of its suburban locations.
Comparing JSON in Eventhouse Without Defining Schema
When working with raw JSON data in Eventhouse, one common challenge is comparing two records to identify what changed—especially when the structure isn’t fixed. Fields may appear, disappear, or shift in type, making traditional column-based comparisons brittle or outright impossible. This post will walk you through how to compare two JSON arrays and find what changed without needing to define the schema first.KQL: First Steps to Analyze and Visualize your Data
So, you’ve started exploring real-time intelligence in Fabric and maybe even followed the Microsoft tutorial to implement a real-time solution. Now, you’re looking to analyze and visualize that data, and that’s where KQL (Kusto Query Language) comes in. When I started, I struggled a bit, so I thought it would be a good idea to provide some basic queries (and explain what they do) for people who want to get started but are unfamiliar with querying data.
