Lessons learned: A Real-Time Intelligence pipeline for Germany’s power grid
I built a real-time pipeline in Microsoft Fabric for live data from Germany’s power grid: generation by source, renewable share, cross-border flows, day-ahead prices and grid frequency per second. The data comes from the public Energy-Charts API of Fraunhofer ISE. On top of the Eventhouse sit a Real-Time Dashboard, seven Activator alerts and a Data Agent. The pipeline in short Source: The Energy-Charts API v2 of Fraunhofer ISE with five endpoints: public net generation, traffic light signal with renewable share, cross-border physical flows, day-ahead prices and grid frequency. No API key is required. Ingestion: Four Python Azure Functions (Python 3.13, Flex Consumption) call the API: grid frequency every minute, generation every 5 minutes, prices every 15 minutes, and a backfill every 3 hours for late corrections. They send JSON events via the Event Hubs protocol. Eventstream: One custom endpoint. Two filters with Manage fields route frequency data into frequency_raw and everything else into energy_raw. he Eventstream splits the data into two tables Eventhouse: Append-only raw tables with permanent retention. The materialized view energy_latest uses arg_max(generated_at) to always return the latest version of each value, so late corrections are simply sent again. For the grid frequency, frequency_1s removes duplicate seconds, frequency_1min aggregates per minute, and the function GridFrequency() adds the deviation from 50 Hz. Real-Time Dashboard: Three pages: Grid Live, Electricity Prices and Grid Frequency. KQL Queryset and Activator: A queryset holds the seven alert queries. Activator turns them into email rules, for example for negative prices, high export, grid congestion, frequency deviations or a data outage. Data Agent: Answers questions in plain English via eight read-only KQL functions and shows the generated query for every answer. It can optionally be published to Microsoft 365 Copilot. Git integration: The function code, all KQL scripts and every Fabric item (Eventstream, Eventhouse, Queryset, Dashboard, Activator and Data Agent) live in one GitHub repository. Design decisions that paid off One event per value (long format). New series need no schema change. A backfill function instead of a longer look-back. The source corrects values for many hours after the first publication. The 5-minute runs stay small, and the backfill catches the rest. Timer schedules offset by seconds. The API allows 2 requests per minute per endpoint, so the functions never call it at the same moment. Lessons learned from the Data Agent Give it functions, not tables. With access to a materialized view, the agent bypassed my functions, returned a forecast value as “now” and showed UTC as local time. Now only eight KQL functions are selected as the data source. Encapsulate analyses. For “How does the price depend on the renewable share?” the agent computed something different on every run. A dedicated function, PriceVsRenewableShare(), fixed that. Return every figure once. When totals were repeated in every row, the model summed them up. The function now returns totals as a separate row. Questions in English. Other languages are not officially supported yet. Full walkthrough and code Blog: From API to Data Agent GitHub: ms-fabric / real-time-intelligence-project-energy Data: Energy-Charts.info (Fraunhofer ISE), Bundesnetzagentur | SMARD.de, CC BY 4.0 About the author I’m Rene Fürstenberg. I’ve worked with Microsoft technologies since 2004, with Azure analytics since 2016, and I’ve been a Microsoft Trainer since 2019. Today my focus is Microsoft Fabric: data engineering, Real-Time Intelligence and Power BI. I offer consulting and training in English and German. If you want to dig deeper or have questions about this project, you can find me here: Blog: renefuerstenberg.de, with more hands-on guides on Fabric and Azure LinkedIn: Rene Fürstenberg. Feel free to connect or send me a message. YouTube: Cloud Junkies, videos on Microsoft Fabric (in German) GitHub: ms-fabric, the code for this and other Fabric projects Medium: @renefuerstenberg
Changing Ingestions from JSON
I have an EventStream receiving IoT Data out of JSONs. In my KQL table the columns timestamp, deviceId and for example x and y exist. The data from one sensor gets ingested in the columns timestamp, deviceId and y. No x since there is no data in the JSON for that. I dont work with a Mapping because it only makes it more complicated and the names in the JSON and in the KQL table are identical. No I changed the named for data from y to x when creating the JSON. Since there is no mapping and the column already exists in the KQL table the data should be saved now in column x. My problem is, it doesn't the data is not in x nor in y. It seems to be gone. Is that a bug in fabric? Can I solve this?
Global Aircraft✈️ Live Tracking with Microsoft Fabric Real-Time Intelligence
In this blog, I’ll walk you through, build a real-time global flight tracking system. We will ingest live flight data from the public OpenSky Network API using a Python polling script, stream it through Microsoft Fabric Eventstream into an Eventhouse (KQL Database), transform the dense raw arrays using KQL update policies and visualize the results on a Real-Time Dashboard complete with maps, KPIs and analytical charts. Prerequisites: Valid Fabric Capacity / Trail License Knowledge on Python Knowledge on KQL Step1: Setup Workspace & Eventhouse (KQL Database) Created a workspace “FlightTracking-[WS]” Created an eventhouse “FLightTracking-EH” Create a raw ingestion table “RawFlightbatc” in KQL Databse Step 2: Setup a Fabric Eventstream Created a Eventstream “GlobalFlightStream” and select ‘Use custom endpoint’ Click ‘Add’ Click on ‘Publish’ Copy the ‘Event hub name’ and ‘connection string-primary key’ into notepad Step 3: Notebook Creation and Setup Python script Created a notebook. Make sure select Python Install azure eventhub package Let’s go back to eventstream and add destination by selecting ‘Eventhouse’ Configure all details and click on save Comeback to Notebook and insert the Python script which is having all the connection strings / passwords etc. Run the notebook Now notebook started running and sending the data to eventstream Data is loaded into eventstream and Click on Publish Now eventstream is ‘Live’ Data is loading into KQL Database Step 4: Regularizing Data with KQL & Update Policies Create a cleaned table “FlightStates” Create the parsing function and update policy Alter table with updated policy Step 5: Building Real-Time Dashboard Click on Realtime dashboard and give a name “FlightOperationsDashboard” Now click on edit Run the below code to get total active flights count Change the chart to Stat and rename, click on Apply KPI added and click on Add visual and take new Stat visual Insert the code and run to get India Origin Flights and Format it and click Apply In the same way, I built other KPIs Select Map Chart Run the below code and Fill all details and Click on Apply. Here we’re calculating Flights trend We can see chart added to Dashboard Select a Bar chart Run the below code, fill all details and click on apply to add Bar chart to Dashboard. Here we’re getting the top 10 countries by aircrafts Run the below code, fill all details and click on apply to add Column chart to Dashboard. Here we’re categorizing the baro-altitude which is critical for airport delay predection Run the below code, fill all details and click on apply to add Pie chart to Dashboard. It splits the aircraft parked versus those actively flying Together, these visuals transform raw flight telemetry into an operational monitoring experience. Key takeaways: Fabric Eventstream provides a streamlined way to ingest external real-time feeds. Eventhouse/KQL Database provides a real-time analytical environment for flight telemetry. KQL mv-expand simplifies the processing of nested flight-state arrays. Update Policies automate transformation from raw streaming data into structured analytical data. KQL geospatial functions enable location-based flight analysis. Real-Time Dashboards transform streaming telemetry into actionable operational insights. Conclusion: This project demonstrates how Microsoft Fabric Real-Time Intelligence can be used to build an end-to-end real-time aircraft tracking solution. we can transform continuously arriving aircraft telemetry into meaningful real-time insights. Do you want to replicate? Get Code file from my GitHub link You can find all the KQL queries, update policy functions, and the complete Python polling script in the official GitHub repository below: [Download from here] Acknowledgements I would like to express my sincere gratitude to @SuryaTejaJosyul , @minniwalia and @rajendraongole1 for their continuous guidance and support throughout this Real-Time Intelligence (RTI) implementation. Their insights and encouragement played a key role in helping me complete this solution successfull Happy learning! — Inturi Suparna Babu [LinkedIn]
