real-time hub
139 TopicsDeep-Dive: Microsoft Fabric Warehouse Internal Architecture (Polaris, V-Order & Lakehouse Tradeoffs)
Hi Fabric Community, I’ve spent the last few weeks researching and writing a complete architectural guide on the internals of the Microsoft Fabric Warehouse, and wanted to share some key design patterns and takeaways with you. When moving from Azure Synapse Dedicated Pools or SQL Server to Fabric, the mental model shift can be challenging—especially regarding serverless execution and open-standard storage. Here is a summary of the 3 most critical findings from my architecture breakdown: 1. The Polaris Engine Mechanics Unlike traditional Dedicated SQL Pools which bind compute to VM instances, Fabric Warehouse runs on the Polaris distributed SQL engine. Query compilation is split into stateless metadata services. Compute nodes are allocated dynamically per query. There is no local page storage; instead, the engine reads and writes open-source Delta Parquet format files directly in OneLake. 2. The Reality of Constraints A common point of confusion is table constraints. Fabric Warehouse allows you to define Primary Keys, Foreign Keys, and Unique Constraints, but they are not enforced during data ingestion. They serve purely as metadata for the query optimizer (to improve join paths). You must handle data validation/enforcement upstream in your Spark/Dataflow pipelines! 3. Direct Lake & Cold Queries Power BI's Direct Lake mode is a game-changer because it bypasses the SQL endpoint and queries the Delta Parquet files directly. However, if the dataset size exceeds the F-SKU memory limits, the query falls back to DirectQuery mode. This fallback forces the engine to run SQL queries on the SQL endpoint, introducing cold-query compilation times. I have published the full 11,000-word engineering handbook covering capacity planning, monitoring DMVs, security setups (RLS/OLS), and a retail case study. If you are planning an enterprise migration or studying for the DP-600 / DP-700 exams, check out the deep dive here: 👉 Read the Microsoft Fabric Warehouse Architecture & Performance Guide Would love to hear how your teams are managing cold query compilation and metadata cache warming on Fabric SQL endpoints. Let’s discuss below!141Views4likes3CommentsUnable to create Eventhouse
I'm an admin on Fabric and am working on the End-to-End Real-time intelligence tutorial. The following screenshot of my Fabric Capacity Metric app is showing statitics with high capacity usage for three workspaces that have been deleted more than 40 hours ago: From Capacity Metric Compute tab: From Health tab: Even more than 40 hours after deleting those 3 workspaces with hig capacity issue, I'm still getting following error when trying to create an Eventhouse: Eventhouse item could not be created because your capacity has reached its maximum limit. Contact the capacity administrator for help. Can someone please help resolve the issue?Solved160Views0likes3CommentsFabric Apps Not in UK
There doesn't seem to be any timeline to rollout Fabric apps in the UK regions; this is a bit of a nightmare for GDPR reasons. It feels like Databricks hosted in the UK with any visualisations or apps we choose would be better. I think it's quite annoying to not have a timeline, and makes it hard to have a full strategy.267Views0likes6CommentsIf you could improve one thing in Microsoft Fabric tomorrow, what would it be?
No platform is perfect-including Microsoft Fabric. While I'm genuinely impressed by how quickly the platform has evolved, there are still areas where I think the experience could become even better. If the Fabric product team gave you one opportunity to improve the platform tomorrow... What would you choose? Would it be: • Better monitoring? • Improved CI/CD? • More governance capabilities? • Better Git integration? • Performance improvements? • Licensing or capacity changes? • Something completely different? I'm interested in hearing opinions from people using Fabric every day. Sometimes the best ideas come from real customer experiences rather than feature announcements. What would be your one improvement?92Views0likes2Commentsazure-kusto-spark enabling writeMode to KustoStreaming is not available
Using documentation as reference https://github.com/Azure/azure-kusto-spark/blob/master/docs/KustoSink.md, kusto connector seems to bring KustoStreaming as posible option for writeMode. I intented to achieve the less latency as possible where I'm computing some results based on streaming input data and sink the output into a kusto table with streaming policy enabled. This KustoStreaming, is not available in the version of the kusto connector available in Microsoft fabric when I use Spark 3.5. However, seems available in Spark 4.0, meaning I can set up but sadly fails in execution time throwing an excepcion I reported https://github.com/Azure/azure-kusto-spark/issues/479. In any case I want to stick to Spark 3.5 and I was wondering how a more modern version of azure-kusto-spark connector (which already have this functionality embedded) can be included in the Spark runtime? Currently I got this in my fabric notebook spark-kusto-synapse-connector_3.5_2.12-1.5.1.jar' This is my code sp_json = json.dumps({ "flushImmediately": True }) query = events_df.writeStream. \ format("com.microsoft.kusto.spark.synapse.datasink.KustoSynapseSinkProvider"). \ option("kustoCluster", kusto_cluster). \ option("kustoDatabase", kusto_db). \ option("kustoTable", events_table). \ option("accessToken", accessToken). \ option("writeMode", 'Queued'). \ option("pollingOnDriver", 'False'). \ option("sparkIngestionPropertiesJson", sp_json). \ option("checkpointLocation", "Files/outage_events_pyspark_eventhouse/_checkpoint") . \ trigger(processingTime= "5 seconds"). \ start() as you can noticed, I'm using Queued becase KustoStreaming is not available. Additionally, I'm mitigating my latency issue with flushImmediately : True However, I would like to use direct ingestion to kusto, that is the very reason of existing a new writeMode : KustoStreaming Best regardsSolved648Views0likes5CommentsDeserialization error
Hello, I want to send one plane JSON with all the relevant sensor regularly. My Controller seems to cut off the JSON after a certain length. One way to solve this would be to generate the JSON dynamically or split them. Fabric would then get different JSON with different sensor types from one device. Is that a problem or can an Event Stream and a KQL database manage that? best regards, MoritzSolved119Views0likes2CommentsIssue Connection Azrue IoT Hub to Fabric via Event Stream
Hello, I am trying to connect Azure IoT Hub to Fabric via Event Stream. The data preview for the IoT Hub shows nothing which is normal I was told, since the IoT Hub isn't public facing. When I look into the table in the KQL database it is empty and no data is gathered. I tried every possible setting for the event stream. Only thing I can't find the consumer group I generated in IoT Hub in Fabric. But it should also work with standard one called $Default. Has anyone faced similar problems regarding connecting IoT Hub to Fabric? How did you solve it? I appreciate any help or idea. BestSolved433Views0likes6CommentsError message when connecting IoT Hub in Fabric
I get this error in Fabric when connecting an IoT Hub: Resource Name: src-azure-iot-hub | Message: Source 'IoTHubInputAdapter' had 5 occurrences of kind 'InputDeserializerError.InvalidData' between processing times '2026-07-03T06:40:40.4081161Z' and '2026-07-03T06:44:41.9815920Z'. Could not deserialize the input Is an invalid JSON format the reason for this? What are the requirements for a JSON in fabric? Or is the reason for this somewhere else? I appreciate any help. Best regardsSolved356Views0likes4Comments