real-time hub
141 TopicsHigh CU consumption after enabling Workspace Monitoring
Hi everyone, I’m wondering if anyone else has experienced very high CU consumption after enabling Workspace Monitoring in Microsoft Fabric. In my case, I tested it on an F4 capacity, and the additional CU usage was significant enough that the capacity became difficult to use for the actual workloads running in the workspace. I understand that Workspace Monitoring creates and uses Fabric resources for collecting and storing logs, so some additional capacity consumption is expected. However, the overhead I’m seeing feels quite high and it makes no sense to enable this feature I’m interested if there are any answers on following questions: Have you noticed a significant CU increase after enabling Workspace Monitoring? Is the impact especially noticeable on smaller capacities such as F2/F4/F8? Have you found any practical way to reduce the monitoring overhead? Are you using Workspace Monitoring in production on smaller capacities, or do you avoid it because of the CU cost? At the moment, my concern is that on smaller capacities the monitoring workload itself can consume enough capacity to materially affect the workloads it is supposed to help monitor. Would be interested to hear about other people’s experience and whether this is expected behaviour.75Views0likes4Comments- 34Views0likes5Comments
Unable to commit the following item type "Activator"
Hi Fabric Community, I'm running into an issue with a type activator in Fabric and could use some guidance from the community. When attempting to commit the activator to my repo, I received the following error message: "We weren't able to commit the following item: Please check the technical details for more information " I'm not entirely sure what this error means or how to troubleshoot it. If anyone has come across a similar issue with type activators in Fabric or has any insights into what might be causing this, I'd really appreciate your help. Thanks in Advance4.6KViews0likes6CommentsDeep-Dive: Microsoft Fabric Warehouse Internal Architecture (Polaris, V-Order & Lakehouse Tradeoffs)
Hi Fabric Community, I’ve spent the last few weeks researching and writing a complete architectural guide on the internals of the Microsoft Fabric Warehouse, and wanted to share some key design patterns and takeaways with you. When moving from Azure Synapse Dedicated Pools or SQL Server to Fabric, the mental model shift can be challenging—especially regarding serverless execution and open-standard storage. Here is a summary of the 3 most critical findings from my architecture breakdown: 1. The Polaris Engine Mechanics Unlike traditional Dedicated SQL Pools which bind compute to VM instances, Fabric Warehouse runs on the Polaris distributed SQL engine. Query compilation is split into stateless metadata services. Compute nodes are allocated dynamically per query. There is no local page storage; instead, the engine reads and writes open-source Delta Parquet format files directly in OneLake. 2. The Reality of Constraints A common point of confusion is table constraints. Fabric Warehouse allows you to define Primary Keys, Foreign Keys, and Unique Constraints, but they are not enforced during data ingestion. They serve purely as metadata for the query optimizer (to improve join paths). You must handle data validation/enforcement upstream in your Spark/Dataflow pipelines! 3. Direct Lake & Cold Queries Power BI's Direct Lake mode is a game-changer because it bypasses the SQL endpoint and queries the Delta Parquet files directly. However, if the dataset size exceeds the F-SKU memory limits, the query falls back to DirectQuery mode. This fallback forces the engine to run SQL queries on the SQL endpoint, introducing cold-query compilation times. I have published the full 11,000-word engineering handbook covering capacity planning, monitoring DMVs, security setups (RLS/OLS), and a retail case study. If you are planning an enterprise migration or studying for the DP-600 / DP-700 exams, check out the deep dive here: 👉 Read the Microsoft Fabric Warehouse Architecture & Performance Guide Would love to hear how your teams are managing cold query compilation and metadata cache warming on Fabric SQL endpoints. Let’s discuss below!173Views4likes3CommentsUnable to create Eventhouse
I'm an admin on Fabric and am working on the End-to-End Real-time intelligence tutorial. The following screenshot of my Fabric Capacity Metric app is showing statitics with high capacity usage for three workspaces that have been deleted more than 40 hours ago: From Capacity Metric Compute tab: From Health tab: Even more than 40 hours after deleting those 3 workspaces with hig capacity issue, I'm still getting following error when trying to create an Eventhouse: Eventhouse item could not be created because your capacity has reached its maximum limit. Contact the capacity administrator for help. Can someone please help resolve the issue?Solved182Views0likes3CommentsFabric Apps Not in UK
There doesn't seem to be any timeline to rollout Fabric apps in the UK regions; this is a bit of a nightmare for GDPR reasons. It feels like Databricks hosted in the UK with any visualisations or apps we choose would be better. I think it's quite annoying to not have a timeline, and makes it hard to have a full strategy.318Views0likes6CommentsIf you could improve one thing in Microsoft Fabric tomorrow, what would it be?
No platform is perfect-including Microsoft Fabric. While I'm genuinely impressed by how quickly the platform has evolved, there are still areas where I think the experience could become even better. If the Fabric product team gave you one opportunity to improve the platform tomorrow... What would you choose? Would it be: • Better monitoring? • Improved CI/CD? • More governance capabilities? • Better Git integration? • Performance improvements? • Licensing or capacity changes? • Something completely different? I'm interested in hearing opinions from people using Fabric every day. Sometimes the best ideas come from real customer experiences rather than feature announcements. What would be your one improvement?102Views0likes2Commentsazure-kusto-spark enabling writeMode to KustoStreaming is not available
Using documentation as reference https://github.com/Azure/azure-kusto-spark/blob/master/docs/KustoSink.md, kusto connector seems to bring KustoStreaming as posible option for writeMode. I intented to achieve the less latency as possible where I'm computing some results based on streaming input data and sink the output into a kusto table with streaming policy enabled. This KustoStreaming, is not available in the version of the kusto connector available in Microsoft fabric when I use Spark 3.5. However, seems available in Spark 4.0, meaning I can set up but sadly fails in execution time throwing an excepcion I reported https://github.com/Azure/azure-kusto-spark/issues/479. In any case I want to stick to Spark 3.5 and I was wondering how a more modern version of azure-kusto-spark connector (which already have this functionality embedded) can be included in the Spark runtime? Currently I got this in my fabric notebook spark-kusto-synapse-connector_3.5_2.12-1.5.1.jar' This is my code sp_json = json.dumps({ "flushImmediately": True }) query = events_df.writeStream. \ format("com.microsoft.kusto.spark.synapse.datasink.KustoSynapseSinkProvider"). \ option("kustoCluster", kusto_cluster). \ option("kustoDatabase", kusto_db). \ option("kustoTable", events_table). \ option("accessToken", accessToken). \ option("writeMode", 'Queued'). \ option("pollingOnDriver", 'False'). \ option("sparkIngestionPropertiesJson", sp_json). \ option("checkpointLocation", "Files/outage_events_pyspark_eventhouse/_checkpoint") . \ trigger(processingTime= "5 seconds"). \ start() as you can noticed, I'm using Queued becase KustoStreaming is not available. Additionally, I'm mitigating my latency issue with flushImmediately : True However, I would like to use direct ingestion to kusto, that is the very reason of existing a new writeMode : KustoStreaming Best regardsSolved676Views0likes5Comments