Switch workspace from west US to east US
Wondering how to switch WS and all contents from west US to east US. Tried backup to Git repo and restore from there - Does not work synch to a new empty workspace. Original workspace was west US, (contains pipelines, notebook, warehouse) synched WS to git repo, disconnected WS Created new WS east US tried to synch git repo to new empty WS created on east US errored out on every level and nothing got synched. It just sucks with no import export option. Any other suggestions.
Diagnose Fabric Data Warehouse workloads with the SQL DW operations skill (Generally Available)
When a warehouse slows down, the investigation usually starts with more questions than answers. Was there a capacity spike? Did a specific query suddenly become expensive? Are requests failing, being canceled, or simply taking longer than usual? Answering those questions often requires switching between the Fabric Capacity Metrics app, Query Insights, and SQL pool diagnostics while manually correlating time ranges across multiple tools. The SQL DW operations skill brings those investigations into a single workflow and is now generally available. Available through the open-source Microsoft Fabric skills repository, the SQL DW operations skill lets you describe a problem in natural language using a compatible AI coding tool such as GitHub Copilot CLI. The skill runs bounded, read-only diagnostics and returns a structured diagnosis, supporting evidence, recommended actions, and validation steps. What you can do with the SQL DW operations skill The SQL DW operations skill helps you: failure-analysis: Separate failed queries from canceled requests, identify affected workloads, and resolve engine error codes. resource-consumers: Find recurring resource-consuming query patterns, regressions, and changes in execution volume or per-run cost. capacity-metrics-correlation: Connect a Capacity Metrics spike to warehouse activity in the same time window. pool-pressure: Diagnose contention and identify workloads that might benefit from custom SQL pools. lakehouse-health: Find lakehouse tables with small-file, deleted-row, or checkpoint issues. query-reference: Use the appropriate read-only system views and Query Insights queries for bounded operational analysis. scenarios: Combine the diagnostics into guided workflows for common warehouse incidents. Each response separates the diagnosis, evidence, ruled-out causes, recommendations, and customer follow-ups. Measurements are tied to their source, and zero-row results are treated as valid evidence instead of prompting an invented explanation. Common use cases Start with the operational question rather than selecting system views or writing diagnostic SQL. Investigate failed and canceled queries Analyze failed and canceled queries in SalesWarehouse during the last 24 hours. The skill uses Query Insights to identify affected users, applications, query patterns, and SQL pools. It resolves failed engine codes through sys.messages and keeps cancellations separate because they can reflect a user-initiated cancellation, a client timeout, or resource pressure. Explain a performance slowdown Explain why FinanceWarehouse was slow between 09:00 and 11:00 UTC yesterday. The skill checks SQL pool pressure, overlapping requests, CPU, elapsed time, and storage scans. It distinguishes contention from a directly expensive query or a broad increase in workload. Find resource-consuming query patterns Find the top resource-consuming queries in SalesWarehouse and compare them with the previous seven days. The skill groups requests by query shape and separates higher execution volume from increased per-run cost, new query patterns, and one-time expensive runs. Investigate a capacity spike Use the Fabric Capacity Metrics app to investigate the CU spike from 14:00 to 15:00 UTC, then identify expensive SQL users and query patterns. Following the warehouse metering update introduced in August 2026, Capacity Metrics shows when consumption occurred and how much was reported based on allocated warehouse compute over time. Query Insights explains what ran during the same period. The skill discovers the installed Capacity Metrics model, identifies a costly warehouse or SQL analytics endpoint, and analyzes Query Insights requests that overlap its time window. It doesn't join Capacity Metrics operation identifiers to Query Insights statement identifiers. Capacity consumption and warehouse CPU are complementary signals, not interchangeable measurements. Assess custom SQL pool candidates Assess whether recurring workloads in SalesWarehouse are candidates for custom SQL pools based on the last 30 days. If repeated pressure is associated with a consistent application name, such as an ingestion service or reporting application, the skill can recommend testing that workload in a custom SQL pool. It identifies the application to isolate and the pressure, latency, CPU, scan, and failure measures to compare before and after the pilot. Get started Prerequisites Before you start, make sure you have: GitHub Copilot CLI or another compatible AI coding tool. An active Fabric warehouse or lakehouse SQL analytics endpoint. Contributor or higher access to the workspace. The Microsoft Fabric Capacity Metrics app installed for capacity-spike investigations. Add the Microsoft Fabric skills marketplace in GitHub Copilot CLI: /plugin marketplace add microsoft/skills-for-fabric Install the Fabric skills bundle: /plugin install fabric-skills@fabric-collection Then open Copilot CLI in a project folder and describe the warehouse issue you want to investigate. Include the workspace, warehouse or SQL analytics endpoint, and UTC time range when possible. For detailed permissions, setup, supported scenarios, and diagnostic time limits, see Diagnose warehouse workloads with the SQL DW operations skill. Next Steps Install or update the Microsoft Fabric skills bundle. Review Query Insights in Fabric Data Warehouse. Review monitoring options for Fabric Data Warehouse. Learn about custom SQL pools.Schema Compare in VS Code: Simplifying Fabric Warehouse Deployments
Deploying databases shouldn't feel uncertain. Synchronizing development and production schemas in Microsoft Fabric Warehouse can be challenging as database objects evolve. A frequent issue arises when the .sqlproj file isn't configured with the Fabric-specific schema provider (SqlDbFabricDatabaseSchemaProvider). Misconfiguration may lead Schema Compare to flag supported Fabric objects as unsupported. To avoid this issue, explicitly specify the schema provider in the .sqlproj XML if the GUI does not offer that option. In this blog, let's take a quick look at how to get started. Before using Schema Compare with Fabric Warehouse, make sure you have the following prerequisites. You need access to an existing Warehouse item within a Microsoft Fabric workspace, with Contributor or higher permissions. You also need Visual Studio Code installed on your workstation. Next, install the .NET SDK, which is required to build and publish database projects. Finally, install these two Visual Studio Code extensions: SQL Database Projects SQL Server (mssql) Both extensions are available directly from the Visual Studio Code Marketplace. After installing the extensions, open Visual Studio Code and select "Add Connection" to connect to Fabric Warehouse. Enter the required server and authentication details, and complete the connection. After connecting, access the Warehouse directly in Visual Studio Code. Open Schema Compare Now navigate to Database Projects in Visual Studio Code. You should see your available database projects and connections. Right-click the database project or connection and select "Schema Compare." Schema Compare gives you an object-level view of the differences between the source and target. You can compare schemas between: .dacpac files Databases SQL database projects Comparison results outline actions to align target with source. Instead of treating the database as a single deployment unit, you can review individual changes and decide what should happen next. You can also selectively exclude actions from the comparison results when a particular change should not be deployed. Schema Compare with Fabric Warehouse Schema Compare's integration with Fabric Warehouse allows developers to identify differences in database objects before implementing changes. Database Projects and Git The database project becomes the artifact under review in the PR, not a raw DDL diff — schema changes get the same scrutiny as application code, and the deployment pipeline consumes a validated project state rather than an ad-hoc script. Schema Compare exposes Fabric Warehouse DDL limitations before deployment, helping you proactively address issues and maintain greater control over the deployment process.Fabric July 2026 Feature Summary
Welcome to the July 2026 Fabric update! This month brings new capabilities across the Fabric experience, from improved deployment and governance experiences to expanded Spark, Eventstream, and Real-Time Intelligence functionality. Whether you're building data pipelines, managing analytics workloads, or monitoring real-time operations, these updates help you work more efficiently and get more value from your data. _______________________________ Events and Announcements Get Fabric certified for FREE This is your chance to take the DP-600 (Fabric Analytics Engineer) or DP-700 (Fabric Data Engineer) certification exams for free. As part of Data Days, we have over 100 live sessions, more than 5 contests and challenges, and dozens of study groups and learning opportunities. And free Fabric exam vouchers. Available now through August 10, 2026. Request your voucher. Join us for FABCON and SQLCON in Barcelona, September 28 – October 1, 2026 Explore what’s possible with Microsoft Fabric and get up to speed on the latest in SQL, analytics, and AI. From 130 sessions and 4 keynotes to workshops, the expo, community spaces, and the Power BI DataViz World Championships, this is where the data community comes together. Learn directly from Microsoft and community experts shaping the future of Fabric and SQL. Register now and save €200 with code FABCMTY200. Fabric Platform Fabric-CICD tool v1.2.0 – new bulk publish mode (Preview) The June release of the fabric-cicd Python library, v1.2.0 introduces bulk publish mode. It lets fabric-cicd publish multiple items in a single API call using the Fabric bulk import API instead of publishing each item individually through a separate API call. This can make deployments more efficient. Why this matters Because dependencies are managed by the API during publication, bpublishinglish reduces the rigidity of item-type-based staging and better supports cross-item dependencies when logical ID references are used. It can reduce the number of item-specific parameter values you need to configure. For unsupported scenarios, fabric-cicd automatically falls back to the standard publishing flow, so you can try bulk publishing without manually switching deployment paths. To learn more, refer to fabric-cicd bulk option documentation and the v1.2.0 release notes. Change Git branch with at least the contributor role (Preview) Fabric Git integration now let’s any workspace member with at least the Contributor role switch the workspace's connected Git branch. Previously, this action required the workspace Admin role, which forced developers to either be over-permissioned or wait on an admin every time they moved between branches. Why this matters Developers no longer need Admin rights to switch branches. Removes a key blocker in the branch-out to existing workspace flow. Keeps developers as Contributors, aligned with least privilege governance. Fewer hand-offs to admins mean faster time to code. You can find this new capability under the Git integration settings. To learn more about Fabric Git integration new setting, refer to the Allow Contributors and Members to switch branches documentation. We refreshed the Fabric CI/CD documentation this month to make it easier to get started and to follow best practices: New CI/CD intro page — a reworked Introduction to CI/CD in Microsoft Fabric that walks through the platform layer by layer with a new enterprise reference architecture. New best practices guide — Understand the best practices for Fabric CI/CD brings together practical guidance for structuring workspaces, branching, and promoting content safely across environments. Auto-bind for Git integration — new guidance on cross-workspace dependency binding, covering how item dependencies automatically rebind when you branch out or update from Git, and which dependency types are supported. Actionable recommended actions for data owners in the OneLake catalog The Govern tab in the OneLake catalog gives data owners a health view of their data estate, along with recommended actions to improve it, such as increasing sensitivity label coverage, removing items that are no longer in use, handle failed refresh, and more. Until now, these recommendations have told data owners what to improve, but not which items were affected, leaving them to track the relevant entities down manually before they could act. That gap turned a clear recommendation into a manual investigation, and the action often stalled before it started. This release closes that gap. When you open a recommended action, you now see the specific items behind the recommendation, with everything you need to act on them in one place, and a direct path to open item details and resolve the issue at the source. Selecting a recommended action now opens a dedicated view that includes: A breakdown of the affected items — a visual summary of how many of your items are impacted. Why it matters — a short explanation of the governance impact. How to fix it — clear, step-by-step instructions for resolving the recommendation. Newly added A complete list of affected items — displayed in a table with key details, including Name, Last Refreshed, Owner, Location, Endorsement, and Sensitivity, making it easy to review and prioritize actions. Filter and search — narrow the list by keyword or filters to focus on the items you want to handle first. Open item details — jump straight to any affected item to make the change, instead of searching for it across your workspaces. The same enriched experience applies across the recommended actions in OneLake catalog Govern, helping you act on each one without leaving the catalog: Increase Sensitivity label coverage — find and label items that create potential security risks while unlabeled. Remove unused items — review items that weren't accessed or refreshed recently to keep your estate organized and reduce costs. Investigate items that failed to refresh — see which items failed to refresh so your data stays current and reliable. Add descriptions to your endorsed items — surface endorsed items without a description so people can better understand and use them. Apply relevant tags to your items — make items more discoverable by tagging the ones that are missing tags. Why it matters Governance only improves when recommendations turn into action. By bringing the affected items and a direct way to reach them into every recommended action, OneLake catalog Govern removes the guesswork from following through. Data owners can move from understanding a recommendation to resolving it in just a few steps, keeping their data estate secure, organized, discoverable, and trustworthy with far less effort. Learn more about the OneLake catalog Govern tab recommended actions. Data Engineering Microsoft Fabric Runtime 2.0 (Preview) Based on feedback received directly from our customers and partners, we have upgraded Fabric Runtime 2.0 to the latest and compatible stack: Apache Spark 4.1 Delta Lake 4.2 Python 3.13 These upgrades bring access to the newest enhancements, performance improvements, and ecosystem innovations, while also providing a longer support window for enterprise customers to build large-scale analytics and AI workloads on Fabric. With Runtime 2.0, customers can take advantage of: Latest Spark and Delta Lake capabilities. Improved compatibility across the modern data ecosystem. Better developer productivity and runtime performance. Future-ready platform investments aligned with long-term supportability. Learn more about Fabric Runtime 2.0. Fabric Runtime Release Channels (Preview) Fabric Runtime Release Channels provide a structured and transparent way for customers to test upcoming runtime changes before they become the default. This feature helps organizations validate their production workloads early with these new changes in early access, avoid unexpected disruptions, and gain better control over Spark runtime upgrades. Instead of receiving silent updates that might break your production workloads, you can opt in to an early access release channel, test your workloads in a development or staging environment, and confirm compatibility before the update becomes default. How release channels work Each Spark runtime has at least two public release channels: Default channel — This production-grade channel runs the default version of the runtime. All users automatically use this channel unless they opt in to early access. Early access channel — This production-grade channel includes upcoming updates and library changes that are scheduled to become the next default channel. You can opt in to test your workloads against upcoming changes. Once the designated validation window ends, the early access release channel automatically gets promoted to become the new default, and a fresh early access channel is introduced with another set of new changes — continuing the cycle. This model gives you a predictable testing window before changes become default for everyone. Why release channels matter Spark runtime updates can include library upgrades, security patches, dependency changes, or even operating system upgrades. While all updates pass internal quality checks before release, those checks can't capture all customer-specific variations and use cases. Early access channels let you identify potential issues early and work with Microsoft by creating a support ticket to address them before updates affect your production environment. Benefit Description ✔ Predictable updates Customers know exactly when a new runtime becomes available and have time to validate against it. ✔ Reduced risk Testing workloads on early access ensures compatibility before changes reach production. ✔ Better visibility Customers can easily tell which runtime version they're running, reference release notes, and verify upgrade timing. ✔ Improved quality and security You receive well-tested builds with security patches applied faster, giving you confidence in runtime stability. Learn more about Fabric Runtime Release Channels: Fabric Runtime Release Channels. Spark Diagnostic Emitter: Spark 4.1 Runtime Support and New Log Ingestion API The Fabric Apache Spark Diagnostic Emitter is now supported on the Fabric Runtime with Apache Spark 4.1. Customers can collect driver logs, executor logs, Spark event logs, and metrics from workloads running on the latest runtime and route them to Azure Event Hubs, Azure Blob Storage, or Azure Log Analytics — using the same spark.synapse.diagnostic.emitter.* configuration model, so existing emitter setups carry forward as customers upgrade. The emitter also now supports the Azure Monitor Log Ingestion API for sending diagnostics to Log Analytics, available on both Spark 3.5 and Spark 4.1 runtimes. The new AzureLogIngestion emitter type replaces the legacy HTTP Data Collector API path, providing a structured ingestion model with DCR/DCE-based authentication, schema definition, and routing into custom Log Analytics tables. Customers currently on the legacy AzureLogAnalytics type are encouraged to migrate — migration involves creating Data Collection Rule and Data Collection Endpoint resources and updating the Spark properties in the Fabric Environment. To learn more, refer to the Collect logs and metrics with Azure Log Analytics for migration guidance and Spark Diagnostic Emitter documentation. The Fabric Spark Operations Skill — AI-Assisted Spark Diagnostics, Open Source (Preview) The Fabric Spark Operations Skill (spark-operations-cli) is now available in the open-source Skills for Fabric library on GitHub (microsoft/skills-for-fabric). The skill brings AI-assisted, read-only diagnostics to Spark workloads in Fabric — troubleshoot failed notebooks and Spark jobs, stuck Livy sessions, and performance bottlenecks (OOM, shuffle, skew) in plain English from GitHub Copilot CLI, Claude Code, VS Code, Cursor, or other compatible AI tools. It returns severity-ranked findings with root cause analysis and fix recommendations, and includes an automated diagnostic workflow spanning job triage, log mining, Spark Advisor findings, and mitigations. Setup takes minutes: install the skill, run az login, and ask, "Why did my notebook fail last night?" Get started with Skills for Fabric in GitHub. Faster Python UDFs, Scala UDFs, and complex data types in the native execution engine (Generally Available) The native execution engine in Microsoft Fabric, which is generally available, now accelerates Python user-defined functions (UDFs), Scala UDFs, and complex data types such as arrays, maps, and structs. You get faster Spark processing for expressive code without changing your existing notebooks or jobs. Python UDFs have historically carried serialization overhead as data moves between the JVM and Python worker processes. The native execution engine optimizes that data path and keeps vectorized processing intact, so scalar and Pandas (vectorized) UDFs run faster automatically. Scala UDFs and queries that work with nested, complex data types benefit from the same native acceleration. Why this matters Faster UDF execution with no code changes. Vectorized Pandas UDFs see the largest gains. Complex types (arrays, maps, structs) run natively. Existing notebooks and jobs benefit automatically. To use Efficient Scaledown, enable the native execution engine on your Fabric Spark pool or environment. Existing Python and Scala UDFs and queries over complex types are accelerated without any code changes. To learn more, refer to the Python UDFs, Scala UDFs, and complex data types in the native execution engine documentation. Efficient Scaledown with the remote shuffle manager (Preview) Efficient Scaledown decouples Spark shuffle data from executor lifetime in Microsoft Fabric. Instead of pinning shuffle output to local executor disks, Fabric Spark routes large shuffles to Azure Blob Storage and migrates blocks off executors before they're released. Clusters scale down faster, compute costs drop, and jobs become more resilient — with no changes to your queries, notebooks, or pipelines. The feature combines four cooperating capabilities: the Remote Shuffle Manager writes and reads shuffle data to Azure Blob Storage; Shuffle Migration moves blocks off an executor before decommissioning instead of dropping them; a Decision Layer routes small shuffles to local disk and large shuffles to remote storage per stage; and AQE Shuffle Write lets Adaptive Query Execution shape partitioning the first time. Why this matters Clusters scale down faster after demand drops. Lower compute cost from quicker executor release. More resilient jobs with fewer stage retries. No changes to queries, notebooks, or pipelines. To use it, enable the native execution engine and run on Runtime 1.3 (Apache Spark 3.5) or later; autoscale is recommended. Remote Shuffle Manager spark.conf.set("spark.remote.shuffle.enabled", "true") Decision Layer — per-stage routing of local vs. remote shuffle spark.conf.set("spark.sql.rsm.decisionlayer.enabled.level", "stage") AQE participates in shuffle write spark.conf.set("spark.sql.adaptive.shuffleWrite.enabled", "true") Shuffle Migration on executor decommission spark.conf.set("spark.storage.decommission.shuffleBlocks.enabled", "true") spark.conf.set("spark.storage.decommission.shuffleBlocks.cleanup", "true") spark.conf.set("spark.storage.decommission.shuffleBlocks.migrateToFallbackStorage", "true") spark.conf.set("spark.storage.decommission.fallbackStorage.cleanUp", "true") To learn more, refer to the Efficient Scaledown and remote shuffle manager in Microsoft Fabric documentation. Customer-managed key encryption for Spark shuffle data on disk (Generally Available) Microsoft Fabric Spark now generally supports customer-managed keys for Spark jobs through a disk encryption set. Shuffle data written to cluster disks during a Spark job is encrypted with a key you supply and control, giving you ownership of the encryption material that protects intermediate data at rest. With a disk encryption set configured, the cluster disks that hold Spark shuffle data are encrypted using your customer-managed key rather than a platform-managed key. You manage the key lifecycle — including rotation and access — in your own key vault, so encryption of intermediate Spark data follows your organization's key management policies. Why this matters You control the key protecting shuffle data. Intermediate Spark data on disk is encrypted at rest. Key lifecycle and rotation stay in your control. Encryption aligns with your key management policies. To use it, configure a disk encryption set backed by your customer-managed key and associate it with your Fabric Spark configuration. Spark jobs then encrypt shuffle data on cluster disks with your key. To learn more, refer to the Customer-managed key encryption for Fabric Spark documentation. Query your data instantly with the Lakehouse Query Explorer (Generally Available) The Lakehouse Query Explorer is a new, fully integrated query editor built directly into the Lakehouse experience in Microsoft Fabric. You can now write and run Spark SQL queries right where your data lives — no need to switch to a SQL endpoint or spin up a notebook for quick exploration. Whether you’re validating datasets, iterating logic, or exploring patterns, Query Explorer keeps you in flow. orer with the new integrated Query Explorer. Key capabilities Fast, lightweight Spark execution powered by the Lakehouse Livy endpoint. IntelliSense + rich editing experience for faster query authoring. Query across schemas and lakehouses in a single tab. Built-in results grid + inline charts to explore results instantly. Multiple dynamic tabs to analyze different slices of data side-by-side. From quick lookups to multi-table exploration, Query Explorer makes working with Lakehouse data faster and more intuitive—right from the explorer. Learn more and get started with the Lakehouse Query Explorer documentation. Analytics and Insights for Materialized lake views (Generally Available) Analytics and Insights bring continuous refresh intelligence to your lakehouse — two new tabs beside Recent run(s) that shift you from reacting to individual failures to staying ahead of performance drift, rising costs, and silent inefficiencies across your entire materialized lake view estate. The Recent run(s) page tells you what happened in a single execution, including which views succeeded, which failed, and how long the job took. That's valuable when something breaks, but it doesn't answer the questions that matter most for day-to-day operations. Are my durations stable or slowly climbing? Is a new error class appearing and spreading across views? Are my schedules still aligned with how often upstream data actually lands? Am I paying for full refreshes on views that could run incrementally? These are the questions that separate a well-tuned deployment from one that quietly accumulates cost and risk until something finally breaks loudly enough to notice. The Recent run(s) page answers all of them. The Analytics tab transforms your run history into trend lines, distributions, and comparisons you can read at a glance: duration trajectories, success-rate shifts, error-class frequency over time. The Insights tab goes further by watching those same patterns the way a seasoned reliability engineer would, recognizing the signatures behind slow, failing, or wasteful runs, and providing a prioritized list of worthwhile changes. Each recommendation names the affected view, explains why it's flagged, and estimates the payoff in runtime savings, so you can act in seconds rather than investigate for hours. Together, they move refresh management from a reactive, break-fix posture to a proactive, continuously improving one — keeping your materialized lake views fast, healthy, and cost-efficient as your estate grows. To learn more, refer to analytics charts, insight categories in Materialized Lake Views. Introducing Event-Driven Refresh for Materialized Lake Views (Preview) Event-driven refresh brings responsive refresh intelligence to your lakehouse — a new scheduling mode alongside time-based schedules that shifts you from refreshing on the clock to refreshing the moment your data is ready, so your materialized lake views reflect reality instead of an arbitrary calendar. Time-based schedules tell your views when to run: every hour, every morning, every night. That's dependable when upstream data lands like clockwork, but it doesn't answer the questions that matter most for day-to-day operations. Did the ingestion pipeline that feeds this view finish before I refreshed it? Am I recomputing gold-layer views on a fixed cadence while the source data only changes twice a day? Am I paying for refreshes that run before new data has even arrived — or worse, serving stale results because the next scheduled slot is still hours away? These are the questions that separate a refresh strategy tuned to your data from one that quietly burns compute on empty runs and lags the moments that matter. Event-driven refresh answers all of them. You bind a view or a lineage sub-chain to the events that should drive it, and we support two types of events: OneLake events — all file and folder events are supported, so a refresh can fire the moment data lands in OneLake (file or folder creation, update, and more). Job events — Pipeline and Notebook events are supported, so a refresh can fire when the ingestion job that feeds your views completes. The moment an event fires, Fabric resolves the dependency chain and refreshes exactly the views that depend on it. Each trigger names the source event, scopes precisely to the affected lineage, and can react to success or failure, so a stalled upstream job never silently cascades into stale downstream reports. Together with multi-schedule support, event-driven refresh moves refresh management from a fixed-clock, guess-the-cadence posture to a responsive, data-driven one — keeping your materialized lake views fresh the instant new data arrives, and idle when it hasn't, as your estate grows. Learn more and get started with the Schedule a Materialized Lake View Refresh documentation. Data Science AI functions: new models, no package dependency, better usage stats (Generally Available) Fabric AI Functions now use gpt-5-mini as the default model, with “low” reasoning enabled. This powers AI Functions across pandas, PySpark, Data Warehouse, and Dataflows Gen2. For more sophisticated transformations, users may configure gpt-5.1 or tune the reasoning_effort parameter for additional compute and higher-quality results. The gpt-4.1 model has been retired. Pipelines pinned to gpt-4.1 have migrated to gpt-5.1, and those pinned to gpt-4.1-mini migrated to gpt-5-mini. We’ve also simplified PySpark AI Function chaining. The PySpark .ai interface now stays bound to the result schema, so chains like summarize → classify no longer require intermediate DataFrames. In addition, PySpark now supports df.ai.stats for detailed token usage after any AI function call, including reasoning token breakdowns. For pandas, AI Functions no longer require the openai-python package. Capacity-limited rows are surfaced as CapacityExceededResult, enabling clean retries via aifunc.split_results. To learn more, refer to the AI Functions documentation. Data Warehouse Lakehouse table health check (Generally Available) Lakehouse Table Health Check gives you a simple, T-SQL–based way to validate whether your Lakehouse tables are optimized for the SQL analytics endpoint. A single stored procedure surfaces common layout issues, such as small files and fragmentation, offering the insights you need to determine if your tables need to be optimized. You can integrate health checks into pipelines and operational workflows, enabling proactive, at-scale optimization instead of reactive troubleshooting. -- Run a health check on a Lakehouse table from the SQL analytics endpoint EXEC sp_get_table_health_metrics 'dbo.FactSales'; aluate table health. To learn more about sp_get_table_health_metrics and how to integrate it into your Pipelines to optimize your table only if anomalies are detected, refer to sys.sp_get_table_health_metrics (Transact-SQL). Scalar User‑Defined Functions - Procedural computation for analytical SQL (Preview) Scalar User-Defined Functions now support procedural computation — including loops, multiple return paths, and rich IF/THEN/ELSE branching — running natively within the warehouse engine. Computation-based Scalar UDFs are designed for analytical query shapes, integrating naturally with CTEs, GROUP BY, HAVING, and ORDER BY, and executing efficiently at data warehouse scale. Defining business rules once, in SQL, and reuse them across queries, reports, and pipelines. To learn more, refer to the Create Function documentation. Usage-based resource estimations (Generally Available) The Query Optimizer uses learned resource estimation to correct underestimation in T-SQL query plans. It saves actual cardinalities from past executions and automatically adjusts row count estimates in subsequent runs. With this release, the optimizer will also begin correcting overestimations — closing the loop in accurate cardinalities, leading to more efficient resource requests and improved concurrency. Real-Time Intelligence Improved tile error experience in Real-Time Dashboard (Generally Available) We improved the way tile errors appear in Real-Time Dashboards to make them clearer, calmer, and easier to act on. Instead of showing a disruptive red error state, tiles now use a neutral grey error state with a short category header, such as Syntax error, Semantic error, Data source issue, Network error, or Something went wrong. Users can select Details directly from the tile to open a popover with the full engine error message, shown exactly as received. This keeps the dashboard readable while still making the technical details easy to access, copy, and share when needed. This update helps users understand what went wrong faster, while ensuring that one failed tile does not block or visually overwhelm the rest of the dashboard. To learn more, refer to the Troubleshoot Real-Time Dashboard Tile Errors documentation. Folders in Eventhouse tree (Generally Available) Folder support in the Eventhouse tree is now available through the UI. Previously, folders could only be created and managed via code. With this update, you can now organize your Eventhouse directly from the tree, making it easier to manage assets at scale. You can group tables, shortcuts, materialized views, functions, and data streams into a structured hierarchy, improving navigation and reducing clutter. Key capabilities Create, rename, and delete folders from the UI. Organize assets into folders. Move items via the context menu (⋯ → Move to). Create folders inline while moving items. This allows for a simplified and more intuitive way of managing growing Eventhouses. To learn more, refer to the Manage and monitor a KQL database table documentation. Eventstream connector private network support (Generally Available) Data is a critical asset for organizations, and access to real-time data is increasingly essential. However, many high-value data sources reside in private network environments — cloud virtual networks or on-premises infrastructure — particularly in highly regulated industries such as banking, finance, and telecommunications, where strict security and compliance requirements are mandatory. Eventstream's private network support establishes a secure, managed bridge using your Azure virtual network and VNet injection, allowing Eventstream streaming connectors to run inside your virtual network and reach private sources without opening them to the public internet. Whether your data sources reside in on-premises networks, private networks on third-party cloud services, or private networks on Azure, you can connect the bridge Azure virtual network to your source's private network using suitable connectivity options — such as VPN or ExpressRoute for on-premises environments, and private endpoints or network peering for Azure-based sources — enabling Eventstream connectors to securely ingest real-time data from these protected environments into Fabric. The solution leverages a new concept — Streaming virtual network data gateways — which abstracts the bridge Azure virtual network and subnet resource within Fabric. By creating a connector with a streaming virtual network data gateway associating it with your connection, the Eventstream connector is provisioned within your virtual network, ensuring secure communication with your private data sources. Once real-time data from your private network source is securely brought into Fabric Eventstream, you can fully leverage the comprehensive analytics tools in Fabric Real-Time Intelligence to power your real-time scenarios with enterprise-grade security. To learn more about configuration and advanced scenarios, refer to the Eventstream private network streaming guide. Azure Event Hubs source in Eventstream now supports workspace identity authentication (Preview) Currently, when configuring an Azure Event Hubs source in Eventstream, the only supported authentication method is Shared Access Key — a connection string containing static credentials. While simple to set up, shared access keys present several security risks in production environments: they have unlimited lifetime unless manually rotated, provide no per-user or per-application identity, and if accidentally leaked through source code, configuration files, or third-party sharing, grant full access to anyone who obtains them. Revoking a compromised key requires regeneration, which disrupts all services depending on it — with no straightforward way to audit which clients used the key. To address these challenges, we're introducing Workspace Identity as a new authentication option for the Azure Event Hubs source connector (Extended features) in Eventstream (Preview). A Fabric workspace identity is an automatically managed service principal associated with your workspace. Fabric manages the credentials entirely — there are no secrets to store, rotate, or risk leaking. It integrates with Microsoft Entra ID, providing identity-based access with full audit trails, fine-grained role-based access control, and automatic credential lifecycle management. To use workspace identity authentication with your Azure Event Hubs source in Eventstream: Navigate to your workspace settings and create a workspace identity on the Workspace identity tab. In your Azure Event Hub namespace, assign the appropriate role (e.g., Azure Event Hubs Data Receiver) to the workspace identity's service principal. When adding an Azure Event Hubs source in Eventstream, select Workspace Identity as the authentication method — no connection string or key is needed. Eventstream will automatically obtain tokens using the workspace identity to securely connect to your Event Hub, eliminating credential management overhead while strengthening your security posture. To learn more about the configuration, refer to the Azure Event Hubs source extended connector configuration documentation. Custom CA and mTLS support in Eventstream streaming connectors (Generally Available) Fabric Eventstream under Real-Time Intelligence provides various streaming connectors, enabling the integration of real-time data from popular sources into Fabric. When the Eventstream connector client establishes a connection with sources, it is required to implement TLS or mTLS encryption to fulfil the necessary security standards. Many organizations use certificates issued by private or internal Certificate Authorities or require mutual TLS (mTLS) authentication where both the client and server verify each other's identity before transmitting data. Without custom CA and mTLS support, Eventstream connectors cannot connect to these secured source systems. The Custom CA and mTLS support feature, is now generally available for MQTT, Apache Kafka, AWS MSK, and Confluent Cloud for Apache Kafka source connectors. Customers can specify their custom CA and client certificates managed in their own Azure Key Vault when configuring their source in Eventstream. Once specified, Eventstream connector will fetch the certificates from the customer's Azure Key Vault and use them to establish a mutually authenticated, encrypted connection — enabling secure, compliant real-time data ingestion across all supported streaming sources. To learn more about the configuration, refer to the Eventstream sources overview page and choose the corresponding source. Introducing the Oracle CDC connector for Eventstream (Preview) Eventstream now introduces the Oracle Database Change Data Capture (CDC) connector, enabling you to stream database change events directly from any Oracle Database — whether running in the cloud or on-premises — into Eventstream for real-time processing and analytics. Many organizations run important operational workloads on Oracle Database and need to react to changes as they happen. With the Oracle CDC connector, you can continuously capture change events from Oracle Database and bring them into Fabric without building custom polling applications or managing separate integration services. With the Oracle CDC connector, you can: Capture and stream databases changes from Oracle Database into Fabric in real time. Connect to Oracle databases running either on-premises or in the cloud. Process incoming change events using Eventstream transformations. Route processed change events to supported destinations such as Eventhouse, Lakehouse, Activator, or custom endpoints. This capability helps you build real-time analytics and event-driven applications from Oracle data. For example, you can route transaction changes to an Eventhouse Kusto table for operational analysis, send selected events to Activator for alerting, or combine Oracle change events with other streaming sources in the same eventstream. To learn more about Eventstream Oracle CDC connector, refer to Add Oracle Database CDC source to an eventstream (preview). Eventhouse update policies now support referencing accelerated shortcuts in update policies (Generally Available) Eventhouse update policies now support accelerated shortcuts in update policy queries, enabling ingestion-time enrichment scenarios. Use this for dimension lookups, such as enriching ingested fact events with customer, device, or product attributes stored in OneLake shortcut data. The shortcut-backed external table must have Query Acceleration Policy enabled, and Hot must cover all data. For update policy scenarios, set: .alter external table DimCustomer policy query_acceleration '{"IsEnabled":true,"Hot":"36500.00:00:00"}' Then join to it from the update policy query: .alter table EnrichedEvents policy update '[{ "IsEnabled":true, "Source":"RawEvents", "Query":"RawEvents | lookup kind=leftouter (external_table(''DimCustomer'')) on CustomerId","IsTransactional":true, "PropagateIngestionProperties":false }]' Processing uses the authorization context captured in the system-populated OwnerPrincipalDetails property: the user who creates or alters the update policy must have access to the shortcut data. This enables ingestion-time enrichment with governed shortcut data without separate orchestration. Shortcut Tables in Eventhouse Now Automatically Synchronize Schema Changes To help maintain consistency between source data and shortcut tables, the default schema synchronization behavior for Eventhouse shortcut tables is changing. Previously, schema changes made to the source table were not automatically propagated to the shortcut table unless schema synchronization was explicitly enabled. As a result, source and shortcut schemas could diverge over time. With this update, all new and existing shortcut tables in Eventhouse automatically synchronize schema changes from their source table by default, including: Adding new columns Changing column data types Renaming columns Deleting columns Automatic schema synchronization helps ensure that shortcut tables remain aligned with the source schema, preserving the latest business context and reducing manual maintenance. Customers who prefer to disable automatic schema synchronization can continue to control this behavior using the existing KQL management command: .create-or-alter external table ExternalTable kind=delta ( h@'https://storageaccount.blob.core.windows.net/container1;secretKey' ) with (AutoUpdateSchema=false) Note: Because schema changes are now automatically propagated, queries, dashboards, and downstream workloads may require updates if they reference columns that are renamed, removed, or otherwise modified in the source table. Investigator Insights in Operations Agent (Preview) When an anomaly is detected, understanding what caused it is often the hardest part. Investigator insights are designed to make that easier by analyzing the surrounding data and surfacing relevant context. When the operations agent detects that a rule has been met, there is an option to run an investigation in the background to identify correlated signals and patterns. Instead of manually digging through telemetry, you get a guided view into what changed, what stood out, and what may have contributed to the issue. This helps you move more quickly from detection to understanding. You can access these insights directly from Teams. Open the agent’s message and select Investigate further to generate a detailed analysis. The investigation provides a structured view of what happened: Investigation scope shows which tables were analyzed, whether from a single source or across related datasets. Key observations highlight the most important findings, including actual values, deviations from baseline, and notable trends or outliers. Pattern analysis surfaces meaningful changes around the time of the anomaly, such as dimensions with significant shifts, and clearly call out when no strong patterns are identified. Together, these insights help you quickly understand not just that something went wrong, but why it happened. To learn more, refer to the Operations Agent Actions documentation. Anomaly Detector Configurations Pane As you build on top of your data, it is often just as important to understand what already exists as it is to create something new. This update makes it easier to discover and build existing anomaly detection configurations. You can now view all anomaly detection configurations that have already been created for a given data source in one place. This lightweight experience gives you quick visibility into how anomaly detection is currently set up, helping you avoid duplicate work and better understand how others are using the data. From this view, you can explore existing configurations or create a new one if your use case is not yet covered. This makes it simple to extend existing setups or start fresh when needed, all without leaving the context of your data source. By making configurations easier to discover and reuse, this experience helps streamline workflows and ensures you can move quickly from exploration to action. To learn more, refer to the Anomaly Detection in Real-Time Intelligence documentation. Ingestion time stamp in Anomaly Detector (Preview) Working with anomaly detection often assumes your data already includes a clean, reliable timestamp. With this update, you can now use the system-generated ingestion time as the timestamp for anomaly detection. This means that even if your dataset does not include a dedicated timestamp column, you can still run analysis without needing to modify or preprocess your data. This is especially helpful for scenarios where events are ingested in real time or where timestamps are missing, inconsistent, or not trustworthy. Instead of blocking data preparation, you can rely on ingestion time to move forward with detection and start identifying meaningful patterns right away. By reducing setup requirements and removing a common dependency on source data quality, this capability makes it easier to apply anomaly detection across a wider range of use cases. To learn more, refer to the Anomaly Detection in Real-Time Intelligence documentation. Configuring Anomaly Detection without a Group by Column (Preview) With this update, Anomaly Detection now supports scenarios where you choose not to use a group by column. This provides an additional configuration option for datasets that already represent a single stream of data, allowing you to apply anomaly detection directly to the metric without first identifying a grouping dimension. This added flexibility helps the configuration experience better align with how your data is structured. Whether you are monitoring a single device, tracking a specific service, or analyzing a focused dataset, you can now create anomaly detectors without requiring a group-by column. At the same time, grouping remains available for scenarios where you want to monitor and compare multiple entities within the same dataset. To learn more, refer to the Anomaly Detection in Real-Time Intelligence documentation. Introducing Fabric Maps Tilesets: High-Performance Visualization for Large Geospatial Datasets (Generally Available) Have location-based data sitting in OneLake but no simple way to see it come alive on a map? With Fabric Maps Tilesets, you can now turn that data into fast, interactive map experiences directly inside Microsoft Fabric. The Tileset Builder lets you create map-ready PMTiles from OneLake data without custom code, manual exports, or separate geospatial infrastructure making it easier for teams to explore large geospatial datasets, uncover patterns, and bring location intelligence into the analytics workflows they already use. Organizations can rely on Microsoft Fabric to manage operational data, analytics workflows, and business reporting. When that data includes locations, routes, assets, boundaries, or events with a location context, teams need a simple way to visualize it on a map. Traditionally, this required separate geospatial pipelines, custom polling services, or manual exports. Leveraging a Fabric Map removes complexity by allowing organizations to create and refresh map-ready tilesets from data already stored in OneLake. Try creating your own Tileset using the following steps: Connect to a lakehouse and select source files Configure tileset metadata Configure layer settings Tileset schedule (Preview) Review and create tileset What is a Tileset? Tilesets are map-optimized representations of geospatial data. Instead of trying to load a large geospatial data file all at once, data is divided into small tiles that are loaded and rendered as needed while users zoom and pan across the map. This makes tilesets especially useful for large datasets, such as infrastructure networks, delivery routes, asset locations, service areas, or operational events. Fabric Maps Tileset Builder Capabilities Build map-ready tilesets directly from OneLake data. Visualize large geospatial datasets with high performance. Enable smooth, interactive map experiences at scale. Keep map content synchronized with source data. Eliminate manual exports and external geospatial processing workflows. Integrate native geospatial visualization into existing Fabric data workflows. Common scenarios Tilesets help teams turn large location-based datasets into fast, interactive map experiences. Utility and energy companies can visualize nationwide power line grid systems as a single dataset to monitor field assets and service coverage. Supply chain and logistics teams can explore daily routes, regional boundaries, and operational areas. Retailers can analyze store territories, expansion opportunities, and new developments. Because the data stays connected to Fabric, these map experiences become part of the broader analytics workflow — not a separate geospatial process. To get started, refer to How to create tilesets. Cross-domain intelligence with Azure Monitor data in Microsoft Fabric (Preview) Have you ever detected an issue in your systems but struggled to understand what it meant for the business? As systems grow more complex, this gap becomes harder to bridge. Incidents no longer affect just systems; they affect customers, revenue, and operations in real time. Azure Monitor Logs mirroring into Microsoft Fabric helps close that gap. In just a few steps, telemetry from Log Analytics workspaces becomes available in OneLake alongside business and operational data — without duplication and with near real-time availability. This creates the foundation for Cross-domain insights and actions: Bring observability, operational, and business data together in Eventhouse for real-time analysis. The same unified data can be used by Real-Time Dashboards for investigation and by operations agents to recommend and drive actions informed by both business and observability context. For example, an operations team can identify that a check-in kiosk outage is impacting high-value customers and act before customer impact grows. Advanced Fabric analytics Apply tools like Spark and Power BI for long-term analysis, machine learning, and a wide range of analytical scenarios. For example, an operations team can create a Power BI report showing trends in customer impact and cost, helping management make informed decisions on resource allocation and understand the true cost of application failures. Together, these capabilities help organizations move from isolated technical signals to business-aware insights, decisions, and actions. To learn more, refer to Cross-domain intelligence with Azure Monitor data in Microsoft Fabric (Preview). Until next month That's a wrap for the July 2026 Microsoft Fabric Monthly Update. As always, we'll continue sharing new capabilities, enhancements, and improvements across Microsoft Fabric in future monthly updates. Thank you for being part of the Fabric community!
IDENTITY columns support for Identity Insert (Generally Available)
IDENTITY columns in Fabric Data Warehouse are now generally available. Since preview, thousands of customers have adopted IDENTITY columns to simplify data warehouse design, reduce custom key-generation logic, and make it easier to build scalable dimensional models in Microsoft Fabric. IDENTITY columns automatically generate unique numeric values when rows are inserted into a table. This is especially useful for surrogate keys in dimension tables, fact-to-dimension relationships, migration scenarios, and any workload where stable system-generated identifiers are needed. Instead of maintaining custom logic such as MAX(ID) + 1, ROW_NUMBER(), control tables, hashes, or application-side scripts, data teams can let the Warehouse engine generate unique values during ingestion. What’s new We have expanded IDENTITY column capabilities to support more enterprise data warehousing and migration patterns. The key additions are: IDENTITY_INSERT support, so customers can explicitly insert values into an IDENTITY column when needed, such as when migrating existing warehouse tables or preserving existing surrogate key values. Reseed operations, so customers can reset or advance the next generated identity value after data movement, backfill, reload, or administrative operations. Production-ready support for the core IDENTITY column experience, enabling automatic, system-managed surrogate key generation in Fabric Data Warehouse. These additions are particularly important for customers migrating from existing SQL-based warehouses, where preserving identity values can be necessary to maintain relationships across dimension and fact tables. They also help simplify operational workflows where teams need more control over identity value generation after loading historical data or performing controlled backfills. Why IDENTITY columns matter Surrogate keys are foundational in data warehousing because they provide stable identifiers that are independent of source-system business keys. IDENTITY columns make surrogate key creation simpler and more reliable by generating unique values automatically as data is inserted. This reduces ETL complexity, avoids duplicate-key risks from concurrent loads, and removes repetitive key-generation logic from pipelines and scripts. Fabric Data Warehouse is designed for distributed, parallel execution. IDENTITY values are generated in a way that supports scale-out ingestion, which means values are guaranteed to be unique but are not guaranteed to be sequential or gap-free. This behavior is expected and enables the Warehouse engine to preserve high-throughput load performance while still providing system-managed unique identifiers. Get started You can use IDENTITY columns in Fabric Data Warehouse by defining a BIGINT IDENTITY column in your table schema and omitting that column from inserts when you want Fabric to generate values automatically. For migration or controlled load scenarios, use IDENTITY_INSERT to preserve existing values, then reseed the table when needed so new values continue from the expected point. To learn more, explore the tutorial and the documentation. Thank you to the thousands of customers who adopted IDENTITY columns during preview and shared feedback with us. Your input helped shape the general availability release and the additional capabilities we are introducing today.
Advancing the Microsoft Fabric SQL Query Editor
The Microsoft Fabric SQL Query Editor is the home for web-based SQL development in Fabric. It gives developers a workspace to explore warehouse data, write and run SQL, among many more capabilities. That work rarely starts and ends with a single query. Developers navigate large schemas, author and refine SQL, inspect results, share findings, and connect validated work to downstream analytics and operational workflows. As warehouses and teams grow, each of those steps can introduce friction, from finding the right object, to managing an increasing number of queries or moving between different Fabric experiences. The latest SQL query editor updates are focused on reducing that friction and making the development experience faster, more scalable, and more connected across Fabric. A scalable editor for any size warehouse Working with a warehouse becomes harder when the development tools do not scale with the environment. Large schemas can make objects difficult to navigate, metadata-heavy environments can slow down authoring assistance, and large query results can become cumbersome to inspect in the browser. The latest updates strengthen the core web SQL query editor experiences across Object Explorer, IntelliSense, and the results grid so developers can stay productive as their warehouse grows. To learn more about the rich capabilities the SQL query editor offers, explore the SQL query editor documentation. A faster, more capable data grid (Generally Available) Running a query is only useful if developers can quickly understand the output. Large or wide result sets can be difficult to inspect when the grid is slow or when values do not fit comfortably on screen, often pushing developers to export data just to review it. The brand-new data grid improves performance and makes data and result previews easier to inspect directly in the web SQL query editor. Developers can resize columns for wide result sets, while expanded support for larger LOB data types makes it possible to review larger values directly in the grid, with many more improvements coming soon. Figure: Animated GIF - Brand new results grid, now with support for resizing columns. For additional information regarding the data grid, refer to the Data preview documentation. Object explorer built for large databases (Generally Available) Finding the right table, view, or schema should not become harder simply because a warehouse contains thousands of objects. The redesigned object explorer is significantly optimized for performance when navigating large warehouse environments while keeping schema browsing responsive as the number of objects grows. Developers can also pin frequently used tables, views, and schemas, reducing the need to repeatedly navigate through large object hierarchies during everyday development. Figure: Animated GIF - New object explorer (left) vs. old object explorer (right) loading times for thousands of user objects. To learn more, refer to the Object explorer documentation. IntelliSense, redesigned for scale (Generally Available) SQL authoring becomes increasingly dependent on good database context as schemas grow. Developers should not need to remember every table, column, or object name before they can start writing a query. With improved IntelliSense responsiveness in larger warehouse environments, developers can spend less time looking up object names and more time building and refining their queries. Better query management for development workflows The number of queries developers work with tends to grow alongside the warehouse. Exploratory queries become reusable queries. Saved SQL queries accumulate across projects. Queries need to be shared for team review, revisited later, or cleaned up once they are no longer useful. New query management capabilities make that ongoing work easier to maintain directly within the query editor. Copy and share queries with their context (Preview) Sharing SQL often means separately copying the query, capturing its output, and explaining which results came from which version of the logic. The new copy query experience makes it easier to keep those pieces together. Developers can copy a query together with its results or generate a link that opens the query directly in the tool of their choice. This makes reviews, validation, and collaboration easier while reducing the extra steps required to pick the work back up in another experience. Figure: Animated GIF - Copy query experience in the SQL query editor. More control with autosave (Generally Available) Not every SQL editing session represents work a developer wants to preserve in the same way. Exploratory changes may be temporary, while active development may need to be continuously protected from accidental loss. Developers can now toggle autosave on or off, giving them more control over how changes are preserved based on the way they are working. Figure: Animated GIF - Autosave on/off configuration in SQL query editor. Manage queries at scale (Generally Available) Saved queries can quickly accumulate across ongoing development, investigation, and experimentation. Managing them one at a time becomes increasingly tedious as that collection grows. Bulk query management makes it easier to select and manage multiple queries at once, helping developers clean up old work and keep their query collections organized as projects evolve. Figure: Animated GIF - Bulk management capabilities for queries. Import and export .sql files (Generally Available) SQL development often extends beyond a single tool or environment. Developers may already have queries stored as .sql files or need to move work between Fabric and other parts of their development workflow. The SQL query editor now supports importing .sql files directly for editing, sharing and execution, as well as exporting queries as .sql files for use elsewhere. This makes it easier to bring existing SQL into Fabric, preserve work in a portable format, and move queries between tools without manually copying and pasting code. Figure: Animated GIF - Export queries as .sql files from the SQL query editor for development in other tools. Extending SQL into analytics, semantics, and operations SQL development often produces the starting point for work that continues elsewhere. A developer may validate warehouse data in SQL and then need to analyze it with another engine, connect it to a semantic model, or use the result as part of an ongoing monitoring workflow. Moving between these experiences can interrupt the development flow and create additional steps between understanding the data and doing something with it. New integrations make those transitions more direct from the SQL query editor. Analyze warehouse data across OneLake (Generally Available) Different analytical problems often call for different tools. A developer may begin by exploring warehouse data with SQL but later need Spark for broader data processing or KQL for another analytical scenario. Traditionally, moving between engines can also introduce additional data movement or setup. Directly from the SQL query editor, developers can now create Eventhouse endpoints or notebooks that work with the same warehouse data using KQL or Spark. This makes it easier to choose the engine that best fits the task while staying connected to the same data in OneLake. Learn more about OneLake analytics in the Eventhouse endpoint documentation. Figure: Animated GIF - Create Notebooks and Eventhouse Endpoints directly from the SQL query editor for Spark and KQL based analysis on warehouse data. To learn more about OneLake analytics, refer to the Eventhouse endpoint documentation. Connect SQL to the semantic layer (Generally Available) Validated SQL and warehouse data frequently become the foundation for downstream reporting and BI. Without a direct path into semantic modeling, developers and BI teams often must leave the web SQL Query Editor and start that workflow separately, even when they are working from the same underlying warehouse data. Developers now have the option to create a Direct Lake over OneLake semantic model directly from the SQL query editor, making it easier to move from exploring and validating warehouse data into building a semantic model without extra navigation or setup. Figure: Animated GIF - Create Direct Lake semantic models from the SQL query editor. To learn more about creating semantic models on warehouse data, refer to the Power BI semantic model documentation. Turn SQL queries into operational workflows with Fabric Activator (Preview) Developers often rerun the same SQL queries to monitor changing conditions and catch issues that need attention. That creates repetitive work for developers and operators who need to continually inspect business conditions or workload signals. The new Fabric Activator integration, now in preview, makes those SQL queries in warehouse more operational. Developers can define conditions based on query results and trigger follow-up actions when those conditions are met. Instead of repeatedly running SQL to look for an issue, the query can become part of an ongoing workflow that surfaces when attention or action is needed. To learn more, refer to the alert rule documentation. A more complete web SQL query editor experience These updates are designed around a simple idea: developers should spend more time working with their data and less time working around their tools. Faster navigation and authoring reduce friction in large warehouse environments. Better query management makes ongoing SQL development easier to maintain. Deeper Fabric integrations reduce the distance between writing a query and using that work across analytics, semantic models, and operational workflows. Together, these capabilities make the SQL query editor a more complete development experience for working with warehouse data, from finding the right object and writing SQL to validating results and carrying that work forward across Fabric. Ready to get started? Refer to the SQL query editor documentation. This is just the start of a series of rich investments to make the Microsoft Fabric SQL Query Editor an even more powerful and productive environment for web-based SQL development in Fabric, with many more capabilities coming soon. Stay tuned!
Expose Power Query as a Standalone Runtime and Command-Line Interface (CLI)
Summary Power Query has evolved into one of Microsoft's most powerful data transformation technologies and is now used across Excel, Power BI, Fabric, Dataflows, Power Platform, and other products. However, Power Query can only be executed through a host application, despite the existence of a mature M language and execution engine. I would like Microsoft to expose the Power Query engine as a first-class standalone runtime and provide an officially supported command-line interface (CLI) and API. The Problem Today, Power Query transformations are often embedded inside: Excel workbooks Power BI Desktop files Fabric Dataflows Power Platform Dataflows While this works well for interactive users, it creates challenges for enterprise-grade automation. Many organizations would like to: Schedule Power Query transformations without opening Excel. Run Power Query from PowerShell scripts. Integrate Power Query into CI/CD pipelines. Execute transformations on servers without Office dependencies. Reuse Power Query code across multiple solutions. Treat Power Query as a reusable transformation layer rather than a workbook artifact. Currently, Power Query feels like a language without an officially supported runtime, even though the engine already powers multiple Microsoft products.
Onelake Storage Report
Hi All, I checked the OneLake storage Report for the first time, and it blew my mind... I have one warehouse that is > 1+ TB, while it only contains two (!!) tables. - 700K rows - 1.2M rows I dived deeper into it by connecting the warehouse to blob storage, and I noticed in the subfolder onelake/xxxxx/xxxxxx/Files/ ; there is OVER 800 GB OF FILES ! The stored procedures of those two are quite complex with different update statements, but this should not generate so much files ; as we are paying them as well. I already changed the time_travel_retention_cutoff_date, from 30 --> 5 days, but this has no impact on the /files, only the /tables from what I've read (after 36h still no impact there as well though) The files are kept into that folder going two months back; - Is there a way to change this setting? - Is there a way to reduce all those files that are written? - Anyone else noticing this?Fabric Data Warehouse best practices for medallion architectures
Part three of a series on medallion architecture with Fabric Data Warehouse. Good medallion architecture is mostly operational discipline. In part one of this series, we chose the pattern, and in part two, we filled in the Bronze, Silver, and Gold layers. Now comes the part that usually determines whether the architecture holds up in production: the operating rules. Many medallion architectures look great on paper but become difficult to maintain as data volumes, business requirements, and consumers grow. A medallion pipeline is easy to explain and easy to demo. It is more challenging to keep clean over time. The challenges usually start small: row-by-row loads, report-specific logic in the wrong place, transformations that cannot be safely rerun, or Gold tables that slowly become another staging layer. Why this matters Most medallion problems are not caused by the names Bronze, Silver, and Gold. They happen because the pipeline stops behaving like a pipeline. Bronze starts cleaning. Silver starts serving dashboards. Gold starts compensating for upstream data quality. Before long, nobody knows where a rule belongs, and every change feels risky. The goal of best practices is not to add ceremony. The goal is to make the pipeline predictable: predictable loads, repeatable transformations, trusted outputs, and clear places to look when something breaks. Best practice 1: Batch the writes Fabric DW is built for set-based work. Treat ingestion and transformations as batches, not as a stream of tiny row-by-row operations. In Bronze, that usually means using COPY INTO, Fabric Pipelines, or other bulk-loading patterns to land data in raw tables. If you are loading from files, aim for fewer well-sized files instead of many tiny ones. When practical, files in the 100 MB to 1 GB range are a healthier starting point than a long tail of small files. In Silver and Gold, the same idea applies: prefer set-based T-SQL transformations, CTAS, INSERT...SELECT, and MERGE patterns over procedural row-at-a-time logic. Do this well Load Bronze in batches, and avoid trickle inserts when the source can be staged first. Add ingestion metadata, such as source file name and load timestamp, so every batch is traceable. Keep operational logging lightweight. If you need very high-write audit events, do not turn the warehouse into a single-row logging engine. Let Bronze preserve the batch; let Silver decide what is valid. Rule of thumb: if a load pattern creates a large number of tiny writes, fix the load pattern before tuning the query. Best practice 2: Make Silver rerunnable Silver is where the pipeline earns trust. That means Silver transformations need to be repeatable, testable, and safe to rerun. If an upstream source reloads, or a cleansing rule changes, you should know how to rebuild the affected Silver tables without guessing which reports need to be patched. This is where idempotent design matters: a transformation should produce the same result when run again against the same inputs. In Fabric DW, use CTAS when you want to materialize a clean table from a query, INSERT...SELECT for controlled incremental loads, and MERGE when late-arriving or changed data needs to update existing Silver rows. Do this well Design transformations so they can run again without duplicating or corrupting data. Use staging tables when the logic is complex. A few clear steps are easier to operate than one unreadable query. Put quality gates in Silver: required fields, valid formats, duplicate handling, and reason codes for rejected records. Choose precise data types and lengths. Silver is the right place to turn loose source data into reliable analytical data. Rule of thumb: if a report needs to clean the data again, Silver did not finish its job. Best practice 3: Shape Gold for consumption Gold is not just “the final table.” Gold is the business-facing serving layer. It should be modeled around how people ask questions, not around how the source systems store data. For some workloads, that means a star schema with fact and dimension tables. For others, it means a data mart, a wide reporting table, or a pre-aggregated summary. The pattern matters less than the principle: Gold should make the common analytical path simple, fast, and trustworthy. This is also where you should be careful not to let Gold become a junk drawer. If Gold is full of one-off fixes, report-specific exceptions, and raw technical fields, the layer is doing too much. Do this well Model the grain explicitly. A fact table without a clear grain becomes hard to explain and harder to debug. Pre-aggregate where the business repeatedly asks the same question. Hide technical fields that helped the pipeline but do not help the consumer. Keep Gold dependent on Silver by default. Direct Gold-to-Bronze dependencies should be rare and deliberate. Rule of thumb: Gold should answer the business question quickly without making the report author rediscover the pipeline. Best practice 4: Use Fabric DW defaults, but do not fight the engine Fabric DW gives you a SQL warehouse over Delta data in OneLake. That means you get transactional behavior, optimized storage patterns, and a managed engine that handles many physical decisions for you. The practical advice: do not bring every habit from traditional data warehousing with you. You do not need to micromanage distribution or indexing the same way you would in older platforms. Focus first on healthy data layout, set-based transformations, good table design, and predictable query patterns. At the same time, do not ignore the basics. Query performance still benefits from clean data types, useful statistics, well-shaped Gold tables, and avoiding unnecessary scans. Do this well Keep V-Order and platform optimizations on unless you have a measured reason to change them. Use the performance guidance for Fabric Data Warehouse before inventing custom tuning patterns. Check query behavior when a Gold table becomes critical to many reports. Treat advanced exceptions as exceptions. Most teams should start with the defaults and tune only when evidence says to tune. Rule of thumb: tune from evidence, not from habit. Best practice 5: Monitor by layer A medallion pipeline should be observable at each layer. If a dashboard is wrong or slow, you should be able to tell whether the issue started in Bronze ingestion, Silver transformation, or Gold serving. In Fabric DW, use Query Insights and the warehouse monitoring views to understand query behavior, expensive operations, and refresh patterns. Pair that with pipeline-level monitoring so you can see not only whether a job failed, but where the failure happened. Measure the pipeline in terms the team can act on: Bronze load duration, Silver transformation duration, rejected-record counts, Gold refresh duration, and Gold query performance. Do this well Track load and refresh duration by layer. Log the number of records received, accepted, rejected, and published. Watch critical Gold queries after refresh, especially the ones that feed executive dashboards or widely used semantic models. Keep operational alerts tied to business impact. A failed Gold refresh matters differently from a delayed Bronze load. Rule of thumb: if you cannot tell which layer failed, your monitoring is not layer-aware enough. Before moving a medallion pipeline into production, use the following checklist to verify that each layer is operating as intended. The best-practice checklist Area Bronze Silver Gold Write pattern Batch ingest with metadata Set-based transformations Scheduled refreshes Quality rule Preserve what arrived Validate, conform, and flag Expose trusted fields only Performance focus Avoid tiny writes Keep logic rerunnable Shape for common queries Takeaway Part two of this series was about one job per layer. Part three is about operating each job like it matters. Batch the writes. Make Silver rerunnable. Shape Gold for consumption. Use Fabric DW’s managed engine instead of fighting it. Monitor the pipeline by layer so failures are easy to locate and fixes happen in the right place. If you follow those rules, your medallion architecture becomes less fragile over time, not more. This post is part of our Medallion Architecture on Fabric Data Warehouse series: Choosing your medallion pattern in Fabric Data Warehouse Building the Bronze → Silver → Gold layers Fabric DW best practices for medallion architectures Securing and governing your layers Performance tuning your medallion pipeline Ready to go deeper? Explore the Microsoft Fabric Data Warehouse performance guidelines and ingestion guidance, then stay tuned for Part four of this series, where we’ll cover securing and governing your layers.
