Mirroring for Google BigQuery in Microsoft Fabric (Generally Available)
Mirroring for Google BigQuery in Microsoft Fabric is now generally available. For organizations running critical workloads on BigQuery, this means a simpler, faster, and lower-cost path to bringing your data into the rest of your analytics estate — with production support, an enterprise SLA, and no pipelines to build or maintain.
Fabric Migration Assistant for SQL database (Generally Available)
Migrating a SQL Server database to the cloud rarely fails on the data — it fails on everything around it. Schema objects that use unsupported features surface only after you've provisioned a target and started deploying. Remediation turns into a manual pass-through error logs. Schema and data move through separate tools, on separate timelines, with separate things to go wrong. A migration most teams could scope in a day ends up planned like a quarter-long project. What makes that work worth doing is where it lands. SQL database in Fabric provisions in seconds, with no networking, storage, or sizing decisions to make — the serverless architecture scales compute and storage automatically, and high availability, disaster recovery, Microsoft Entra authentication, customer-managed keys and SQL auditing are built in rather than configured. Your existing T-SQL, SSMS and Visual Studio Code skills carry over unchanged. And the part customers tend to notice first: near real-time replication to OneLake makes your operational data available to Power BI, Spark notebooks and AI workloads without a separate pipeline. The Fabric Migration Assistant for SQL database is what gets you there — one guided workflow that assesses your schema, provisions the target, deploys with AI-assisted fixes for what doesn't translate, and copies your data without leaving the wizard. Now generally available and supported for production migrations, and it ships with the capability the preview community asked for most: the ability to validate your schema before you create anything in Fabric. Migration Assistant for SQL database in Fabric landing page. Validate before you provision The new Validate function gives early signals of incompatibilities. Upload a DACPAC generated from your source database and the Migration Assistant analyzes it on its own, before a target database exists. You get the compatibility picture up front: which schema objects deploy cleanly, which ones use features that SQL database in Fabric does not support, why each incompatible object fails, and how the objects depend on one another. Nothing is provisioned, nothing is consumed, and no capacity is committed while you assess. That changes how a migration gets planned. Architects can scope effort and risk during the assessment phase, hand a concrete remediation list to the team that owns the source database and walk into a change-advisory conversation with evidence instead of an estimate. Teams evaluating several candidate databases can validate all of them in an afternoon and sequence the easy wins first. The migration itself starts only once you already know how it will end. Getting Started The new validation experience changes the migration conversation from reactive to proactive. Instead of discovering compatibility issues after you've created a target and started deploying, you can assess your schema upfront, understand exactly what needs attention, and move forward with a migration plan grounded in real results. When you're ready, the Migration Assistant guides you through assessment, deployment, and data movement in a single workflow. Export your source schema as a DACPAC using SQL Server Management Studio, the MSSQL extension for Visual Studio Code, or SqlPackage. Complete the prerequisites. From your Fabric workspace, select Migrate and choose Migrate to SQL database in Fabric. Run Validate first. Upload the DACPAC and review the compatibility report: migrated objects, failed objects, the reason behind each failure, and the dependency chain between them. Resolve what you can at the source or plan the remediation. Provision the target when you are ready. Name your SQL database in Fabric and the wizard creates it in seconds, with no networking, storage, or sizing decisions to make. Deploy the schema. The assistant applies the schema and surfaces any remaining incompatibilities, with Copilot-powered fix suggestions you can accept or reject interactively. Primary objects are resolved before dependent objects follow. Copy the data using built-in Fabric Copy Jobs powered by Data Factory. The assistant integrates with the Fabric Data Gateway to reach on-premises sources securely and supports pre-deployment and post-deployment scripts so constraints can be disabled and re-enabled around the load. Validate and go live. Review results in the Migration Assistant panel inside the SQL editor, verify your data, update application connection strings, and complete the migration session. Learn more Migrating a database shouldn't require weeks of investigation before you can determine whether the move is viable. With Fabric Migration Assistant, you can assess compatibility, understand remediation requirements, deploy your schema, and move your data through a single guided workflow. Download a DACPAC from an existing SQL Server database and run a validation assessment today. You'll get immediate insight into compatibility and migration readiness, helping you prioritize the right databases and accelerate your move to SQL database in Microsoft Fabric. For detailed guidance, review the migration documentation.Fabric March 2026 Feature Summary
Welcome to the Fabric March 2026 Feature Summary—and welcome to FabCon! As we kick off FabCon, this update captures the momentum we’re seeing across the Fabric platform and the conversations happening with customers and partners right now. March brings a wide range of enhancements across governance, data engineering, real-time intelligence, data science, extensibility, and AI—all designed to help teams build, operate, and scale end‑to‑end data solutions with confidence. Many of the capabilities highlighted here reflect direct feedback from the community and real‑world usage we’ve learned from—including insights shared leading up to (and during) FabCon. We are eager to share what’s new and to continue the conversation throughout the week. If you haven’t already, check out Arun Ulag’s hero blog “FabCon and SQLCon 2026: Unifying databases and Fabric on a single, complete platform” for a complete look at all of our FabCon and SQLCon announcements across both Fabric and our database offerings. Contents https://youtu.be/xhrSMNNX5ho?si=aTWiAMq0PvtTxej8 Events and Announcements Don’t miss the next Monthly Data Days Sessions On March 26 we have a special edition of Fabric Data Days featuring two topics. Join us at 8 AM Pacific for a session on getting started with Fabric IQ. Then at 3 PM Pacific we’ll discuss mapping and spatial analytics in Fabric. Register now! Couldn't make it to Atlanta or just want more FabCon + SQLCon? Join us in Barcelona this September. FabCon Europe is happening again in 2026. Mark your calendars for September 28 – October 1, 2026. Register now to access Super Early Bird pricing! Fabric Platform OneLake Catalog Govern for admins (Generally Available) In today’s data-driven world, effective data governance is crucial to ensure the integrity, security, and usability of data. OneLake catalog is available for Fabric admins, providing tools and insights to govern and secure data estates within Fabric in one place. Figure: OneLake Catalog: Govern for Admin view Figure: OneLake Catalog: Govern for admin—view more report. OneLake Catalog search API and MCP tool (Preview) OneLake Catalog’s Search API brings cross-workspace discovery to code. Instead of traversing workspace-by-workspace and “listing everything,” a single search request can locate matching items across your accessible estate based on catalog metadata and the user’s permissions. Search is designed to help even when the exact name isn’t known. Free-text matching includes the item’s display name and description, so a keyword you remember is often enough to find the right entry. Results can be filtered by the item's type to narrow down the scope of your search. The set of supported metadata signals and filters is expected to grow, enabling richer and more targeted discovery scenarios. The catalog search capability is also included as a built-in tool in the Fabric Core MCP server so AI agents can reliably locate the right Fabric asset as part of a broader workflow, then continue with follow‑up actions using other tools. Workspace tags (Generally Available) Fabric tags add meaningful metadata so people can find the right content faster and organize it consistently. That capability is available for workspaces. Workspace tags add shared context (like team, project, or cost center) at the workspace level, helping teams discover and manage workspaces more efficiently, while also enabling scalable governance through APIs. Figure: Workspace settings screen showing applied workspace tags Workspace tags are built on the existing Fabric tags model: tags are defined once, then applied on items and workspaces. Workspace admins can apply and remove tags in workspace settings, making it easy to add shared context at the workspace level. A workspace can have up to 10 tags applied. Workspace tags are surfaced in key discovery experiences so they’re usable in day-to-day navigation: Workspaces can be filtered by tags both in the workspaces list and in OneLake Catalog Explorer, a tags indicator also appears in the workspaces list and in OneLake Catalog Explorer next to every tagged workspace. Tag names are shown on the workspace screen itself, making the workspace context immediately visible. Tagging can be retrieved and managed at scale using APIs, enabling consistent application and reporting across workspaces. Data loss prevention policies for Fabric—Extending restrict access to structured data in OneLake (Preview) When handling sensitive data, it might be challenging to find the right balance between federating data and keeping it secure and compliant. Data Loss Prevention (DLP) policies enable organizations to detect sensitive data and surface it to users and admins when it is found. The Restrict Access action allows you to restrict access to your data once the sensitive information is detected. DLP Restrict Access reduces the risk of exposure to unauthorized users, without slowing analytics or collaboration. Customers can scale Fabric with confidence, meeting compliance requirements while enabling secure, enterprise-wide data sharing. With this release, you’ll be able to apply access restrictions through DLP on: Warehouses KQL databases SQL databases Lakehouses (previously supported) Semantic models (previously supported) Cosmos DB and mirrored databases are coming soon. Figure: Fabric warehouse with a restrict access indication Admins can ensure that sensitive data is protected consistently wherever it lives and however it is accessed. Learn more about restrict access in DLP. Lakehouse Signals in IRM (Generally Available) Microsoft Purview Insider Risk Management cross-references millions of signals across all your products, to create comprehensive profiles of potentially unethical behavior inside your organization. Using Lakehouse indicators in Insider Risk Management enables security teams to detect and investigate risky data activity in OneLake with greater precision and context. By incorporating Fabric Lakehouse signals directly into insider risk policies, security teams can correlate data access and movement with DLP, labeling, and audit signals in a single investigation experience—reducing blind spots and accelerating response to potential data exfiltration or misuse. This provides stronger protection for high‑value analytics data while maintaining built‑in privacy controls and avoiding the operational overhead of deploying separate monitoring tools. Figure: Lakehouse indicators used within the IRM tool Learn more about Fabric indicators in Insider Risk Management. Quick policy for data theft for Fabric (Generally Available) A new quick policy for the Data Theft rule is available for Fabric. This streamlined experience makes it easier to set up protection against data exfiltration scenarios, helping security teams take action faster when sensitive Fabric data is at risk. Learn more about IRM quick policies. Insider Risk Management PAYG Usage Report (Generally Available) The Microsoft Purview Insider Risk Management pay-as-you-go feature usage report is designed to provide transparency to customers, enabling more accurate budget planning and policy tuning. IRM admins can check the distribution of PAYG processing units billed across workloads (Fabric), sub-workloads (Power BI, Lakehouse), and indicators (downloading Power BI reports, etc.) to fine-tune their policies and plan PAYG budgets accordingly. Figure: Pay-as-you-go Usage Report Purview DSPM for AI for Fabric Copilots and data agents (Preview) As AI adoption accelerates, organizations need built‑in protections to keep data safe. With Purview Data Security Posture Management (DSPM) for AI, customers gain visibility and control over AI interactions. DSPM for AI helps teams spot sensitive data risks in AI prompts and responses, identify risky AI behavior, and apply consistent governance using familiar tools like DSPM, Insider Risk Management, Audit, and eDiscovery—so organizations can move faster with AI, without compromising security or compliance. bric_Copilot Figure: Purview DSPM for AI report showing Data Agent interaction in Fabric Learn more about DSPM for Fabric Copilots. Branched workspace with Git integration (Preview) Branched workspace is a new developer experience designed to simplify how teams work with feature workspaces during a branch‑out flow. With clearer visual cues and richer context, developers can easily understand workspace relationships and work more confidently when branching and iterating on features. This feature will be released by the end of March 2026. Figure: Fabric workspace tree showing the new relation between workspace and branched workspace Follow our new Git developer experiences in Microsoft Fabric (Preview) announcement. Selective branching with Git integration (Preview) Fabric Git Integration Branch-out with selective branching introduces a more focused branch‑out experience in Fabric. Developers can select only the items they need for a feature, reducing clutter in the target workspace, improving reliability, and accelerating time‑to‑code. By working with a smaller, purpose‑built workspace, developers can iterate faster and with greater confidence. Figure: Branch-out selective branching dialog Follow our new Git developer experiences in Microsoft Fabric (Preview) blog announcement. Compare code changes with Git integration (Preview) The new compare code changes experience helps developers confidently sync their Fabric workspace with a connected Git branch by clearly showing what changed before taking action. It provides a familiar code‑compare experience that highlights the exact differences since the last sync—down to the item and file level—whether the change originated in the workspace or in the repository. This makes it easier to review updates, understand their impact, and resolve conflicts by comparing workspace and Git versions side by side before committing, updating, or undoing changes. Figure: Git Integration code compares changes dialog Follow our new Git developer experiences in Microsoft Fabric (Preview) blog announcement. Connection reference item type in Variable Library (Preview) The new connection reference item type in Variable Library introduces a new way to manage external data connections in Microsoft Fabric. This new variable type lets you reference existing connections—such as Azure SQL or Snowflake—by storing a connection ID in the Variable Library, instead of embedding static connection strings in code. Figure: Variable Library “connection reference” item type option Connection reference variables work seamlessly with CI/CD and Git, enable safer environment‑specific configuration across dev, test, and prod, and ensure only authorized connections can be selected through the UI. This makes it easier to build, deploy, and manage Fabric solutions with cleaner configuration, stronger governance, and improved portability across CI/CD stages. Bulk import and export items definition APIs (Preview) These APIs enable you to programmatically export, import, and synchronize Fabric item definitions across workspaces at scale—all through the Fabric REST API. Every Fabric item—whether it’s a Notebook, Report, Semantic Model, Data Pipeline, or KQL Dashboard—has an underlying item definition: a portable schema containing the item’s full configuration and content (encoded in Base64). The Import & Export Batch APIs let you: Export item definitions individually or in bulk from any workspace Import (create) items from definitions into a target workspace Update existing item definitions in-place for continuous deployment List & paginate through all items in a workspace for batch operations Key scenarios Workspace migration: Moving items across workspaces, tenants, or regions is one of the most common requests from Fabric customers. The batch APIs let you export all items from a source workspace into a portable JSON manifest, then import them into any target workspace. This is invaluable for replicating environments across different tenants and cloning a production workspace for testing purposes. CI/CD and DevOps integration: To support enterprise DevOps practices in Microsoft Fabric, organizations can integrate the new Bulk Export and Import APIs into their CI/CD pipelines. Fabric item definitions can be treated as code—exported and versioned in Git using Fabric Git Integration or the bulk-export API, validated through pull request workflows, and promoted through a well-defined release process. When deploying across workspaces, the bulk-import API enables consistent, automated promotion into test and production environments using the underlying Fabric dependency logic that creates new items in the correct order, retains the original relations, and updates existing ones in place. Metadata backup and recovery: Schedule periodic batch exports to capture the full state of your workspace as versioned JSON manifests. Store them in Azure Blob Storage, a Git repository, or any durable storage. If something goes wrong, re-import the manifest to restore your workspace to a known-good state. Metadata scanning and lineage analysis: Tools that analyze report definitions to discover data lineage—such as which semantic model columns are used in each report—can extract hundreds of report definitions in bulk instead of one at a time, reducing scan time.Export (read) operationsMethod Endpoint Description POST /workspaces/{workspaceId}/items/bulkExportDefinitions?beta=true Export an item’s full definition as Base64-encoded parts. May return 202 for LRO. Import (write) operationsMethod Endpoint Description POST /workspaces/{workspaceId}/items/bulkImportDefinitions?beta=true Create/Update an existing item’s definition in-place. Ideal for CI/CD sync. Resources Full announcement: Bulk import/export items definition APIs API documentation: Fabric Items API Reference Comprehensive guide: Item Management Overview Item definition structure and formats: Item Definition Reference How to handle async operations with polling: Long-Running Operations Guide App registration and authentication setup: Microsoft Entra ID Documentation CI/CD tutorial using Bulk API: CI/CD tutorial using the Bulk Export and Import APIs Fabric CLI v1.5—Power BI Scenarios, CI/CD Deployments, and DX Improvements The Fabric CLI v1.5 is the most scenario-driven update yet. Power BI developers can now trigger semantic model refreshes, rebind reports, and script end-to-end deployment workflows—all from the terminal, without portal context-switching. The release also adds a new deploy command for CI/CD, interactive REPL mode, JMESPath filtering, notebook export in multiple formats, Python 3.13 support, and expanded coverage for Fabric items. Many of these improvements are community-contributed, making the CLI a comprehensive open-source automation surface for Fabric. CI/CD deployments from the CLI—deploy workspaces in One Command A new deploy command integrates the Fabric CI/CD Python library directly into the Fabric CLI, enabling full workspace deployments—including item rebinding and configuration—from a single command. Teams can run deployments from their terminal, GitHub Actions, or Azure DevOps pipelines. Combined with Service Principal authentication and federated credentials for GitHub OIDC, this enables zero-touch, Git-based promotion workflows that fit modern DevOps practices—no custom scripts or additional tools required. For usage examples, refer to the CI/CD examples and setup guide. Fabric CLI as an Execution Layer for AI Agents Fabric CLI is designed to work well with AI agents. A structured agent instructions file and a dedicated Fabric CLI Skill provide AI assistants like GitHub Copilot and Claude with the context they need to generate correct CLI commands from natural language. Improved error messages with actionable guidance help agents self-correct, and the interactive REPL mode enables persistent terminal sessions for multi-step agent workflows. Using a CLI as the execution layer for AI agents is an emerging industry pattern—instead of agents calling raw REST APIs (which require extensive token-heavy context about endpoints, auth, and payloads), agents issue concise CLI commands that encapsulate that complexity, making AI-driven Fabric automation more practical and reliable. Learn more with Fabric CLI agent docs and AI assets on GitHub. Fabric Remote MCP Server: AI agents operate directly in your Fabric environment Fabric Remote MCP is a cloud-hosted MCP server that allows AI agents to perform real operations in your Fabric environment—create workspaces, manage permissions, work with item definitions, and more. No local installation is required. Agents authenticate via Entra ID and operate within your existing RBAC boundaries, with every tool invocation recorded in audit logs. The preview launches with capabilities spanning workspace management, item CRUD and definitions, and permission management. It works with any MCP-compatible client, including GitHub Copilot, Cursor, and Claude Desktop. Learn more in this blog post: Introducing Fabric MCP (Preview). Fabric MCP AI code assistants (Generally Available) The Fabric Local MCP is an open-source MCP server that runs on your machine. This solution integrates AI coding assistants with the comprehensive Fabric API, offering OpenAPI specifications, best-practice guidelines, item definition schemas, and example payloads to enable agents to produce precise, production-ready code while minimizing errors. OneLake tools enable live file operations including upload, download, table inspection, and item creation. This update introduces integrated authentication, automatic retry, production SLAs, and telemetry. Install via npx Microsoft/fabric-mcp in any MCP-compatible client—it works with VS Code, Claude Desktop, Cursor, and more. Fabric Local MCP on GitHub.Fabric Fabric Extensibility Extensibility (Generally Available) After six months in preview mode, gathering feedback, resolving bugs, and strengthening the platform, we’ve reached the next milestone. Partners and customers can build, validate, and publish custom Fabric workloads to production with full Microsoft support. Key highlights: All core capabilities are stable and supported: OneLake storage, native item lifecycle, Entra token acquisition, iFrame relaxation, Workload Hub publishing. The Starter Kit ships with production-ready UI components (ItemEditor, WizardControl, OneLakeView, and more) that reduce time to first workload. GitHub Copilot integration and a new DevContainer/GitHub Codespaces setup reduce setup effort—no local machine required. The first Fabric Extensibility Community Contest drew strong community participation, with real workloads already appearing in the Workload Hub. Learn more about Fabric Extensibility (Generally Available). CI/CD & remote support (Preview) Three new features further enhance the professional development experience for Fabric workloads. CI/CD Support Workload items are first-class citizens in Fabric's CI/CD platform. Items participate in Git integration and Deployment Pipelines with no custom tooling. Variable Library support means items automatically pick up workspace-specific configuration (e.g., the right Lakehouse reference) when promoted across dev, test, and production—no hard-coded IDs, no manual reconfiguration. Figure: CICD enablement for Hello World Sample Variable Library Support Items can be read from Fabric's Variable Library, allowing workspace-specific configuration (e.g., the right Lakehouse reference) to resolve automatically when an item is promoted across dev, test, and production stages—no hard-coded IDs, no manual reconfiguration, and no custom deployment hook logic required. An opt-in webhook that fires whenever a workload item is created, updated, or deleted—regardless of whether it happened through the UI, the REST API, or a CI/CD pipeline. It’s designed for licensing checks, infrastructure provisioning, and external system synchronization. There’s no impact on workloads that don't register an endpoint. Figure: Variable Picker within Fabric Cloud Shell Item Remote lifecycle notification API Workloads are no longer just passive objects sitting in a workspace. The Remote Lifecycle Notification API is an opt-in capability—there is no requirement to use it. If your workload does not need backend notifications, you simply don't register an endpoint, and everything works exactly as before. Fabric Scheduler / Remote Jobs This feature allows workload items to expose named job types that users can schedule directly from Fabric. When a scheduled job fires, Fabric calls a registered endpoint on your workload backend—passing along the item context and a delegated user token. For all these features, you’ll find samples in the Toolkit Starter Kit. Learn more about Fabric Extensibility CI/CD and remote capabilities in this blog post. What's new in workload management As the Fabric extensibility ecosystem grows, with partners publishing workloads and organizations building custom solutions, managing workloads at scale demands more than a single settings page. IT admins need centralized governance and a clear overview of what's being used across the organization, and workspace teams need self-service agility. Three key workload management features for Microsoft Fabric Extensibility will launch by April 1, 2026: Workload admin portal (Generally available) Add workload to workspace (Generally available) Workload management admin APIs (Preview) These will enhance governance through portal, API, and self-service capabilities. Admin portal: centralized admin workload overview (Generally Available) The Fabric Admin Portal now includes a dedicated Manage Workloads tab, a single pane of glass for workload governance across your organization. Centralized workload visibility: view all workloads available for assignment in your tenant in a single centralized view, including status information and workload details. Tenant assignment controls: manage workload assignment at the tenant and workspace level. Add workload to workspace (Generally available) The workspace-level workload assignment was previously introduced in Preview. It allows workspace admins to add workloads directly to one or more workspaces. How it works (Workspace admins): Navigate to the Workload Hub from the left menu in Microsoft Fabric or from your workspace settings. Browse or search for the workload you want to add. Select "Add Workload" and select "To Workspace" from the dropdown. Select your workspaces: search, check the workspaces you want, and use "View more/less" to manage the list. Select "Add": the workload is immediately available in your selected workspaces. Workload Management Admin APIs: Overview and Control at Scale (Preview) Capabilities For Fabric admins who need a programmatic view of their workload landscape, the new Workload Management Admin APIs provide governance and oversight across the tenant through a REST interface. List all workloads: view all workloads available to be added in the tenant, and view which workloads were added. List all workload assignments in the tenant. Drill down into a specific workload and view where it was added (tenant, workspace, capacity). Manage workload assignments (add or remove) to capacities, workspaces, and tenant. Self-service workload publishing (Generally Available) A frequent question from ISV partners using the Microsoft Fabric Extensibility Toolkit is: "How do I get started publishing?" Key features Self-service workload publishing is expected to be generally available by the end of March 2026. ISV partners will be able to publish workloads directly to selected customer tenants for private preview without requiring a manual submission request. This can accelerate time to market and support faster iteration with customers. Self-Service Workload Publishing gives ISV partners full control over their private preview journey: Publish to up to 20 customer tenants: share your workload with selected customers for testing and validation, no Microsoft certification required. Workload name reservation: reserve your globally unique workload name (e.g., Contoso.DataQuality) to protect your brand identity before formal publication. Automated validation: your workload package is automatically validated against manifest schema, naming conventions, and security requirements at upload time. Seamless path to general availability: once validated with customers, use the same workload package to pursue formal certification and publish to the global Workload Hub. OneLake Third-party support for OneLake security This month, we announced third‑party support for OneLake security, taking an important step toward interoperable data security. As customers increasingly build lake‑first architectures on open formats like Delta and Iceberg, they expect the freedom to use multiple analytics engines without copying data or redefining security. OneLake security addresses this need by enabling security to be defined once and enforced consistently wherever data is accessed. At the core of this capability is the authorized engine model. Security policies—including role‑based permissions, row‑level security (RLS), and column‑level security (CLS)—are centrally defined and managed in OneLake, while enforcement happens at query time inside the engine reading the data. Authorized third‑party engines securely retrieve the relevant metadata and effective security definitions through OneLake APIs and apply them during query execution. This ensures users see only the rows and columns they are permitted to access, while OneLake remains the single source of truth for access control. To support adoption, we’ve published implementation guidance and setup documentation for both engine builders and users. The APIs are designed to be engine-agnostic and easy to integrate by providing pre-computed effective access definitions. Looking ahead, we’ll continue evolving OneLake security APIs, including adding support for bitmap-based RLS enforcement. With this release, data vendors can integrate directly with OneLake security, customers can maintain a single security model, and users gain the flexibility to query OneLake data using the engines of their choice. OneLake file explorer (Generally Available) You can easily access and organize all your OneLake data from Windows using the OneLake file explorer. The file explorer lets you browse every workspace and data asset, and upload, download, or edit these files using the same familiar experience as OneDrive. By bringing data lakes into the Windows file system, the file explorer makes enterprise data more accessible for business users. Data Engineering Fabric Runtime 2.0 (Preview) Fabric Runtime 2.0 (Preview) is a next-generation runtime that is purpose-built for large-scale data computations in Microsoft Fabric and introduces key features and components that enable scalable analytics and advanced workloads. Apache Spark: 4.0 Components include Operating System: Azure Linux 3.0 (Mariner 3.0) Java: 21 Scala: 2.13 Python: 3.12 Delta Lake: 4.0 This screenshot demonstrates how you can switch to Runtime 2.0 at the Workspace settings and the Environment levels. Figure: Change runtime at the workspace settings level Explore the full documentation and start using Runtime 2.0 in Fabric. Custom Live Pools for Fabric Data Engineering Modern data engineering workloads are rarely one‑size‑fits‑all. Teams often need predictable performance, isolated resources, or customized configurations for critical production pipelines and high‑value interactive development. At the same time, Spark session startup times can degrade in real-world enterprise environments, especially when users have custom library dependencies. Workspaces or tenants are secured with Private Links or Managed Private Endpoints. In these scenarios, Spark clusters must be created on demand within strict network boundaries, and libraries need to be resolved and installed dynamically, adding noticeable startup latency. Custom Live Pools address this challenge by introducing dedicated, long‑lived Spark pools that stay warm inside your network boundary and come preconfigured with the required dependencies. With Custom Live Pools, Fabric Data Engineering now enables you to: Create dedicated Spark pools and schedule them tailored to your workload needs. Reduce session startup overhead by keeping pools warm with libraries preinstalled. Run reliably within Managed VNets and Private Link–enabled environments. ompute_configuration_panel_within_a_data_analytics_platfo Figure: Animated GIF demonstrating the setup of custom live pools in an Environment Because these pools are already provisioned within the workspace’s network boundary and fully initialized with dependencies, users can start working immediately, without paying the repeated cost of cluster spin‑up and library installation. Custom Live Pools are ideal for: Production pipelines that require consistent SLAs. High‑value interactive notebooks used by data developers. Teams operating in secure or regulated environments. How to set up a Custom Live Pool: Navigate to your Compute tab in your Environment. Select Spark pool and enable the option for Live Pool. Specify the Schedule, Time period of inactivity, and Retrigger frequency. Job concurrency and queue monitoring experience for Fabric Data Engineering As organizations scale their Fabric usage, understanding what’s running, what’s queued, and why becomes essential. The new job concurrency and queue monitoring experience delivers deep visibility into Spark workload execution across your environment. View active, queued, and completed jobs in a single place. Understand why jobs are queued and how concurrency limits are applied. Identify bottlenecks caused by capacity or concurrency constraints. Make informed decisions to tune workload scheduling and resource allocation. Figure: GIF demonstrating the new job concurrency and queue monitoring view in the Data Engineering/Science Spark settings page of Workspace settings Accessing workspace monitoring To view concurrency and queue signals for your specific workspace: Navigate to Workspace settings. Select Data Engineering/Science > Spark settings. Select Jobs to view the live view of your workspace level Spark queue and concurrency. Resource Profiles for Fabric Data Engineering Modern data engineering teams shouldn’t need to be Spark experts to get great performance. With Resource Profiles in Fabric Data Engineering, users simply describe what they’re trying to do, and Fabric automatically recommends the optimal compute configuration. Figure: GIF demonstrating the new resource profiles experience in workspace settings Simple inputs, smart recommendations Instead of tuning dozens of Spark settings, users provide a few high‑level workload details through an intuitive UI: Primary use case, such as a specific medallion layer (Bronze, Silver, or Gold) or task‑based optimization (read‑heavy or write‑heavy workloads). Typical data volume. Data characteristics, such as whether input data contains many small files. Maximum capacity units (CU) for the Spark pool. Once these inputs are provided, users select Get recommendation, and Fabric automatically generates an optimized configuration tailored to that workload. Based on the inputs shown above, Fabric recommends: The appropriate Resource profile. Node family and size. Autoscale and dynamic executor settings. Optimized Spark driver and executor cores and memory. A compatible runtime version. All recommendations are derived from proven best practices and internal performance tuning, removing guesswork and trial‑and‑error. Where to configure Users can enable and manage Resource Profiles from workspace settings: Go to Workspace settings > Data Engineering and Data Science > Resource optimization. Select or edit the optimized profile for the workspace. Rerun the Optimize for your use case flow as workloads evolve. Apply consistent configurations across all Spark workloads in the workspace. Once configured, all notebooks and pipeline‑triggered Spark jobs inherit these optimized settings automatically, without requiring per‑notebook configuration. Why this matters This experience enables: Performance by default: optimized compute without manual tuning. Consistency: the same performance characteristics across users and jobs. Better price‑performance: right‑sized resources aligned to workload intent. Lower operational overhead: fewer tuning cycles and support escalations. As workloads change over time, teams can simply revisit the optimization flow, update a few inputs, and let Fabric adapt the configuration—without rewriting code or Spark settings. Figure: Introduction to Resource Profiles Experience Figure: Recommendations generated based on user inputs To learn more about the Resource Profiles experience in Microsoft Fabric Data Engineering, refer to the Microsoft Learn documentation. Installing libraries with Quick mode in Spark Environment (Preview) Managing libraries shouldn’t slow down your development workflow. In Microsoft Fabric Environments, we’re introducing a more efficient way to iterate on libraries while keeping production workloads stable and reliable. Fabric Environments now support two complementary library installation modes that you can use side by side: Quick mode: a fast, on-demand installation path designed for development and experimentation, where libraries are installed when a notebook runs. This avoids heavy processing during the environment publishing and significantly reduces publish time and notebook startup latency when you’re iterating on lightweight or frequently changing dependencies. Full mode: a snapshot‑based installation path optimized for production workloads and pipelines, where libraries are fully resolved, validated against the Spark runtime, and published as a stable snapshot to ensure consistency and reproducibility. Figure: Add libraries in Quick mode and Full mode This new feature lets you move faster during development without compromising production stability. You can keep your core, production‑ready libraries in the snapshot‑based mode, while using the on‑demand path to quickly test new packages or iterate on custom libraries, all within the same Environment. Dynamic session sharing limit up to 50 for high concurrency Fabric High Concurrency Spark sessions enable both interactive exploration and large‑scale, pipeline‑driven notebook execution, supporting parallel, scheduled, and event‑driven workloads at enterprise scale. Customers often achieve higher density by packing notebooks into a shared High Concurrency (HC) session using session tags, effectively fitting up to five notebooks per session to control startup overhead and cost. While effective, this approach relies on static limits and manual tuning. With this update, Fabric Data Engineering allows the maximum number of notebooks attached to a High Concurrency session to be increased up to 50, enabling dynamic session sharing at much higher scale. Where to set the configuration You can set the configuration in the Environment item that your notebooks or pipeline‑triggered notebooks use: Go to Workspace → Environments Select the Environment attached to your notebook or pipeline Open Spark Properties Add the High Concurrency configuration Set spark.highConcurrency.max to a value between 2 - 50 Note: This update does not change the default limit of five. This enables: Interactive notebooks, used for exploratory analysis and collaboration. Notebook jobs triggered by pipelines, running in parallel within shared HC sessions. Dynamic adjustment of session sharing limits based on workload intensity, cost, and price‑performance goals. By increasing the session sharing limit, customers can: Improve session acquisition times during peak load. Increase notebook density without fragmenting sessions. Tune concurrency to match workload demand rather than fixed defaults. Achieve better price‑performance efficiency while preserving isolation and fairness across jobs. To learn more about increasing your session sharing limit in High Concurrency mode, please check out Microsoft Learn documentation. Data export settings for notebooks With data export settings for notebooks, Microsoft Fabric empowers administrators with explicit, tenant-level control over how data leaves notebooks. This feature helps ensure that interactive analytics do not inadvertently become channels for data exfiltration. Administrators can restrict the downloading of notebooks, preventing files that may contain sensitive data, credentials, or proprietary logic from leaving the environment. Additionally, they can disable downloads of rich output content, such as table results generated from DataFrames, within the notebook experience. By managing these controls, Fabric admins can effectively prevent unintended data exfiltration from interactive notebook workflows and consistently enforce security and compliance policies across all workspaces and teams. Figure: New data export tenant setting for notebooks Figure: New data export tenant setting enabled What users experience when downloads are blocked When an administrator blocks data export: The Download option is removed from the notebook UI. Users can no longer download notebook files or rich output content generated from DataFrames in the notebook experience. Interactive exploration continues in‑place, but data cannot be extracted outside Fabric through the notebook UI. Figure: Notebook with download controls disabled due to tenant-level enforcement This ensures that notebooks remain a secure analysis surface, rather than a data export mechanism, without disrupting day‑to‑day exploration inside the platform. Why this matters Notebooks often contain more than just code: Embedded datasets Derived analytical results Business logic Confidential insights By controlling export behavior at the platform level, Fabric helps organizations: Reduce risk of accidental data leakage. Meet regulatory and audit requirements. Standardize governance across teams and regions. Data Export Settings for Notebooks reinforce Fabric’s commitment to secure‑by‑default analytics, enabling powerful interactive experiences without compromising enterprise security posture. Session starts insights into Fabric Data Engineering Fast session startup is critical for interactive analytics, and Fabric’s Starter Pools are designed to deliver Spark sessions in ~5 seconds by default. However, when that target isn’t met, users have historically had little visibility into why. Session Start Insights closes that gap by making session acquisition transparent, debuggable, and actionable. Why sessions don’t always start in five seconds In practice, session startup delays are almost always driven by user‑side configurations, not platform regressions. Common causes include: Custom compute configurations that prevent reuse of pre‑warmed Starter Pools. Pre‑installed libraries or environment dependencies that require cluster customization. Managed VNets or private networking that force isolated cluster provisioning. Unexpected high regional demand triggering fallback to on‑demand clusters. What Session Start Insights delivers Previously, users could see that a session was “starting,” but not what was happening under the hood. With this feature, Fabric surfaces clear, explicit reasons for session startup behavior directly in the product experience: Whether the session was served from a Starter Pool or required an on‑demand cluster. The exact reason a fast‑path session could not be used (for example, libraries, networking, or custom configs). Where time was spent during session acquisition. Using the session detail view to diagnose delays Navigate to the notebook’s session status or monitoring pane. Open Session Details for the active or recent session. Figure: Notebook with Session Details option Review the delay reason and session source (Starter Pool vs. on‑demand) Figure: Notebook with Session Details pane showing session start details This makes it immediately clear whether the delay was: Expected due to configuration choices Related to libraries or networking Learn more about session start insights in the Microsoft Learn documentation. Z-order and liquid clustering support in the Native Execution Engine With the Native Execution Engine, Fabric Data Engineering continues to raise the bar on price‑performance leadership for large‑scale analytics. Beyond execution‑time optimizations, the engine now includes native support for Z‑Order and Liquid Clustering, allowing advanced data layout techniques to fully benefit from vectorized, C++‑based execution paths. This ensures that storage‑level optimizations and execution‑level acceleration work together, delivering compounding performance gains for real‑world analytical workloads. Why this matters Modern analytical queries frequently: Filter on multiple high‑cardinality columns. Scan large Delta tables repeatedly. Rely on selective predicates to narrow down results. Without intelligent data layout, even a highly optimized execution engine can spend unnecessary time scanning data. By combining the Native Execution Engine with Z‑Order and Liquid Clustering, Fabric ensures that: Related data is co located on disk, enabling aggressive file and row‑group skipping. Queries scan fewer files and fewer bytes. CPU‑efficient native operators are paired with I/O‑efficient data access. On a one‑billion‑row dataset, internal benchmarks comparing fallback execution versus Native Execution Engine with clustering showed: 20–32 seconds absolute runtime reduction per query. Roughly 20%–27% improvement across multiple clustered column combinations. Performance gains observed consistently across different predicate shapes and data distributions. This brings a compounding performance effect: faster scans, fewer CPU cycles, and lower cost per query, without requiring users to rewrite Spark code or change query semantics. This helps deliver strong price-performance for analytics workloads. How users enable and use this 1. Enable the Native Execution Engine Users must first ensure that the Native Execution Engine is enabled for their Spark workloads (at the workspace, environment, or session level). Once enabled, supported Delta operations automatically run through native execution paths. 2. Use Z‑Order or Liquid Clustering on Delta tables Users can apply clustering using standard Delta Lake commands: Define Liquid Clustering at table creation or apply it to existing unpartitioned tables Use OPTIMIZE … ZORDER BY for multi‑column access patterns To learn more about the Z-Order and Liquid Clustering support or Native engine, refer to the Microsoft Learn documentation. Copilot for data engineering and data science Microsoft Fabric notebooks now include a context-aware Copilot experience designed to support you across the full notebook lifecycle. By automatically understanding your workspace environment—including attached Lakehouses, notebook structure, and runtime behavior—Copilot provides assistance that stays aligned with how your notebook is built and executed. It’s easy to get started with no session startup required. Choose the Copilot icon on the toolbar to open the chat panel. Copilot can help accelerate notebook development by generating and refining code, explaining unfamiliar logic, and assisting with larger notebook workflows. For more complex tasks, Copilot can first propose a plan and then help implement it across the notebook, allowing you to move from idea to working solution more quickly. Copilot also improves the troubleshooting experience when notebook executions fail. Instead of navigating long stack traces or ambiguous error messages, you can use Copilot to analyze failures, identify likely root causes, and review suggested fixes directly within the notebook. Figure: Fix with Copilot provides error summary and suggested fixes Throughout this process, built-in guardrails ensure you remain in control. Copilot suggestions are transparent, and proposed code changes can be reviewed before being applied. Together, these capabilities help teams reduce development friction, resolve issues faster, and build more reliable data workflows. Try the new Copilot experience today. To learn more, visit the Copilot for Data Engineering and Data Science documentation. Fabric notebook custom agent inside VS Code The Fabric notebook custom agent is a Fabric-native AI development agent embedded in the Fabric Data Engineering VS Code extension. It helps data engineers build, debug, and publish Microsoft Fabric notebooks and Spark workloads. Unlike generic coding assistants, this agent operates with full awareness of the Microsoft Fabric workspace, runtime, environments, and Lakehouse resources. It ensures every action—code generation, execution, artifact management, and publishing—is context-aware, validated, and safe for enterprise environments. Prior to the introduction of the Fabric Notebook custom agent within the VS Code extension, there were notable limitations in how language models understood and interacted with the Microsoft Fabric environment. For instance, when users provided a prompt such as "read the parquet file from the current default Lakehouse and save it to a delta table," the language model was unable to interpret what was meant by "default Lakehouse." As a result, it would generate standard Spark code without leveraging the built-in spark variable available within the notebook, which is essential for initializing and managing Spark sessions in the Fabric environment. With this new agent, the following code will be generated and ready to run. # Read parquet file from default lakehouse df = spark.read.parquet("Files/green_tripdata_2022-08.parquet") # Write to delta table in dbo schema df.write.mode("overwrite").format("delta").saveAsTable("dbo.raw_green_tripdata_202208") This custom agent should be automatically activated once the Notebook is open. Figure: Fabric notebook custom agent For more detail, refer to the Author notebook inside VS Code documentation. Tenant switching inside Fabric Data Engineering VS Code extension ISV and partners often collaborate with multiple end customers, each typically operating within their own dedicated Microsoft Fabric tenant. To address this need for flexibility, the Fabric Data Engineering VS Code extension now enables tenant switching. With this enhancement, ISVs and partners can easily transition between different customer projects within the same VS Code window, eliminating the need for repeated sign-in processes. This streamlined experience simplifies managing multiple projects and improves overall productivity for professionals working across diverse customer environments. To switch to a different tenant, select the currently signed-in Fabric user in the status bar and pick the target tenant from the list. Figure: Switch Fabric tenant inside VS Code Enable new kernels inside Fabric Data Engineering VS Code extension Users can now run Fabric notebooks within VS Code using a variety of new kernels. Previously, running notebooks required users to specify the language of each cell using cell magic commands and rely on PySpark as the execution environment. With this enhancement, three additional kernels have been introduced, allowing users to select their preferred programming language directly at the kernel level. This eliminates the need for cell magic commands and streamlines the process, enabling notebooks to be executed in Python, Scala, or Spark SQL natively within VS Code. Choose Microsoft Fabric Runtime from the top-level kernel list. The available languages then appear in the second panel. Figure: Microsoft Fabric Runtime entry Figure: Supported Fabric notebook languages in VS Code For more detail, please refer to the documentation Author notebook inside VS Code. Support for multiple schedules in Fabric materialized lake views MLVs now support multiple named schedules per lakehouse. Previously, all MLVs shared a single schedule, and teams needing different refresh timings resorted to notebook-triggered refreshes. This workaround bypasses dependency management, centralized error reporting, and retry logic; failures can persist for weeks undetected. Each named schedule now targets a specific subset of views. A finance pipeline can refresh hourly while an analytics pipeline runs every six hours, with no scripting required. When a schedule fires, Fabric refreshes upstream dependencies in order, runs independent views in parallel, surfaces errors centrally, and skips overlapping runs. Figure: Schedules panel for materialized lake views, showing configured schedules and available actions For more information, refer to the Schedule a materialized lake view run documentation. PySpark support for Fabric materialized lake views (Preview) MLVs now support PySpark authoring (Preview), letting data engineers create, refresh, and replace MLVs from Fabric notebooks using the DataFrameWriter API. Previously, teams wrote definitions in Spark SQL, which made custom cleansing logic, UDFs for business rules, and procedural transformations harder to express. With PySpark authoring, MLVs gain access to the entire Python ecosystem. A gold-layer MLV can score transactions against a fraud detection model, standardize addresses using a geocoding library, or validate records against external regulatory rules. All existing MLV capabilities, including data quality constraints, table properties, and scheduled refreshes, work identically with PySpark-authored definitions. Full refresh only today; optimal refresh is coming soon. For more information, refer to the PySpark reference for materialized lake views (Preview) documentation. Move data from source to Lakehouse in a few moves using Copy job Getting data into your Lakehouse should be straightforward. For many customers, the first interaction with Microsoft Fabric begins right after creating a Lakehouse and selecting Get data. With this update, Copy job appears at the top of the Get data experience in Lakehouse, making it a more discoverable way to bring data into Fabric. Whether you’re onboarding your first dataset or scaling ingestion across multiple sources, Copy job can help you move data with minimal setup so you can focus on insights instead of configuration. Fabric notebooks now support lakehouse auto‑binding when used with Git, making notebooks far more portable across environments such as dev, test, and prod. Instead of hard‑binding a notebook to a specific lakehouse, Fabric automatically resolves the correct lakehouse as the notebook moves across Git‑connected workspaces, reducing manual rebinding and environment‑specific fixes. This feature is opt‑in and must be enabled from the notebook settings page. Once enabled, it applies to all lakehouses referenced in the notebook, including the default and any additional lakehouses. The configuration is stored in a system‑managed notebook-settings .JSON file in the Git repo, which should not be edited manually. Overall, lakehouse auto‑binding helps teams focus on versioning notebook logic while keeping data and environment configuration cleanly separated. Try it out in just a few steps: Create or open a Lakehouse. Select the Get data dropdown in the ribbon. Figure: Start ingesting data into a Lakehouse directly from the Get data dropdown using Copy job Select New Copy Job You’ll be redirected to the Copy Job experience, where you can choose the source data you want to ingest from. In just a few clicks, your data is copied into the Lakehouse and ready for exploration, analysis, and downstream analytics. Learn more: What is Copy Job in Data Factory – Microsoft Fabric Notebook supports Lakehouses auto-binding in Git Fabric notebooks now support lakehouse auto-binding when used with Git flow, making notebooks more portable across environments such as dev, test, and prod. Instead of hard-binding a notebook to a specific lakehouse in the original workspace, auto-binding lets the notebook automatically resolve the linked lakehouse as it moves across Git-connected workspaces. This reduces manual rebinding and environment-specific fixes. This feature is opt‑in and must be enabled from the notebook settings page. Once enabled, it applies to all lakehouses referenced in the notebook, including the default and any additional lakehouses. Figure: Entry of auto-binding setting in notebook The configuration is stored in a system‑managed ‘notebook-settings .json’ file in the Git repo. Overall, lakehouse auto‑binding helps teams focus on versioning notebook logic while keeping data and environment configuration cleanly managed. Notebook Resources Folder Support in Git Notebook projects often depend on more than just notebook code—such as reusable Python modules, configuration files, or small supporting assets. Fabric notebooks now support committing the built‑in Resources folder to Git, enabling true end‑to‑end source control for notebook‑based projects. These resources are versioned alongside the notebook and automatically restored during Git sync. To support real‑world workflows, this feature includes fine‑grained controls. Teams can define Git exclusion rules or use standard .gitignore files inside the Resources built in folder to avoid tracking large files, temporary assets, generated outputs, or test data. Figure: Define resources in git settings in notebook The feature is disabled by default to ensure safe adoption and does not introduce noticeable performance impact during commit or sync. The support for Environment resources folder, deployment pipelines, and public APIs is coming soon. Learn more: Notebook source control and deployment - Microsoft Fabric Fabric notebook public APIs (Generally Available) Fabric Notebook Public APIs enable notebooks to be managed and executed programmatically as first‑class assets. The APIs provide full CRUD support—enabling teams to create, update, list, and delete notebooks at scale—making them ideal for CI/CD and automated environment management. In addition, notebooks can be executed on demand via the Job Scheduler API. You can parameterize notebook runs, customize session configuration, specify environments and lakehouses, monitor execution status, and cancel runs if needed. Secure service principal authentication is also supported. A key enhancement is the ability for notebook runs to return exit values, enabling conditional branching and richer orchestration in pipelines. Together, these APIs unlock seamless integration with Fabric pipelines, external schedulers, and enterprise automation platforms. Learn more: Items - REST API (Core) and Job Scheduler - REST API (Core). Improved Copilot completion for Fabric notebooks We’re introducing upgraded Copilot completion in Fabric notebooks to deliver a faster, more accurate, and more intuitive coding experience. With this update, auto-completion is closer to what developers expect from VS Code‑style inline suggestions, helping you stay in flow while writing notebook code.You can enable the feature from the Copilot completion button in the notebook status bar. It supports both Python and PySpark notebooks. Figure: How to enable copilot completion A More Natural, Inline Coding Experience The upgraded auto‑completion is designed to work inline as you type, offering context‑aware code suggestions that better match your intent. Whether you’re writing Python logic, data transformations, or helper functions, Copilot now provides suggestions that feel more predictable, relevant, and easy to accept—reducing friction compared to earlier experiences. Faster and More Responsive Performance has been a key focus of this upgrade. Auto‑completion now responds more quickly, reducing latency between keystrokes and suggestions. This makes Copilot feel less intrusive and more like a natural extension of the editor, especially during rapid iteration or exploratory development. Higher‑Quality Suggestions That Fit Notebook Workflows Beyond speed, the quality of suggestions has improved. Copilot is better at understanding notebook context, including surrounding cells and in‑progress code, resulting in completions that require less manual editing. The goal is simple: help you write correct, readable code with fewer interruptions and less back‑and‑forth. Designed for Everyday Notebook Development This upgraded auto‑completion brings Fabric notebooks closer to the editing experience developers are already familiar with, while remaining optimized for data engineering and analytics workflows. Learn more by exploring Develop, execute, and manage notebooks - Microsoft Fabric. Create files in the notebook resources folder Fabric notebooks now let you create and manage files directly in the built‑in Resources folder, making it easier to develop and maintain notebook dependencies. You can create and edit Python modules, configuration files, and other lightweight assets alongside your notebook code and use them directly within the notebook. Figure: Entry of creating new file in notebook resources folder To learn more, refer to How to use notebooks - Microsoft Fabric. Data Science and AI Fabric data agents (Generally Available) Data sources: Build and consume data agents on a broad set of data sources, including Lakehouse, Warehouse, semantic models, Eventhouse, SQL databases, and mirrored databases. Configurations: Configure data agents using agent-level instructions, data source–specific instructions, and example queries to tailor behavior to your scenarios. Publish and share: Publishing and sharing data agents within Microsoft Fabric is generally available, making it easier to operationalize and collaborate on data agents. Figure: End-to-end data management workflow from creation through consumption This release also includes diagnostic, Git integration, and deployment pipelines as part of Microsoft Fabric’s Application Lifecycle Management (ALM) capabilities, enabling troubleshooting and lifecycle management of your agents! Advanced security and governance in data agents (Preview) Data agents in Microsoft Fabric (Preview) include capabilities that elevate security and governance standards. Through integration with Microsoft Purview, organizations gain access to comprehensive auditing, eDiscovery, data lifecycle management, communications compliance, and classification. These tools capture prompt and response telemetry along with user context, supporting enterprise protection and regulatory compliance. Additionally, we are introducing outbound access protection support for the Data Agent artifact to help mitigate sensitive data exfiltration risks and adhere to strict security policies at the individual workspace level. With these updates, organizations can monitor, control, and safeguard all data agent interactions more effectively. Data source enhancements for data agents (Preview) The latest preview brings significant enhancements to data agent source capabilities in Microsoft Fabric. Users can now connect Graph as a data source, enabling them to model and analyze complex relationships within their data for richer, AI-driven insights. Additionally, support for KQL user-defined functions (UDFs) and SQL functions is available, allowing for more sophisticated and efficient querying in KQL- and SQL-enabled sources. These enhancements make data agents more flexible and powerful, supporting faster analytics and expanded scenario coverage. Insider Risk Management PAYG Usage Report (Generally Available) Multimodal support for AI functions in Fabric enables notebook users to apply AI capabilities directly to their unstructured data—including PDFs, images, and text files. With just a few lines of code, users can perform tasks such as summarization, classification, sentiment analysis, and more, all within their existing workflows. This capability is designed to work across both pandas and Spark, making it easy to bring AI-driven insights to a wide range of data science and analytics scenarios in Fabric. Figure: Load files into a table or classify insurance claim images with multimodal AI functions Check out the multimodal AI functions documentation to learn more. AutoML in Fabric (Generally Available) AutoML delivers a fully production-ready, low-code machine learning experience. In addition to the core AutoML capabilities powered by FLAML—automated model selection, feature engineering, and hyperparameter optimization—the end-to-end UI experience, making it easy to configure experiments, monitor training progress, compare models, and deploy the best performer directly from the interface. With integrated experiment tracking, reproducibility, and seamless deployment workflows, teams can confidently move from raw data to high-quality predictive models faster—while maintaining transparency, governance, and control within Fabric. Figure: AutoML includes the fully integrated UI experience for configuring experiments, tracking model performance, and deploying models end to end Check out the AutoML in Fabric documentation to learn more. Data Warehouse Fabric Data Warehouse recovery (Preview) In fast moving production environments, an item gets dropped accidentally due to an incorrect script. Suddenly, critical reports are broken, and teams are asking the same question: “How fast can we recover?” With dropped warehouse recovery in Microsoft Fabric, a deleted warehouse no longer means starting over. You can now restore a dropped warehouse together with everything that goes with it—data, schemas, snapshots, permissions, and saved queries—in minutes, without rebuilds, re‑ingestion, or complex restore workflows. Figure: Warehouse recovery in action—from drop to restore in minutes There’s no need to recreate environments, rerun pipelines, or scramble through backups. Recovery is simple, predictable, and designed to bring your warehouse back exactly as it was before the drop. This capability is built for the realities of modern analytics: rapid iteration, frequent deployments, and shared production environments. Instead of turning accidental deletes into prolonged outages, Fabric makes recovery a routine, low-stress operation. No panic. No need to rebuild. Just built in resilience—designed for real-world production analytics. To learn more, refer to the manage workspaces documentation. Alerts and actions Microsoft Fabric Data Warehouse provides operational intelligence closer to the data by integrating SQL queries with Fabric Activator rules. Traditionally, identifying an issue in query results is only the first step. Teams then need separate systems or manually follow up to notify the right people and act. With this integration, Data Warehouse makes it possible to define rules directly from SQL query outputs, so changes in data can trigger alerts and downstream actions automatically. This unlocks a simpler way to monitor business-critical conditions using familiar SQL workflows. Teams can create queries that detect scenarios such as SLA risks, failed processes, unusual trends, or threshold breaches, then attach rules that evaluate results continuously and respond in real time. The result is a more proactive analytics experience, where insights move beyond the warehouse and are acted on immediately. Figure: Create rules on SQL query results to detect data issues, monitor KPIs, and automatically trigger alerts or Fabric workflows. Analyze unstructured text using T-SQL AI functions (Preview) Microsoft Fabric Data Warehouse extends modern analytics beyond structured and semi‑structured data by introducing built‑in AI functions for working directly with the unstructured text. Traditionally, processing free‑form content such as notes, logs, or comments requires external services or complex pipelines. With these new capabilities, Fabric Data Warehouse enables text extraction, classification, sentiment analysis, and transformation directly in T‑SQL language, allowing data engineers and analysts to keep AI‑driven text processing inside the warehouse. The new AI functions simplify common text analytics scenarios using familiar SQL patterns. You can extract structured insights from unstructured text, analyze sentiment in feedback or messages, and classify content such as application logs or incident reports using contextual understanding rather than fragile rules or expressions. Fabric Data Warehouse also supports text transformation scenarios, including summarization, grammar correction, and translation, making it easier to standardize and enrich text data as part of existing data preparation workflows. The following visual is example of processing unstructured text in the Comments table: Figure: Analyzing comment text with AI functions This query enriches each user's comment by determining its sentiment; labeling the type of feedback or intent; and extracting key discussion signals such as the main topic, user intent, and any requested action using built-in AI functions. For advanced scenarios, Fabric Data Warehouse enables custom prompt-based processing through a generic ai_generate_response(instructions, text) function. This function allows teams to define precise transformation or extraction rules as prompts; apply domain specific logic; and reuse AI behavior consistently across queries and pipelines. Together, these capabilities significantly broaden the scope of data warehousing scenarios supported in Fabric, unlocking new ways to analyze and operationalize unstructured text using T-SQL. Refer to AI functions in Fabric Data Warehouse to learn more about analyzing text with built-in AI functions. ANY_VALUE aggregate Fabric Data Warehouse provides the ANY_VALUE() aggregate, which lets you return an arbitrary value from each group in T-SQL query. This is especially useful when you need to group results by a key (such as GeographyID) but you still want to project descriptive attributes (such as city name, and country) that are functionally the same for every row in that group. An example of such a query is illustrated in the following picture, demonstrating how ANY_VALUE() aggregate can be used to get the values from the group that are not changing. Figure: Using ANY_VALUE() to project descriptive columns while aggregating trips by GeographyID In this pattern, city, state, and country don’t add meaning to the aggregation because they’re constant for a given GeographyID. Adding these columns in the GROUP BY clause or applying more complex or costly aggregates like MIN or MAX is unnecessary overhead and makes queries harder to read and maintain. ANY_VALUE keeps the grouping logic minimal and the intent clear: aggregate by the key and simply carry through the descriptive columns. Refer to the ANY_VALUE function in Fabric Data Warehouse documentation to find additional scenarios where it helps simplify grouping and aggregation logic. Fabric warehouse custom SQL pools (Preview) Custom SQL pools for Fabric Data Warehouse gives administrators finer-grained control over how SQL compute resources are allocated across workloads. Custom SQL pools build on the warehouse’s autonomous workload management by letting you define your own isolation boundaries, explicitly assign resources, and route queries based on application context. With Custom SQL Pools, you can create multiple isolated SQL pools within a single workspace and allocate a percentage of available compute to each. Queries are routed to the appropriate pool ensuring that critical workloads get the resources they need without being impacted by other activities in the warehouse. Figure: Custom SQL Pool Configuration Key benefits include: Predictable performance for critical workloads—Reserve compute for business‑critical reporting or dashboards, so they aren’t disrupted by ad‑hoc queries or background processing. Flexible workload isolation without added complexity—Allocate resources where they matter most without needing to split workloads across multiple workspaces or scale capacity just to protect one workload. Custom SQL Pools are especially useful when multiple applications share a single Fabric warehouse or SQL analytics endpoint and have different performance or priority requirements. As your capacity scales up or down, your pool allocations automatically scale with it, preserving the relative resource distribution you’ve defined. Learn more about custom SQL pools in Fabric Data Warehouse: Custom SQL Pools - Microsoft Fabric | Microsoft Learn. SQL Audit Logs (Generally Available) SQL Audit Logs for Fabric Data Warehouse enable organizations to capture and analyze database activity for security monitoring, compliance, and forensic analysis. Figure: Configuring SQL Audit Logs With this release, we are expanding support and improving accessibility: Support for SQL Analytics Endpoint auditing. Direct access to audit files stored in OneLake, through OneLake Explorer. Ability to download or copy audit files through OneLake Explorer. Ability to open the .xel audit files directly in SQL Server Management Studio (SSMS) for deeper investigation. These capabilities make it easier for security and compliance teams to perform detailed investigations, long-term retention, and external analysis of workflows. For configuration steps and usage details, see the documentation: SQL Audit Logs in Fabric Data Warehouse - Microsoft Fabric | Microsoft Learn COPY INTO and OPENROWSET support for OneLake sources (Generally Available) Previously, this capability supported Lakehouse sources only. With this release, we are expanding support to all OneLake items (except Warehouses). This enables much more flexible ingestion scenarios, including: Using partner workloads such as COPY Jobs. Using staging areas across different Fabric items. Loading data stored anywhere in OneLake-backed items. Customers can now leverage OneLake as a unified staging layer for ingestion workflows while maintaining a consistent SQL experience. Figure: Executing COPY INTO from OneLake sources For full usage examples and configuration guidance, see the documentation: Ingest Data into Your Warehouse Using the COPY Statement - Microsoft Fabric | Microsoft Learn COPY INTO (Transact-SQL) - Azure Synapse Analytics and Microsoft Fabric | Microsoft Learn Outbound Access Protection (OAP) support for Warehouse (Generally Available) Outbound Access Protection for Fabric Data Warehouse provides stronger data exfiltration protection for enterprise environments. Warehouse now supports connector rules that allow organizations to control which external sources the warehouse can access. Customers can define rules to allow access to: Specific Azure Data Lake Storage Gen2 accounts Other Fabric workspaces Approved external connectors This expands the model introduced during Preview, where access was limited to OneLake and local workspace sources only. With connector rules, organizations can enforce controlled and auditable outbound connectivity, helping meet strict governance and compliance requirements. Figure: Supporting OAP Data Connection Polices For details on configuring connector rules, see the documentation: Workspace outbound access protection for data warehouse workloads - Microsoft Fabric | Microsoft Learn Full query text available in Query Insights Query Insights now show the full SQL query text, removing the previous 8,000‑character truncation. The complete query text is available in: The command column of queryinsights.exec_requests_history. The Query Details pane in the Query activity tab. This makes it significantly easier to understand what ran, especially for large, auto‑generated queries from BI tools, ORMs, or complex workloads. You can now retrieve the full query text directly using: SELECT distributed_statement_id, submit_time, total_elapsed_time_ms, command FROM queryinsights.exec_requests_history ORDER BY submit_time DESC; This query takes the most frequently executed query (from Frequently Run Queries) and pulls every historical execution with the full, untruncated SQL text, making it easy to understand exactly what is running and how often: WITH TopQuery AS ( SELECT TOP 1 query_hash FROM queryinsights.frequently_run_queries ORDER BY number_of_runs DESC ) SELECT erh.query_hash, erh.distributed_statement_id, erh.submit_time, erh.total_elapsed_time_ms, erh.status, erh.allocated_cpu_time_ms, erh.data_scanned_remote_storage_mb, erh.command AS full_query_text FROM queryinsights.exec_requests_history AS erh JOIN TopQuery AS tq ON erh.query_hash = tq.query_hash ORDER BY erh.submit_time DESC; Because the full statement is preserved, users can: Immediately understand what logic was executed, not just which query ran. Compare query text across executions to detect subtle changes or regressions. Correlate performance issues with specific joins, filters, or aggregations. This enhancement removes a major gap between observability and action, making Query Insights a more complete tool for day-to-day production troubleshooting. Live connectivity in Migration Assistant for Fabric Data Warehouse (Preview) The live connectivity in Migration Assistant for Fabric Data Warehouse lets you migrate object metadata by connecting directly to your source system into a new Fabric warehouse. This helps you accelerate migration and reduce upfront prep by eliminating the need to generate and upload a DACPAC for the metadata step. The object metadata of schemas, tables, views, functions, and stored procedures gets migrated to warehouse. Figure: Migration using direct connection to the source system Learn more about Migrate with a Direct Connection. Simplify data access with data sources (Generally Available) Fabric Data Warehouse lets you define external data sources that act as named references to locations in your lake (for example, a root folder of a Fabric Lakehouse or an Azure Storage account). External data sources (Preview) were introduced in preview in October 2025, now they are generally available and fully integrated into the Fabric experience, with the full IntelliSense and Copilot support in the SQL query editor. The following visual shows how to create a reusable reference to the Fabric Lakehouse root folder that represents a landing zone where you can store the files before ingesting them in warehouse: Figure: Creating external data source in Fabric Data Warehouse. In the following visual, you can see how you can query files using short, relative paths that are resolved against the data source root: Figure: Accessing files in the referenced data source using the relative path Using a data source keeps queries clean and portable. You can write easy to remember relative paths (like /Files/bronze/logs/*.jsonl) instead of embedding long, environment-specific URLs everywhere in the code. This makes scripts simpler to maintain and easier to share across workspaces and environments. Find more examples about the reading files form the lake in OPENROWSET(BULK) (Transact-SQL) documentation page. Real-Time Intelligence Business Events in Microsoft Fabric (Preview) With Business Events, organizations can move from observing what happened to acting on what matters, in real time. It enables organizations to respond faster, operate more intelligently, and scale real-time decision making across analytics, automation, and AI. You can generate business events from user data functions (UDFs) and notebooks. Once generated, a single business event can power multiple downstream actions, such as: Trigger alerts and automations with Activator, responding immediately via email or Teams. Execute custom logic using user data functions, reacting programmatically to business events. Run analytics and workflows in notebooks, using events to drive downstream analysis. Provide real‑time context with AI and ML, enriching models with governed business signals. Integrate with Spark jobs, dataflows, and Power Automate, enabling distributed processing and business process automation. With Business Events in Real-Time Hub, you can explore, define, and act on critical business signals for the whole organization in a unified experience. Figure: Business Events creation experience For more information about this feature, please refer to the documentation: Business Events in Microsoft Fabric. Building event-driven, real-time applications on database changes with Fabric Eventstreams Deltaflow (Preview) Building intelligent systems that react quickly to operational database changes is simpler with this update. With the release of DeltaFlow, Fabric Eventstreams can seamlessly capture inserts, updates, and deletes from operational databases, transform them from their raw Debezium format, and make them available to downstream event-driven applications using Activator and for real-time analytics in Eventhouse. There’s no need for custom Debezium/JSON processing code or to explicitly manage destination tables through source table schema changes. Easily connect to, ingest from, and transform raw CDC feeds into analytics-ready form. to_connect_to_a_CD Figure: Enabling DeltaFlow when connecting to an Azure SQL database Detect, fetch and register source database & table schemas in the Eventstream schema registry as they evolve. Figure: Automatic registration and use of source table schemas Automatically manage tables in analytics store as they continuously evolve with source schema changes without breaking pipelines. Figure: Automatically created Eventhouse tables with analytics-ready shapes For more information about these features, please refer to the document Building real-time, event-driven applications with Database CDC feeds and Fabric Eventstreams DeltaFlow (Preview). Real-time stream processing with Fabric Eventstreams and Spark notebooks (Preview) This update brings together Fabric Eventstreams and Spark Structured Streaming, making it easier for Spark developers and data engineers to work with real-time data in Microsoft Fabric. These enhancements enable you to access streaming data in Eventstreams directly from Spark notebooks, supporting low-latency processing and end-to-end real-time AI pipelines. Easily discover Eventstreams and real-time sources available through the Real-Time Hub, right from within Fabric notebooks. Figure: Real-Time Hub view inside Fabric notebook—discover Eventstreams in seconds Connect to and process streaming data within minutes using auto-generated PySpark code snippets. Figure: Auto-generated PySpark snippet in a Fabric notebook for an Eventstream Load and use existing notebooks from the Fabric Eventstreams portal. Figure: Load a Spark notebook as an Eventstream destination—reuse and collaborate Securely connect to any Eventstream from a Fabric Spark job without connection strings and secrets, using the enhanced Spark adapter for Eventstreams. Securely connect to any Eventstream from a Fabric Spark notebook using the enhanced Spark adapter—without connection strings or secrets, and with built-in auto-retry support. For more information about these features, please refer to the blog post Bringing Together the world of Real-time Intelligence and Spark Structured Streaming (Preview). Anomaly Detector full-item experience Introducing a refreshed Anomaly Detector full‑item experience that makes it easier to create, run, and explore anomaly detection workflows from end to end. Instead of working through disconnected steps or modal flows, you now get a single, full‑page canvas that brings configuration, analysis, and results together in one place. The updated layout follows Fabric’s shared item experience, so navigation and interactions feel consistent with the rest of the platform. With this new experience, you can more quickly define detection scenarios, run analyses, and immediately dig into detected anomalies without losing context. Clearer sectioning and action cues guide you through the process—from selecting signals and parameters to reviewing anomaly trends and individual data points. Results stay visible alongside configuration, making it easy to iterate, compare outcomes, and refine your setup in real time. Figure: An updated Anomaly Detection UI that simplifies setup, visualizes trends and anomalies in real time, and highlights high‑confidence events for faster investigation The full item Anomaly Detector experience also sets the stage for deeper investigation and richer insights. By consolidating analysis and results into a unified view, you can spend less time navigating and more time understanding what’s changing in your data. Whether you’re monitoring operational metrics or exploring unexpected behavior in real-time signals, these improvements help you move from detection to insight faster and with greater confidence. Operations agent playbook improvements and messages This month sees several improvements for operations agents’ ability to monitor your data and take actions. Based on usage and feedback we’ve heard so far, operations agents are better at mapping between your instructions and the fields in the Eventhouse you connect them to. You’ll also see they can build different types of rules to monitor the specific conditions in your data, including comparing string values in the data and counting data points over time. Finally, you’ll also see better messages if the agent can’t generate a playbook based on the data, goals, and instructions you’ve configured. In those cases, the LLM will try and describe the issue for example not being able to ground a field it inferred from your instructions to a field in the Eventhouse KQL database, or the parameters for a condition or action not being clear in the instructions. For example: This makes it easier for you to debug and unblock the agent configuration. Finally, we’ve updated our best practices and sample for how to give guidance and steering to the operations agent for it to follow your instructions. Learn more about this in the Operations Agent Best Practices and Limitations documentation. Live update for Real-Time Dashboards Real-Time Dashboards now support Live update, a feature that automatically refreshes dashboard visuals when new data is ingested into your underlying data sources. Instead of relying on fixed-interval refresh - which polls your data source on a set schedule regardless of whether new data exists - Live update uses a lightweight background query to detect when data arrives and triggers a refresh only when needed. This event-driven approach offers several benefits. Your dashboards stay current without the compute overhead of constant polling, making it particularly valuable for high-frequency data monitoring scenarios where you need to see data the moment it arrives. For organizations running multiple dashboards or monitoring large data volumes, Live update reduces compute load by eliminating unnecessary refresh cycles during quiet periods. Dashboard viewers also gain flexibility with the ability to pause live updates temporarily. If you're investigating a specific data point and don't want the visuals to change, you can pause updates to analyze the current state without interruption, then resume when you're ready to return to real-time monitoring. Dashboard editors can enable Live update through the dashboard settings, with configuration options including Live update (recommended), manual update only, or a fallback refresh interval for visuals that don't support ingestion detection. Learn more about configuring Live update for your dashboards, with the What is Real-Time Dashboard? documentation. Eventstream SQL Operator (Generally Available) During preview, the Eventstream SQL Operator introduced SQL-based stream processing in Fabric, enabling customers to transform live event data using familiar SQL with rich authoring, preview, and debugging capabilities. Write to multiple destinations from a single SQL operator Consolidate your real-time processing logic into one streamlined SQL block and route results to multiple destinations in a single step. This simplifies pipeline design, making it more efficient and lowers operational overhead. The updated authoring experience makes it easy to add multiple destinations directly within the SQL editor and preview results for each output independently. During testing, dedicated output previews let you validate transformations before you publish. Figure: Route data to multiple destinations from one SQL operator. Event ordering and late event arrival handling Configure event ordering policies directly within the SQL operator to handle late-arriving and out-of-order events. Define thresholds for how long to wait for delayed data and ensure accurate, event-time–correct processing—even in the presence of network delays or asynchronous producers. These policies help build more resilient real‑time pipelines that reflect how data behaves in the real world—not just in perfect conditions. Learn more about Fabric Eventstream SQL Operator. Together, these enhancements make Eventstream SQL Operator more powerful, more intuitive, and ready for production‑grade real‑time workloads. Anomaly Detection as a source in Eventstream Anomaly Detection can be added as a source in Fabric Eventstream, allowing you to publish anomaly events directly into your Eventstream for processing and action. You can add Anomaly Detection as a source from either Eventstream or Real-Time Hub. You can enrich your anomaly events by adding business context and additional information. Further, you may route real-time events to downstream workloads for automated alerting and dashboard visualization. Where to Add This Source You can add Anomaly Detection as a source in two ways: From Eventstream – Create a new Eventstream, select Anomaly detection events as a source. Figure: Adding Anomaly Detection source within Eventstream From Real-Time Hub – Navigate to Real-Time Hub, find the Fabric events and select Anomaly detection events. Figure: Adding Anomaly Detection events in Real-Time Hub Once added, anomaly events flow seamlessly into your Eventstream, ready for transformation and routing to downstream workloads. Get Started Anomalies are now another streaming event—ready to be transformed, enriched, and acted upon. Try out Anomaly Detection as a source in Eventstream today and unlock the power of real-time anomaly pipelines. Learn more about this in the Operations Agent Best Practices and Limitations documentation. Data series colors for real-time dashboard visuals When monitoring operational data, color choices matter. A status indicator showing “Critical” in red and “Healthy” in green communicates meaning instantly—viewers can interpret the visual without reading legends or labels. With data series colors, you can make these intentional choices rather than accepting system defaults. To configure data series colors, switch to Editing mode, select the Edit icon on your tile, and expand the Data series colors section in the Visual tab of the formatting pane. From there, you can select a color for each data series in your visual. Figure: Visual formatting: data series colors setting Figure: Color palette for setting visual elements colors Learn more about customizing your Real-Time Dashboard visuals, refer to the Customize Real-Time Dashboard visuals documentation. Use Copilot to create visuals in real-time dashboards (Preview) Dashboard editors can now use Copilot to create and edit visuals in Real-Time Dashboards using natural language. When you're in Edit mode, open the Copilot pane while creating a new tile or editing an existing one. Describe the insight you need - for example, "Show me the top 10 repositories by push events this week" - and Copilot generates the KQL query, returns the data, and suggests a visual that fits your results. Figure: Real-time dashboard visual in edit mode after Copilot answer has been applied You can accept Copilot's suggestion, refine your question with follow-ups like "Group by event type" or "Filter to the dotnet organization," or edit the query directly. Once you're satisfied, add the visual to your dashboard and use the no-code formatting options to customize its appearance. To learn more, refer to the Copilot-assisted real-time data exploration documentation. Instantly run and preview functions in Microsoft Fabric Eventhouse: no code required (Preview) Previously, working with an Eventhouse function involved manually writing KQL queries. You needed to enter the function name, provide parameters in the right format, and execute the query just to see what results you would get. If you wanted to view the function's body or metadata, you had to run a separate command. That's no longer the case. With the new Preview Functions capability in Microsoft Fabric Eventhouse, you can open the function definition, run the function, and instantly preview its results, with no manual KQL, no parameter guesswork, and no extra commands. Why this matters Eventhouse functions are powerful, but environments evolve. Databases grow, teams change, and you often inherit functions you did not write. Instead of guessing what a function does or manually building a query just to test it, you can: View the function definition instantly. Run the function and preview results with a single click. Test parameterized functions interactively. Browse your function list with search and sorting. This removes friction from everyday workflows. Whether you are exploring unfamiliar logic, validating outputs before building reports, or troubleshooting unexpected results, you get clarity in seconds instead of minutes. How to view or preview a function: In DB Explorer, expand Functions and select a function. A read-only version of the function opens. Select Preview results to instantly run the function and see the output. If the function has parameters, enter your values and preview the results based on your input. The preview shows up to 100 records, providing a quick snapshot of the function’s output. Figure: DB Explorer with the Functions folder expanded and a function selected. The function opens in read-only mode, and the Preview results option is available to run the function and display the output, with fields provided to enter parameters You can view a complete list of all stored functions, including their folder, description, and optional sorting. Built-in search makes it easy to find specific functions, making navigation and discovery simple even in large databases. Figure: Functions list with all available functions in the database, with options to sort, search, and open a menu with additional actions The new Run & Preview Functions feature in Microsoft Fabric Eventhouse lets you instantly inspect function definitions and preview results without writing KQL or handling parameters manually. Quickly explore, test, and manage all your stored functions in one place, saving time and reducing friction. Learn more with the Stored functions list documentation. Workspace monitoring dashboard templates in Microsoft Fabric Eventhouse (Preview) Fabric workspace monitoring provides rich telemetry across your workspace assets, including Eventhouses, Power BI Semantic Models, Data Engineering (GraphQL), and Mirrored Databases. The workspace monitoring data is stored in an Eventhouse, part of Fabric Real-Time Intelligence. To help you turn this data into actionable insights, we have created ready-to-use real-time dashboard templates with out-of-the-box visualizations. Currently, two templates are available: one for Eventhouse items and one for semantic models. From any Workspace Monitoring Eventhouse, users can create these dashboards directly. To get started, go to your Workspace Monitoring Eventhouse, open the upper ribbon, and select Fabric Monitoring. Figure: The ribbon in the Workspace Monitoring Eventhouse allows you to create out-of-the-box dashboards for monitoring From there, choose to create: Eventhouse Monitoring Dashboard—to monitor Eventhouse items in your workspace. You can track: Ingestion results and logs. Commands and queries monitoring. Metrics related to Eventhouse performance. Semantic Model Monitoring Dashboard—to monitor Semantic Models in your workspace. You can use semantic model logs to: Identify periods of high or unusual Analysis Services engine activity by capacity, workspace, report, or user. Analyze query performance and trends, including external DirectQuery operations. Track semantic model refresh durations, overlaps, and processing steps. Monitor custom operations sent using the Premium XMLA endpoint. Once created, the dashboards are ready to use immediately, giving you instant visibility in your workspace. You can also customize them to fit your specific needs. These templates make it fast and easy to track, analyze, and act on workspace activity, all in one place. Learn more in the documentation: Visualize your workspace monitoring . Databases Database Hub in Fabric The Database Hub in Fabric is a new unified database management experience that brings databases across edge, on‑premises, cloud, and Fabric into a single, coherent view. It provides teams with one place to explore, observe, govern, and optimize their entire database estate. Built for scale, the Database Hub uses agent-assisted intelligence to continuously reason over estate-wide signals, surface what changed, explain why it matters, and guide teams toward the right next actions. With built-in observability, delegated governance, and Copilot-powered insights, database agents help teams move from insight to action faster, while humans remain firmly in control of goals, boundaries, and trust. The result is a simpler, more confident way to manage databases at scale today, and a foundation for increasingly autonomous, intelligent database operations over time. Sign up for early access. SQL database in Fabric Since reaching general availability in November 2025, SQL database in Fabric has seen rapid customer adoption as organizations modernize SQL workloads with less operational overhead and tighter integration with analytics and AI. Guided by customer feedback, the platform emphasizes simplicity, autonomy, security, and AI optimization. We are introducing a set of improvements and new features that make it easier to migrate, manage, and optimize SQL workloads in Fabric: Simplified migration with new assistant: The Migration Assistant in public preview helps SQL developers move SQL Server and Azure SQL workloads into Fabric by importing schemas, assessing compatibility, and guiding migration with minimal manual effort. Configurable autonomous management: While maintaining a SaaS-first approach, new options allow database-level control over vCore scaling, expanded compatibility levels, enhanced T-SQL features, and settings that ease application transitions without code changes. Support for all collations: All Azure SQL database collations are supported when creating a new SQL database in Fabric for enhanced global data compatibility and app development flexibility. Collations control text sorting and comparison in SQL databases, impacting filtering, searches, and multilingual content management. Users can specify collations seamlessly during database creation via the REST API across deployment methods. Check out the How to set a different collation for SQL database in Fabric demo and explore the sample code in Git repo. Enhanced data mirroring and security: Auditing and Customer Managed Keys are generally available. CMK for Fabric SQL lets you encrypt databases with your own Azure Key Vault keys to gain full control over key ownership, access, rotation, and compliance. Users can selectively manage which tables are mirrored to OneLake for immediate analytics access. AI and monitoring integration: SQL database in Fabric supports vector search with DiskANN and integrates with Azure AI Foundry for advanced semantic search and AI scenarios, alongside workspace performance dashboards for unified monitoring and optimization. Enhanced data recovery: In Fabric, when a database is deleted, it goes into a soft-deleted state into the Fabric Workspace’s Recycle Bin tab. Depending on the retention configured, the deleted database can be recovered from the Recycle bin while in retention. In addition to this Recycle bin experience, the Fabric SQL database also has the backup retention period configurable from 1-35 days. When the database is hard deleted from the Recycle bin, the backups are still available for the configured backup retention period. This new improvement allows you to restore the backup into a new database to any point in time within the restorable period. Cosmos DB mirroring with Private Link and VNET Cosmos DB mirroring with Private Link and VNET enables customers to mirror data from privately secured Azure Cosmos DB accounts into OneLake. This allows organizations to maintain consistent network security and compliance while supporting near‑real‑time analytics and AI workloads in Microsoft Fabric—strengthening Fabric’s enterprise readiness by design. Figure: Mirroring data from Azure Cosmos DB accounts secured with Private Endpoints or VNETs into OneLake To learn more, refer to the Cosmos DB Fabric Mirroring for Private Networks documentation. Data Factory—Copy Job Richer Change Data Capture (CDC) with Oracle, Fabric DW, and SCD Type 2 Copy job continues to improve the no‑code CDC experience with richer, enterprise‑ready replication patterns. This release introduces a set of enhancements in Copy job in Microsoft Fabric Data Factory that make CDC replication more powerful and easier to use without writing code: Oracle CDC source—Capture changes directly from Oracle databases. Fabric Data Warehouse sink—Replicate CDC data into Fabric Data Warehouse. SCD Type 2—Preserve full history with valid dating, and handle deletes as soft deletes. With built‑in SCD Type 2 and soft delete handling, Copy job automatically preserves every version of a record as it changes over time, instead of overwriting history. This makes it easy to answer point‑in‑time questions, support regulatory audits, and run accurate historical analytics—capabilities that traditionally require complex MERGE logic or custom code. Figure: Enabling SCD Type 2 in Copy job with One Click Learn more in the Change data capture (CDC) in Copy Job documentation. Every row is traceable with built-in audit columns Audit columns are additional metadata columns that Copy job can automatically append to every row it writes to the destination. These columns don't come from your source data—they're generated by the platform to describe the data movement itself. When you enable audit columns in Copy job, each row in your destination table can be enriched with information such as: Audit Column What It Captures Data extraction time The timestamp when the row was extracted from the source by a Copy job run File path The source file path the row was read from (applicable for file-based sources) Workspace ID The Fabric workspace ID where the Copy job resides Copy job ID The unique identifier of the Copy job item Copy job run ID The unique identifier of the specific Copy job execution Copy job name The name of the Copy job that moved the row Lower bound The lower bound value of the incremental window for the current run Upper bound The upper bound value of the incremental window for the current run Custom A user-defined static value—add any additional context your team needs. For example, you can add your source server name here Table: Audit column list With audit columns enabled, you can answer the following questions for any row in your destination table: When was this data extracted? Exact timestamp from when the row was read from the source. Where did it come from? Which file path, which data store. Which job moved it? Which Copy job from which Workspace, which specific run, by name and ID. What was the incremental scope? Lower and upper bounds tell you exactly what slice of data this run covered. No custom code. No expression authoring. Add as many audit columns as you want, and every row in every table your Copy job writes will include this metadata automatically. Figure: Setup audit column in Copy job. Figure: Output on destination data Learn more in What is Copy job in Data Factory - Microsoft Fabric. Workspace Monitoring for Faster, Scalable Troubleshooting As Copy jobs scale from a handful to hundreds, visibility becomes critical. Fabric Workspace Monitoring brings centralized, log‑level observability to Copy job executions, streaming detailed run data into a query able Monitoring Eventhouse inside your workspace. Teams can analyze failures, throughput, duration, and data volumes across all Copy jobs in one place—without inspecting jobs individually. With historical logs, cross‑item correlation, and integration with Data Activator for alerts, Workspace Monitoring helps DataOps teams detect issues earlier and troubleshoot faster at scale. Figure: PBI Report against Fabric Workspace Monitoring metric from Copy job Learn more in Workspace Monitoring for Copy Job in Microsoft Fabric - Microsoft Fabric Boost performance automatically with AutoPartitioning Moving large tables efficiently often requires careful partition tuning—but Copy job now does this automatically. With auto‑partitioning, Copy job detects large datasets and applies an optimal parallel read strategy without any manual configuration. This delivers dramatically higher throughput out of the box, whether you’re copying millions or hundreds of millions of rows. The system adapts dynamically based on data size and source characteristics, ensuring consistent performance across tables while eliminating per‑table tuning effort. Figure: Enabling auto partitioning in Copy job Learn more in What is Copy job in Data Factory More flexible incremental copy with new watermark column types Incremental copy is a core pattern for keeping analytics data up to date—but in real world systems, changes aren’t always tracked with a clean datetime column. To address this, Copy job now supports additional watermark column types, making incremental copy more flexible and applicable across a broader range of source systems Copy job now supports ROWVERSION, Date, and String (interpreted as datetime) watermark columns. This allows you to choose the column that best represents change in your source system, while Copy job continues to automatically manage state, checkpoints, and incremental windows. ROWVERSION enables precise and reliable change tracking in SQL‑based systems, capturing every insert and update without relying on application‑managed timestamps. Date watermark support works seamlessly with common columns like LastUpdatedDate or ModifiedAt, with built‑in delayed extraction to prevent data loss or overlap between runs. String (interpreted as datetime) support removes the need for custom queries or schema changes when timestamps are stored as strings, improving compatibility with real‑world schemas. These enhancements make incremental copy easier to configure, more resilient in production, and better suited for diverse enterprise data models—without adding complexity for users. Learn more in What is Copy job in Data Factory - Microsoft Fabric. Data Factory—Dataflow Gen2 Preview-only steps (Generally Available) This capability improves authoring performance without changing runtime behavior. Preview-only steps let you run specific transformations during data preview only, automatically excluding them from dataflow execution and refresh. That means your production logic stays exactly the same—while the authoring experience becomes faster, smoother, and more responsive. This capability addresses a common challenge when building Dataflow Gen2 items: iterating on logic can be slow when previews must evaluate full datasets. Preview‑only steps make it possible to temporarily reduce data volume or complexity during development, enabling faster validation of transformations without introducing conditional logic or modifying the final query definition. Common uses include filtering or isolating subsets of data to accelerate previews, testing transformation logic without waiting for full evaluation, and exploring new data sources while keeping production execution intact. Because preview‑only steps are ignored during refresh and run operations, they provide a safe way to optimize the authoring workflow without risking unintended changes in published outputs. Figure: The "Enable only in previews" option within the applied steps section Preview‑only steps are also integrated into specific authoring dialogs, including file system views and the Combine files experience. In these contexts, Dataflow Gen2 can automatically introduce preview‑only logic to limit sample data used during preview evaluation, further reducing load time while preserving the behavior of the final dataflow. With general availability, preview‑only steps become a standard part of the Dataflow Gen2 authoring model—helping teams iterate faster, validate transformations more efficiently, and maintain a clear separation between development‑time experimentation and production execution. Learn more: Preview only step in Dataflow Gen2 (Preview) - Microsoft Fabric. Fabric Variable Library integration (Generally Available) Throughout the last couple of months, we’ve removed some of the limitations such as: Variable limit: the previous limit was 50 variables. You can now reference as many variables as you need in your Dataflow. Power Query editor support and using a default value: you can now see how the variable gets evaluated within the actual Power Query editor. Not only that but within the Dataflow Gen2 experience, after enabling the input widgets through the Options menu, you will also have a new way that simplifies how you can reference variables with a complete no code experience fully integrated into the experiences and dialogs that you love from Dataflow Gen2: Figure: The Filter rows dialog within Dataflow Gen2 showing the input widget and the option to Select a workspace variable After selecting this option in any of the dialogs, the experience of selecting a Variable from a library will appear. Figure: The select variable dialog invoked from within a Dataflow Gen2 Be sure to test this improved experience and share your feedback. Learn more: Use Fabric variable libraries in Dataflow Gen2 (Preview) - Microsoft Fabric New data destinations Dataflow Gen2 continues to expand where curated data can land, supporting both lake‑first architectures and hybrid data estates. With new and updated destinations, teams can publish transformed outputs in the formats and platforms that best fit their downstream consumers—whether that’s open lake storage, lakehouse files, enterprise warehouses, or business‑friendly file formats. Azure Data Lake Storage Gen2 (Generally Available) Dataflow Gen2 now supports Azure Data Lake Storage Gen2 (ADLS Gen2) allowing teams to land curated outputs directly into their data lake using open formats and folder structures aligned to organizational standards. This enables lake‑first ingestion patterns for organizations that treat ADLS as their system of record, while still authoring transformations using low‑code Dataflow Gen2 experiences. Common scenarios include reusing curated outputs across Fabric (Spark and SQL) as well as external systems that are read directly from Azure Data Lake Storage Gen2. Figure: ADLS gen2 destination option Lakehouse files (Generally Available) Dataflow Gen2 can write outputs directly into the Files area of a Fabric lakehouse. This is useful when downstream consumers expect file‑based outputs rather than tables, or when teams need to align with existing folder and file conventions inside the lakehouse. This enables patterns where transformed extracts are consumed by Spark notebooks, pipelines, or external tools, while also supporting hybrid designs where some Dataflow outputs are tables and others are files within the same Fabric workspace. Figure: Lakehouse files option Snowflake databases (Preview) This enables transformed outputs to be published directly into Snowflake databases as part of Fabric‑based, low‑code transformation workflows. This supports hybrid data estates where Fabric is used for transformation while Snowflake remains the target platform for analytics or data sharing. This preview helps standardize transformations across platforms and enables analysts to departments to publish governed outputs into Snowflake without duplicating transformation logic. Figure: Snowflake destination option Excel files (Preview) Dataflow Gen2 is introducing the ability to write outputs as Excel files (Preview) for supported filesystem destinations such as SharePoint and ADLS Gen2. This makes it easier to support business processes that still rely on Excel, while keeping transformation logic centralized and governed in Fabric. Typical scenarios include publishing refreshed Excel extracts for operational reporting or legacy workflows, and standardizing Excel output formatting from a single Dataflow definition. Schema support in Fabric data destinations (Generally Available) As Dataflow Gen2 adoption grows, many teams run into organizational challenges when publishing tables into shared destinations. Without schema control, teams often resort to creating separate databases, warehouses, or lakehouses just to keep tables logically grouped—adding complexity and making collaboration harder. With this release, Dataflow Gen2 data destinations now support writing into specific schemas (where applicable). This capability is now generally available for destinations such as Fabric SQL databases, Lakehouses, and Warehouses, giving teams more control over how Dataflow outputs are structured and governed. What’s improved Better organization without extra destinations: Teams can organize tables by domain—such as finance, sales, or HR—using schemas instead of creating separate destinations for each area. This keeps environments cleaner while still enforcing logical separation. Smoother collaboration in shared environments: Multiple teams can publish tables into the same warehouse or SQL database while maintaining clear ownership and structure through schemas. This reduces naming conflicts and supports shared analytics models without friction. Figure: The connection settings for the Warehouse connector using the advanced options to set the Navigate using full hierarchy to true By aligning Dataflow Gen2 outputs with enterprise schema conventions, this enhancement makes it easier to support multiteam data platforms, improve governance, and scale Dataflow Gen2 usage across the organization without restructuring existing destinations. Learn more: Dataflow Gen2 data destinations and managed settings - Microsoft Fabric AI-Powered Prompt Transform (Generally Available) Fabric AI Prompt is integrating generative AI features into the low-code data transformation process. Authors can enrich and transform data using natural language prompts without building or managing machine learning models, while staying within the Dataflow Gen2 execution model. The AI Prompt capability is accessed from the Add column experience, where authors define a prompt and select columns to provide contextual input. This allows AI-driven enrichment to be expressed inline alongside existing Power Query transformations, keeping logic centralized and auditable. Figure: The AI Prompt dialog in Dataflow Gen2 Moving forward, all operations associated to the usage of AI Prompt within Dataflow Gen2 will be accounted towards an explicit AI meter with the operation name of “AI Functions”. Learn more: Fabric AI Prompt in Dataflow Gen2 (Preview) - Microsoft Fabric Publish experience UX + performance improvements (parallelized query validations) Publishing complex Dataflow Gen2 items can be time‑consuming, especially when dataflows contain many queries or multiple destinations. In these cases, validations are required before a dataflow can be published, and waiting for those checks to complete often slows down iteration and troubleshooting. With this release, we’ve improved the Dataflow Gen2 publish experience through a refreshed user interface and performance enhancements that parallelize query validations. By running validations concurrently, publish operations complete faster and surface issues earlier in the process. What’s improved Less time waiting: Dataflows with multiple queries and destinations publish faster, reducing idle time during validation and helping teams move through development and testing more efficiently. Clearer guidance during publication: Validation results are available sooner, making it easier to identify and resolve issues without repeated publish attempts or back‑and‑forth edits. Together, these improvements shorten the publish cycle, reduce friction when working with larger dataflows, and help teams iterate on Dataflow Gen2 solutions with more predictable and responsive feedback. Learn more: Dataflow Gen2 with CI/CD and Git integration. Save As Improvements: Scheduled Refresh Policies and Public APIs Save As continues to improve the migration experience to Dataflow Gen2 (CICD), especially for teams moving large numbers of dataflows across workspaces or tenants. One common challenge during migration is preserving refresh behavior—copied dataflows often require manual reconfiguration before they are production ready. With this release, Save As now supports Scheduled Refresh Policies for Dataflows Gen1, ensuring that refresh configurations are retained when copying a dataflow. This reduces post migration cleanup and helps teams move faster with fewer manual steps. In addition, we’re introducing a new public Save As API for Dataflows Gen1 designed for automation and bulk operations. This enables organizations to programmatically copy dataflows at scale, making it easier to support structured migration plans and repeatable rollout processes. What’s improved Streamlined refresh configurations: Dataflows created using Save As can now inherit scheduled refresh policies from the source dataflow, helping ensure consistent refresh behavior without re‑authoring schedules after migration. Automation at scale: The new Save As public API enables automated and bulk copy scenarios, allowing teams to migrate many Gen1 dataflows to Gen2 programmatically. This is particularly useful for multi‑workspace and multi‑tenant deployments where manual migration isn’t practical. Together, these enhancements reduce migration friction, minimize manual edits, and help teams adopt Dataflow Gen2 more efficiently—whether migrating a handful of dataflows or rolling out Gen2 at enterprise scale. Figure: Dialogs for the refresh and scheduling mechanism when using the Save as experience for Dataflow Gen2 Learn more: Save As Dataflow Gen2 documentation and public Save As API reference. SharePoint site picker in Modern Get Data and Data destinations (Preview) SharePoint Site Picker replaces manual URL entry with a browsable dropdown, so you can select the right SharePoint site directly instead of finding and pasting URLs Why this matters Eliminates manual URL copy-and-paste and context switching. Reduces connection errors caused by wrong URL formats. Surfaces Recent sites and Favorite sites instantly on dropdown open and enable you to search to find sites. Where the experience is available SharePoint site picker is available for SharePoint sources of Get Data in Dataflow Gen2, Pipelines, Copy Job, and Lakehouse shortcuts, and as a destination in Dataflow Gen2. How to use SharePoint Site Picker Simplified SharePoint Site Selection: Instead of copying URLs manually, use the Site URL dropdown to choose from Recent sites and Favorite sites. Figure: SharePoint Site Picker dropdown Quick Search Capability: Find related sites faster by typing in the dropdown search box. Once you select a site, you can load data into the Power Query editor for transformation. Figure: SharePoint Site Picker searched for results Learn more about SharePoint folder connector, SharePoint list connector, and SharePoint online list. Diagnostics download (Preview) Dataflow Gen2 diagnostics download provides a simple way to collect logs and diagnostic artifacts for both cloud-based and VNET gateway dataflows. Instead of rerunning refreshes or guessing at failures, you can download the information needed to investigate issues directly. This helps teams fail faster and fix issues sooner. Downloadable diagnostics make it easier to identify refresh failures, performance bottlenecks, and connectivity problems, including complex networking scenarios that rely on VNET data gateways. Figure: The recent runs dialog showing the new button at the bottom left of the dialog to Download detailed logs With clearer signals available upfront, support investigations are shorter and operational friction is reduced. Learn more: An overview of refresh history and monitoring for dataflows. Advanced Edit for destinations (Preview) The new Advanced Edit experience for Data Destinations enables editing of the underlying M logic that configures destination settings. This unlocks deeper customization, including the ability to leverage parameters to drive destination behavior—an important step for teams standardizing deployments across environments. Parameter-driven destinations: switch target schema/table, file paths, or naming conventions without rewriting queries. Unblock advanced scenarios that require destination settings not yet available in the simplified UI. Figure: The new Advanced editor for data destinations Learn more: Advanced edit for data destination queries in Dataflow Gen2 - Microsoft Fabric. Data destination validations during publish (Preview) Dataflow Gen2 now validates data destinations during publication, helping catch common issues earlier in the development cycle. These validations surface problems such as missing permissions, invalid destination settings, or naming conflicts before the first refresh runs. By shifting these checks to publish time, authors get clear, actionable errors sooner—when changes are easiest to fix. This shortens the feedback loop and reduces time spent troubleshooting runtime refresh failures after deployment. For creators, earlier validation means fewer broken dataflows entering production. This reduces operational noise, minimizes rework, and helps ensure that published dataflows are refresh ready and more stable by default. Learn more: Dataflow Gen2 data destinations validation rules. Evaluate query API (Preview) The Execute Query API (Preview) enables on-demand execution of Power Query logic in Dataflow Gen2 scenarios—without requiring a full scheduled refresh cycle. It’s designed for cases where you need to trigger transformations programmatically (or in response to events) and retrieve results quickly for downstream processing. Event-driven pipelines: run a transformation when new data arrives and push outputs to a destination or consumer immediately. Streaming and near-real-time scenarios: execute queries more frequently than a typical scheduled refresh to support operational dashboards and alerting workflows. Automation at scale: integrate with orchestration tools and scripts to run specific queries as part of broader ETL/ELT jobs. Faster debugging: re-run targeted queries to validate fixes without republishing the entire dataflow. Learn more: Execute Query API (Streaming) documentation (Preview). Data Factory Data Factory MCP (Preview) Dataflow Gen2 offers a suite of pipeline functions, including dataflow creation, M (Power Query) scripting, connection management, query execution, and refresh coordination. These tools are directly accessible to AI assistants. Access is available through platforms such as VS Code, Claude, ChatGPT, Gemini, or via the command line. Why it matters AI assistants create, test, and deploy dataflows through natural language—no browser tabs or manual configuration required. Iterative M development via execute_query lets the AI test transforms against live data before committing to a full refresh. MCP Apps provide guided UI forms (connection setup, gateway selection) inside the chat panel. Open source (GitHub), ships as a NuGet package, runs locally—credentials never leave your machine. Learn more: Data Factory GitHub repo. IBM Netezza ODBC Driver (Generally Available) As we move away from using the embedded Simba driver, customers now have a more dependable and supported option by using their own Netezza driver. This update ensures continued connectivity, long-term support, and a more future-ready experience for organizations using the Netezza connector. Customers do not need to install the new connector; you may reuse your existing connector but will need to install the new IBM Netezza ODBC driver. Figure: IBM Netezza Connector Selection in Fabric UI Reference the IBM Netezza ODBC documentation for more information Google BigQuery connector (Generally Available) This update reflects a shift to the newer GBQ connector as the supported, long-term path forward, providing customers with improved reliability, and alignment with our evolving security standards. With this update, customers can use a connector designed for durability, compliance, and future enhancements. Figure: Google BigQuery Connector in Fabric UI Additional details are available in the Google BigQuery connector documentation. QuickBooks Online connector retirement The QuickBooks Online connector is being retired and will no longer be supported as of March 2026. As part of our ongoing platform evolution, this change streamlines our connector portfolio and ensures our continued commitment to only the highest level of secure data connectivity. After retirement, customers will no longer be able to create new connections, and existing connections may no longer function. Lakehouse Maintenance activity in Fabric Pipelines (Preview) Keeping your Lakehouse healthy shouldn’t require a long checklist or manual scripts. The new Lakehouse Maintenance activity (Preview) makes it easy to automate common upkeep tasks directly inside Fabric Data Factory pipelines. Figure: The Lakehouse maintenance activity in Fabric pipelines With this activity, you can schedule and run actions like vacuuming old files, optimizing table layouts, and managing storage—all in a repeatable, governed workflow. It’s a simple way to keep performance high and storage costs in check, especially for teams managing large or fast‑growing datasets. Figure: The Lakehouse maintenance activity settings Whether you run maintenance nightly or as part of a broader DataOps process, this activity helps support reliable Lakehouse operations. Check out our Lakehouse Maintenance documentation. Refresh SQL endpoint activity in Fabric pipelines (Preview) The process of keeping your SQL analytics layer current is now simpler. The new Refresh SQL endpoint activity (Preview) lets you refresh your Lakehouse SQL endpoint on-demand or as part of your pipeline orchestration. Figure: The Refresh SQL endpoint activity You can trigger targeted refreshes after data ingestion, run coordinated refreshes alongside your transformations, or ensure downstream consumers always see the latest state. It’s built for operational consistency – especially for BI, reporting, and real‑time analytics scenarios that rely on predictable SQL performance. Figure: The Refresh SQL Endpoint activity settings. This activity gives you more control, less manual overhead, and a smoother end‑to‑end refresh experience. Check out the RSQL documentation for more details. Generate Pipeline expressions with Copilot (Generally Available) Writing expressions doesn't have to be time-consuming; simply describe your needs in natural language, and Copilot will generate pipeline expressions for you. Figure: Generate Pipeline expressions with Copilot Whether it’s building dynamic folder paths, conditional logic, string parsing, or parameterized values, Copilot now handles the expression authoring for you. This feature removes friction for both new users and power users – saving time, reducing errors, and making expression logic easier to understand. Workspace monitoring for Fabric Data Factory’s pipelines and Copy job (Preview) Operational observability continues to evolve in Fabric. We’re taking the first major step toward workspace‑level observability in Microsoft Fabric Data Factory. Until now, understanding how pipelines and copy jobs behave at scale often meant inspecting individual runs via Monitoring Hub. With the introduction of workspace monitoring (Preview), Data Factory begins a shift to a workspace‑wide view of operational health. The newest workspace monitoring updates bring clearer visibility and faster troubleshooting across your pipeline ecosystem. What’s available A workspace-wide view of item-level runs Rich filtering, sorting, and drilldown Insight into failure patterns, duration trends, and operational health Faster navigation—no need to click into each pipeline This gives DataOps teams a unified lens to understand performance and diagnose issues quickly. Figure: A view of your pipelines and Copy jobs within the workspace monitoring solution Coming Soon Activity-level L2 monitoring for pipelines. Copy job L2-level monitoring (Preview) for deeper insights and debugging. These improvements continue building toward a more comprehensive, intuitive monitoring experience for production workloads. Check out our docs on Enable Workspace Monitoring in Microsoft Fabric and Workspace Monitoring for Copy Job in Microsoft Fabric for more information on how to use this experience. Interval-based schedules The latest enhancement to Fabric Data Factory pipelines is the availability of interval-based schedules! This powerful new feature allows you to automate data workflows at regular non-overlapping intervals, like the popular tumbling window trigger in Azure Data Factory. Figure: Interval-based schedule configuration in Fabric Data Factory With interval-based scheduling, you can easily configure recurring pipeline runs that ensure timely data processing and seamless integration across your architecture. New Airflow APIs New Airflow Operators Apache Airflow jobs in Fabric Data Factory facilitate the execution of a wide range of Fabric artifacts through native operator integration. Users can run artifacts such as Notebooks, Spark job definitions, Pipelines, Semantic Models, and user data functions directly from their DAGs. Apache Airflow jobs now provide support for executing Copy jobs and dbt jobs! Figure: Airflow operators for Fabric items, including Copy job and dbt job execution To learn more, refer to Run a Fabric item using Apache Airflow DAG. PowerShell model for gateways (Generally Available) The PowerShell model for gateways now delivers fully supported, production-ready automation for gateway lifecycle, update, restore, and configuration management. This release introduces new commands for version discovery and upgrade control, along with reliability and usability improvements that make large-scale, script-driven gateway operations easier and more robust. Figure: Using the Gateway PowerShell module to manage gateway operations from the command line Learn more through the gateway PowerShell documentation and cmdlet reference on Microsoft Learn. Certificate and proxy support for VNet data gateway (Generally Available) Certificate and proxy support for VNet data gateway enables secure, compliant connectivity in enterprise environments. Organizations can use enterprise-issued certificates for gateway authentication and configure proxy routing when direct internet access is restricted. Together, these capabilities strengthen security, support corporate network policies, and expand deployment flexibility in controlled and regulated infrastructures. nshot_of_certificate_and_proxy_settings_for_a_Virtual_Network_Data_Gateway Figure: Configure certificate authentication and proxy for a Virtual Network Data Gateway Learn more through Manage virtual network (VNet) data gateways. Virtual network data gateway supports up to nine instances This update enables greater scalability and higher throughput for enterprise workloads. With expanded instance capacity, organizations can handle increased data movement and processing demands, improve parallel job performance, and enhance reliability for mission-critical tasks. This update provides more flexibility to scale gateway infrastructure in line with growing business needs. Figure: Virtual Network Data Gateway now supports scaling up to nine instances per cluster Learn more What is a virtual network (VNet) data gateway. SSIS Pipeline Activity (Preview) SQL Server Integration Services (SSIS) has been a cornerstone of enterprise data integration for decades, powering mission-critical ETL workloads across thousands of organizations worldwide. Invoke SSIS Package activity in Data Factory in Microsoft Fabric (Preview), provides the power of your existing SSIS investments directly into Fabric's unified SaaS analytics platform. Figure: Add an Invoke SSIS Package activity Many enterprises have significant investments in SSIS packages that orchestrate complex ETL workflows across on-premises databases, file systems, and cloud services. Until now, running these packages required either an on-premises SQL Server, or the Azure-SSIS Integration Runtime in Azure Data Factory. Both options meant managing additional infrastructure and staying outside the Fabric ecosystem. Figure: Invoke SSIS package activity configuration But the Invoke SSIS Package pipeline activity in Microsoft Fabric Data Factory changes this. It allows you to execute your existing SSIS packages directly from a Fabric pipeline, enabling true lift-and-shift of legacy ETL workloads into Fabric—no package rewrite required. There is no need for integration of runtime management or stopping and starting IRs; simply incorporate them into your pipeline. Seamlessly upgrade Azure Data Factory and Synapse pipelines to Microsoft Fabric (Preview) Microsoft Fabric Data Factory now offers a guided (Preview) migration experience to help you move existing Azure Data Factory (ADF) and Azure Synapse Analytics pipelines into Fabric—starting with an assessment-first approach so you can migrate intentionally and validate before switching production workloads. Review readiness and plan next steps: The assessment categorizes pipelines and activities so you can decide what to migrate now vs. what to fix or defer. You can also export results to CSV for offline review and remediation planning. Figure: Review pipeline and activity readiness results in Azure Data Factory (ADF only) Mount your factory to Fabric For Azure Data Factory migrations, you’ll mount your ADF into a Fabric workspace and then continue the remaining steps inside Fabric. Figure: Continue the migration flow in Fabric after mounting Migrate selected pipelines in Fabric In Fabric Data Factory, open the mounted factory (ADF) or chosen workspace (Synapse), then migrate the pipelines you want to migrate. Map linked services to Fabric connections and complete migration During migration, you’ll map ADF/Synapse linked services to Fabric connections. For guidance on creating and managing connections in Fabric, refer to Data source management. Figure: Map Linked Services to Fabric Connections Validate and promote After migration, validate connections and credentials, run end-to-end tests, and then re-enable triggers as needed. Pipelines migrate safely, with triggers disabled by default so you stay in control of execution. Learn more: Upgrade your Azure Data Factory pipelines to Fabric. Data Factory—Mirroring Mirroring for SAP (Generally Available) Built on top of SAP Datasphere’s Premium Outbound Integration, mirroring for SAP seamlessly integrates Fabric’s advanced mirroring engine with SAP Datasphere’s replication flows, unlocking connectivity through SAP’s native data extraction technologies. This means direct access to the full suite of SAP applications—whether it’s SAP S/4HANA (on-premises or cloud), SAP ECC, SAP BW, SAP BW/4HANA, or cloud solutions like SAP SuccessFactors, SAP Ariba, and SAP Concur. Mirroring capabilities allow you to: Eliminate data silos by bringing SAP data alongside other enterprise sources in OneLake. Maintain end-to-end data lineage and governance for compliance and auditability. Accelerate time-to-insight with near real-time data replication no custom ETL required. Figure: Mirrored database for SAP Learn more in Microsoft Fabric Mirrored Databases From SAP. Mirroring for Oracle databases (Generally Available) Mirroring for Oracle is now available in Microsoft Fabric, bringing a production‑ready, enterprise‑grade way to continuously replicate Oracle data into OneLake with no custom ETL pipelines. This milestone reflects strong validation from customers already running Mirroring for Oracle in production and marks a major step forward in Fabric’s zero‑ETL data integration strategy. With near real‑time data replication, customers can keep analytics, BI, and AI workloads continuously in sync with their operational Oracle systems. This release delivers improved stability, scale, and operational readiness, informed directly by customer feedback from public preview deployments. Mirroring for Oracle integrates natively with Fabric experiences like Power BI, Notebooks, and Lakehouses, enabling faster insights without disrupting existing Oracle workloads. As a fully supported capability, Mirroring for Oracle is now ready for broad enterprise adoption with long‑term investment from the Fabric platform team. Figure: Mirroring for Oracle creation steps Learn more at Mirroring for Oracle in Microsoft Fabric. Mirroring for Azure Database for MySQL (Preview) Mirrored databases now support Azure Database for MySQL. This capability enables you to directly replicate data from Azure Database for MySQL Flexible Server into Fabric in near real time, ensuring that information remains current, readily query-able, and seamlessly integrated throughout the analytics stack without the need for traditional ETL processes. Mirrored MySQL data is managed alongside other data sources, facilitating cross-source querying, unified reporting, and comprehensive analytics. Figure: Screenshot of configuring a mirrored database for Azure Database for MySQL Learn more in Microsoft Fabric Mirrored Databases for MySQL. Mirroring for SharePoint List (Preview) Mirroring for SharePoint Lists enables continuous replication of SharePoint Lists and Document Libraries into OneLake without building custom ETL pipelines. This capability keeps SharePoint data automatically synchronized in near real time, ensuring analytics in Fabric stay aligned while SharePoint remains the system of record. When mirrored, both list tables and document library metadata land in OneLake in an analytics‑ready format, with document libraries replicated via shortcuts and converted into Delta Lake tables. Figure: Mirroring setup for a SharePoint list Fabric automatically creates a mirrored database and a read‑only SQL analytics endpoint, providing a rich analytical surface over the replicated data. As changes are made in SharePoint—such as new columns or updated rows—those updates flow continuously into Fabric, keeping schemas and data in sync. This public preview unlocks a simple, unified way to analyze SharePoint operational data across Fabric workloads including SQL, Power BI, notebooks, and data engineering experiences. Extended Capabilities in Mirroring: Change Delta Feed and Snowflake Mirroring Support for Views (Preview) Optional enhancements that build on core mirroring to support more advanced, real‑world analytics scenarios. These capabilities are designed for customers who need more than basic replication—enabling faster freshness, incremental processing, and business‑ready data without building or maintaining complex ETL pipelines. Including Change Data Feed (CDF), which captures inserts, updates, and deletes at a granular level and applies them incrementally into OneLake, allowing mirrored data to stay continuously fresh without full reloads. Extended Capabilities also include Mirroring Views for Snowflake (with support for other sources coming soon), which replicate logical views from the source system into OneLake so that source‑defined business logic—such as joins, filters, and transformations—can be preserved directly in Fabric. Together, CDF and Views enable incremental pipelines, near real‑time analytics, and shaped datasets that are immediately ready for consumption across Fabric workloads. Extended Capabilities are enabled during mirror setup and operate on top of core mirroring, allowing customers to selectively opt into advanced functionality as their analytics and AI needs grow. Billing will be available as part of these extended capabilities starting April 1, 2026. More details about these capabilities and billing can be found on our documentation: Extended Capabilities in Mirroring – Overview. Mirrored database now supports up to 1000 tables To meet growing business demands and improve scalability, mirrored databases now support up to 1000 tables, raised from the previous limit of 500. This enhancement significantly expands the scale of datasets that can be mirrored from the source database, enabling customers to bring more comprehensive data into Fabric without fragmentation, drive deeper analysis and scale the data solution to meet evolving requirements. Learn more in Mirroring in Microsoft Fabric. That’s a wrap! Publishing this update on the first day of FabCon feels especially meaningful. The features in this release reflect not just ongoing platform investment, but the ideas, feedback, and candid conversations we continue to have with the Fabric community—in sessions, online, and across preview programs. Thank you for showing up, sharing your experiences, and helping shape where Fabric goes next. We encourage you to explore these updates, ask questions, and tell us what’s working—whether that happens here at FabCon, in community forums, or through ongoing feedback channels. We’re grateful to be building Fabric alongside such an engaged community, and we’re excited to keep learning from you throughout FabCon and beyond.
Securing Outbound AI Connectivity in SQL Server 2025 (Generally Available)
AI is now part of the data layer SQL Server is evolving beyond a system that only stores and queries data. In SQL Server 2025, the engine can participate directly in AI-enabled workflows through governed integrations. Database-side workloads can connect to external AI services for scenarios such as retrieval-augmented generation (RAG), where model outputs are grounded in authorized, enterprise data context. This shift helps organizations reduce custom middleware and ad- hoc integration paths while keeping governance, security, and access control closer to the data platform. For example, a financial institution analyzing suspicious transaction patterns can execute approved outbound AI calls under database governance instead of exporting sensitive data through scattered application services. However, that shift creates a valid security question: when SQL Server can reach external AI endpoints, how do you maintain control—especially for outbound connectivity to external AI services? SQL Server 2025 addresses this by extending its existing defense-in-depth model. AI connectivity is not treated as an exception path; it is enforced within SQL Server’s existing trust and control model. Why this matters Many teams adopted AI by pushing sensitive data out of the database into application code or integration services. That approach often leads to three issues: Data moves more than necessary. Security policy becomes inconsistent and fragmented across systems. Audit trails are more challenging to correlate. SQL Server 2025 supports a more controlled operating model. AI integration can happen closer to the data, under database governance, while still enabling modern AI scenarios. This approach reduces unnecessary data movement while preserving existing application patterns. This benefits security, platform, and engineering teams by simplifying control and reducing fragmentation. Strong controls over external connectivity SQL Server 2025 supports outbound AI connectivity through features such as direct REST invocation and external model objects. Both rely on explicit SQL Server permissions and approved authentication, but the controls apply differently depending on the feature. External REST invocation uses sys.sp_invoke_external_rest_endpoint to call approved HTTPS endpoints from T-SQL. External model registration uses CREATE EXTERNAL MODEL to define model objects for embedding and vector search scenarios. The shared security idea is simple: define what can be used, grant only the required permissions, and authenticate access with approved credentials or identities. Enablement applies to REST invocation, which must be turned on before use. Permissions control who can enable REST calls, create or alter model objects, and execute approved operations. Authentication determines which credential, identity, or secret is used when SQL Server reaches an external service. For sys.sp_invoke_external_rest_endpoint, SQL Server provides a direct way to invoke HTTPS REST endpoints. The feature is disabled by default in SQL Server 2025 and must be enabled through sp_configure. Changing that setting requires ALTER SETTINGS server permission (implicitly held by sysadmin and serveradmin), which keeps enablement under administrative control. After REST invocation is enabled, runtime use still depends on permissions, endpoint policy, and credential handling, including EXECUTE ANY EXTERNAL ENDPOINT for callers. Organizations should grant invocation rights only to required principals, restrict allowed destinations, and require authenticated requests. Depending on the target service, authentication can use managed identity, certificates, API keys, or other supported mechanisms. For CREATE EXTERNAL MODEL, SQL Server creates a database object that stores the model endpoint, authentication method, and model purpose. Creating or changing that object requires CREATE EXTERNAL MODEL or ALTER ANY EXTERNAL MODEL, while using it requires EXECUTE on the model. This makes model access explicit without implying it uses the same enablement switch as REST invocation. The result is a clearer security boundary: REST calls require explicit enablement and controlled invocation, while external models require governed object creation and execution. In both cases, least privilege, approved authentication, and auditability remain central. High-privilege access is still the real security boundary AI features do not change the core rule of SQL Server security: the real boundary is who has permission to do what. When privileges are tightly scoped, features remain safe to adopt. That’s why least privilege matters. Users and applications should have only the permissions they need for their specific tasks. Nothing more. In practice, this means: Avoid broad administrative role membership wherever possible. Grant only the minimum permission required for the specific operation. Separate operational roles (configuration, model management, invocation). Review role assignments and grants on a defined cadence. Remove stale access quickly after ownership changes. Strong security is not about blocking capabilities. It is about controlling who can use them, when they can use them, and under what conditions, while integrating with existing governance processes. Visibility and accountability are built in Prevention is only one part of security. Detection and response matter just as much. Because these operations execute through SQL Server interface, they can be observed with established SQL monitoring and auditing patterns, including SQL Server Audit and Extended Events. That helps teams answer operational questions such as: Who enabled external connectivity? Which principal invoked an external endpoint? Which external model object was executed? When did behavior deviate from baseline? Governed SQL-side AI execution can improve both incident response and audit readiness. By centralizing execution and logging, organizations can more easily collect, trace, and explain evidence when investigating incidents or responding to compliance reviews. Architecture still determines risk Traditional AI patterns often move data out of SQL Server into other runtime layers before calling external models. This increases the attack surface and fragments policy enforcement. SQL Server 2025 enables a different operating model: Keep data closer to where governance already exists. Use explicit model and endpoint objects with permission checks. Apply outbound network controls and endpoint allow patterns. Centralize audit trails for security and compliance review. In addition, network connectivity and SQL permissions alone should not be the final security boundary. External AI endpoints should also enforce independent authorization decisions, such as RBAC, allowing access only to approved managed identities. This creates an additional layer of protection and helps limit the impact of credential misuse or configuration errors. Secure usage best practices Use this checklist before production rollout: Enable only the connectivity features required for the scenario. Grant least privilege for configuration, model management, and invocation. Restrict outbound traffic to approved destinations. Use managed identities where supported; otherwise use centrally managed, least-privilege credentials. Enforce endpoint-side authorization and RBAC controls for service identities and credentials. Audit configuration changes and invocation activity. Establish behavioral baselines and alert on deviations. Run periodic access and configuration reviews. Include incident response runbooks for external AI call paths. These practices align with standard SQL security guidance and are not unique to AI features. Most control failures come from weak operating discipline around the feature. Strong defaults help, but repeatable processes are what keep systems safe over time. Next Steps Review permission design for AI connectivity operations in a non-production environment. Define approved endpoint policy and outbound control requirements. Pilot one AI scenario with audit-first instrumentation. Document operational ownership for enablement, credential management, and monitoring. Get Started Refer to sp_invoke_external_rest_endpoint, apply the recommended risk-mitigation controls for unauthorized access and data transfer, then validate your CREATE EXTERNAL MODEL permissions in a dev environment using the best practices checklist above, and finalize your SQL Server Database Engine permission design before production rollout.
Change Event Streaming for SQL database in Microsoft Fabric (Preview)
SQL database in Microsoft Fabric already brings operational and analytical workloads together—now it can stream changes as they happen. Change Event Streaming sends inserts, updates, and deletes directly to Fabric Eventstream or Azure Event Hubs. Build near-real-time pipelines from operational data without leaving Fabric.A new analytics frontier: GPU-accelerated Fabric Data Warehouse (Early Access Preview)
As data volumes grow, concurrency rises, and analytics workloads become more dynamic and AI-driven, performance becomes harder to predict and harder to scale. Every query sits in the critical path, adding pressure to the warehouse, and every second counts. This is the core tension in analytics today. The expectations have changed, but the underlying technology has not, leaving agents, applications, and AI systems waiting on data. To meet this moment, analytics needs a new kind of execution engine.Microsoft recognized as a Leader in The Forrester Wave™: Data Lakehouses
For years, organizations have invested in data platforms to understand what happened across their business. Dashboards, reports, and KPIs are now table stakes. But as AI becomes central to how organizations operate, the bar for the data lakehouse is getting much higher. The next generation of applications and agents needs governed access to every kind of data: structured, semi-structured, and unstructured; batch, streaming, and real-time—working together on one open foundation. When that data remains spread across fragmented systems, teams are left reconciling copies, duplicating governance, and stitching together context before they can create value. Today, we're proud to share that Microsoft has been recognized as a Leader in The Forrester Wave™: Data Lakehouses, Q3 2026. In the report, Forrester describes Microsoft Fabric as “a strong fit for enterprises seeking a unified, AI-enabled lakehouse platform integrated with the Microsoft ecosystem.” We believe this recognition reflects the bold vision behind Microsoft Fabric and Microsoft OneLake: helping organizations eliminate the integration tax of fragmented data estates by bringing data, analytics, governance, and AI together on one open lakehouse foundation. Fabric: A unified foundation for the AI-era lakehouse The Forrester report frames the modern lakehouse as more than a system of record for analytics. As agentic AI systems begin to reason, plan, and act on enterprise data, the lakehouse is becoming the operational foundation where intelligence is grounded and activated in real time. Microsoft Fabric was built for this shift. With OneLake, Fabric gives organizations a single, governed data lake and one SaaS platform where data teams, analysts, developers, and business users can work from the same trusted foundation. Forrester notes that Microsoft’s approach emphasizes deep integration across Power BI, Copilot, Microsoft 365, OneLake, databases, and AI services—helping unify analytics, operational, and AI workloads within a single ecosystem. Forrester also highlights Microsoft’s “bold vision of a unified, AI-powered, open data platform that brings together analytics, data engineering, business intelligence, and operational data.” Furthermore, they add that “innovations such as OneLake shortcuts, mirroring, AI-powered transformations, and cross-cloud interoperability support this vision by reducing silos and simplifying access to distributed data.” One open foundation for data and analytics OneLake is the governed data lake at the heart of Fabric, designed to unify your entire multi‑cloud data estate. It connects data across clouds and on‑premises systems using zero‑copy, zero‑ETL access, so teams work from a single, governed copy of data. With native support for open formats like Delta Lake and Iceberg, this data remains accessible from any analytics engine or platform, including Microsoft Fabric, Snowflake, and Azure Databricks. Once data is connected or stored in OneLake, the OneLake catalog helps secure, govern, and organize it into a logical data mesh, making trusted data easy for everyone to discover and use. Govern once, across every engine Security, identity, lineage, and governance are built into Fabric rather than bolted on tool by tool. State of the art OneLake security can define object-, row-, and column-level controls once and enforce them consistently across Spark, SQL, KQL, Power BI, Copilot, and third-party engines through OneLake security APIs. The OneLake catalog centralizes sensitivity labels, classification, and end-to-end lineage, helping organizations simplify governance while giving users trusted access to the data they need. Every workload on one lakehouse Fabric brings relational, real-time, analytical, document, and vector workloads into one platform experience on OneLake. Spark powers data engineering with the Native Spark Execution Engine in Microsoft Fabric, which accelerates workloads by running much of the execution in highly optimized native C++ code with vectorized processing, while preserving the same Spark APIs, notebooks, and DataFrame code users already know. With the native execution engine, Spark in Fabric delivers up to 6x faster performance than open-source Apache Spark, helping improve price performance by completing the same workloads with less compute and lower costs. Additionally, the warehouse engine and lakehouse SQL endpoint serve interactive queries over the same Delta tables; Real-Time Intelligence supports streaming and event-driven scenarios; and Power BI queries OneLake directly through Direct Lake without importing or moving data. AI native to the data platform Fabric’s integration with Copilot and agents enable natural language analytics and intelligent automation at scale. Vector embeddings can sit alongside structured data for AI retrieval, so teams can build analytics, AI, and applications on one governed copy of data. Instead of moving data to each workload, organizations can bring more workloads and more AI-powered experiences to the same open lakehouse. Why organizations choose Fabric Customers are seeing real impact from using Fabric: less duplication, cleaner governance, faster development, and a simpler path from data to AI. At the foundation is a common pattern: organizations are consolidating fragmented data estates onto a single governed lakehouse. London Stock Exchange Group (LSEG), a partner to the world's leading financial institutions, set out to simplify a complex, fragmented data landscape and give its customers a consistent, unified experience. Using Fabric, LSEG is building a unified data platform that consolidated 30 systems, 1,200 datasets, and 33 petabytes of data, accelerating product development, improving data quality, and advancing AI readiness. Product development timelines have moved from years to months, delivering faster, cleaner data to everyone from global firms to individual traders. "When you need to pull data together across disparate sources that are in different formats and varying levels of modernisation and maturity, it makes it difficult to react to market demand quickly. We knew it would be far more efficient to bring everything into a single, modern platform. It would mean we could run the organisation leaner and react to market demand faster.” Dave Byrne, Group Head of Data Platforms at LSEG Once data is unified, organizations can apply governance and analytics at enterprise scale. UNC Health standardized its enterprise data estate on Fabric, creating a single, governed lakehouse foundation for clinical analytics, operations, population health, and research. Fabric powers UNC Health’s AI solutions, which reduced care-gap chart review time by nearly 50% and provides the governed data foundation for a secure research environment supporting 25 active studies. That same foundation also creates new opportunities for AI-driven innovation. Eastman, a global specialty materials company, adopted Fabric to modernize its legacy data architecture and create a unified, governed lakehouse foundation for analytics and AI. Using OneLake shortcuts and data mirroring, Eastman shares data across domains without unnecessary duplication, ingested roughly one billion rows from eight systems, and established a scalable platform for analytics, machine learning, and AI-powered applications. “We operate in a lot of markets that are fundamentally different from one another. Being able to aggregate all this loose, unstructured data into something that’s actionable is really helping our commercial organization build better strategies for the year ahead.” —Andrew Ervin, Manager of Generative AI, Eastman Taken together, these examples illustrate the evolution of the modern lakehouse: first unifying data, then governing it consistently, and ultimately turning it into a foundation for AI-powered innovation. This is the shift that many organizations are making as they prepare for the next generation of applications and agents. Strategic takeaways for enterprise leaders Three shifts stand out for leaders preparing their organizations for the next generation of AI: The lakehouse is now the default foundation for AI. Agents and AI applications need governed access across every data type; not siloed systems stitched together after the fact. Openness prevents lock-in. Open table formats and bi-directional interoperability help organizations unify their estate without walking away from the tools and platforms they already run. Reducing duplication changes the economics. Bringing workloads to a shared, governed foundation helps organizations simplify operations, strengthen consistency, and accelerate innovation. As organizations prepare for the next generation of AI-powered applications and agents, the need for a unified, open, and governed data foundation will only grow. Microsoft Fabric was built to bring data, analytics, governance, and AI together on that foundation, and we’re excited to keep innovating alongside our customers. Forrester’s recognition reinforces what we’ve believed from the start: the organizations that succeed in the AI era will be those that eliminate fragmentation, simplify governance, and build on one open lakehouse. Learn more Read the complimentary report. Explore Microsoft OneLake and Microsoft’s vision of an open data lake ecosystem. Join us at the next Fabric + SQL Community Conference in Barcelona to hear directly from our product and teams and community members. Statement from Forrester Forrester does not endorse any company, product, brand, or service included in its research publications and does not advise any person to select the products or services of any company or brand based on the ratings included in such publications. Information is based on the best available resources. Opinions reflect judgment at the time and are subject to change. This report is part of a broader collection of Forrester resources, including interactive models, frameworks, tools, data, and access to analyst guidance. For more information, read about Forrester's objectivity here.Fabric November 2024 Feature Summary
Welcome to the November 2024 update for Microsoft Fabric! This month, we're excited to bring you a host of new features and improvements designed to enhance your experience and productivity. From the introduction of Copilot in Power BI mobile apps to the new Fabric Databases, there's something for everyone. Whether you're looking to streamline your data analysis, improve your reporting capabilities, or simply stay up to date with the latest innovations, this update has you covered. To learn more, read about all these announcements, and more in Arun’s blog post Accelerate app innovation with an AI-powered data platform | Microsoft Fabric Blog Be one of the first to use SQL database on Fabric In this series on SQL database on Fabric, you will learn how Fabric brings together both transactional and analytical workloads, creating a truly unified data platform. You’ll also learn how developers can build reliable, highly scalable applications where cloud authentication and encryption are secured by default. Starting December 3rd, join us for six sessions with database experts and see just how easy it is to get started. Sessions are available live and on-demand. View the sessions and register for the series. Don’t miss Microsoft Ignite 2024 and FabCon 2025 Attend Microsoft Ignite 2024 online for free November 19 – 21, 2024 to learn about the latest innovations in Data & AI. Join sessions that will cover solutions that help modernize and manage intelligent applications, safeguard data, and accelerate productivity. Join us at FabCon Las Vegas from March 31 to April 2, 2025, for the ultimate Microsoft Fabric, Power BI, SQL, and AI community-led event. With more than 144 sessions, 18 pre- and post-conference workshops, unique community experiences, a dedicated pre-day for partners, all-day Ask-The-Experts hours, 20+ expo booths, plus after-hours events and socials you don't want to miss. Contents Be one of the first to use SQL database on Fabric Don’t miss Microsoft Ignite 2024 and FabCon 2025 Certifications Get certified in Microsoft Fabric - for free! Explore the newest Fabric certification for Data Engineers Copilot and AI Copilot in Power BI mobile apps (Preview) Copilot summaries in subscriptions (Preview) Reporting Path Layer for the Azure Map visual Visual calculations (Preview) ‘Set Alert’ with Activator and Real-Time Intelligence (Generally Available) Small multiples for the new card visual (Preview) New visual - text slicer (Preview) Modeling Define new measure in DAX query view quick queries Metric sets: a new era of metric management in Fabric (Preview) Performance improvements for models with calculation groups and format strings in Excel DLP policies restrict access action for semantic models (Preview) Semantic modeling in Visual Studio Code with the new TMDL extension (Preview) Developers + APIs Fabric Git: TMDL format for semantic model export Semantic model client library updates Visualizations KPI by Powerviz Zebra BI Tables 7.3 Waterfall PRO by ZoomCharts: the most interactive waterfall visual for financial data Lollipop bar chart by Nova Silva Sales velocity chart Donut Chart by JTA New book: Data Visualization with Microsoft Power BI Other Support for Power BI language settings when a paginated report is viewed on the Power BI service Platform Introducing OneLake catalog Tenant switcher control Automate GitHub integration with Microsoft Fabric REST APIs Switch branches from the source control pane Announcing general availability of the Fabric Workload Development Kit OneLake External data sharing is now generally available Mirroring Introducing Open Mirroring Fabric Database Mirroring Public REST APIs are now generally available Mirroring for Azure SQL Database now Generally Available Introducing Mirroring for Azure SQL Managed Instance (Preview Databases Introducing Fabric SQL database (Preview) Data Warehouse Cold query performance improvement Service principal support for Fabric Data Warehouse Data Engineering Notebook display chart upgrade Fabric API for GraphQL is now generally available with exciting new features Esri’s ArcGIS GeoAnalytics integration with Fabric Spark (Preview) Jar libraries are now supported in Fabric Environments Support of spaces and special characters in Delta table names Data Science Introducing low code AutoML Real-Time Intelligence Ingest & Process Announcing the general availability of Real-Time Hub Announcing the general availability of Enhanced Eventstream Announcing the general availability of connector sources in Eventstream Introducing Azure Service Bus Connector for Eventstream (Preview) New Fabric events (Preview) Eventstream Data Preview on database CDC sources Monitoring experience on connector sources with Runtime Logs and Data Insights in Eventstream Processing and routing events to Activator with Eventstream (Preview) Introducing Eventstream’s CI/CD support Automate Eventstream Item Operations with Eventstream REST APIs Stream Data to Eventstream Securely using Entra ID Authentication Analyze & Transform Eventhouse monitoring (Preview) Eventhouse Query Acceleration for Shortcuts (Preview) Synapse Data Explorer to Eventhouse migration (Preview) New explorer for database objects in KQL Queryset Entity Diagram view in KQL Database Visualize & Act Easily share Real-Time Dashboards with others Announcing the general availability of Real-Time Dashboards Announcing the general availability of Activator Data Factory Table and partition refreshes added to semantic model refresh Import and export your Fabric Data Factory pipelines New connectors available Simplify data ingestion with Copy Job - CI/CD upsert & overwrite New capabilities in Copilot for Data Factory to efficiently build and maintain your Data pipelines OneLake datahub is now the OneLake catalog in Modern Get Data experience Dataflows now support CI/CD (Preview) https://youtu.be/eyjVj-k8m1M?si=Jt1-TkIY58r8LyKr Certifications Get certified in Microsoft Fabric - for free! Get ready to fast-track your career by earning your Microsoft Certified: Fabric Analytics Engineer Associate certification. For a limited time, we will be offering 5,000 free DP-600 exam vouchers to eligible Fabric Community members. Complete your exam by the end of the year and join the ranks of certified experts. Don’t miss this opportunity to get certified. Explore the newest Fabric certification for Data Engineers We are excited to announce a brand-new certification for data engineers. The new Microsoft Certified: Fabric Data Engineer Associate certification will help you demonstrate your skills with data ingestion, transformation, administration, monitoring, and performance optimization in Fabric. To earn this Certification, pass Exam DP-700: Implementing Data Engineering Solutions Using Microsoft Fabric, currently in beta. Copilot and AI Copilot in Power BI mobile apps (Preview) We’re excited to announce the release of Copilot in Power BI mobile apps (Preview)! This new feature brings the power of AI directly to your fingertips, enhancing your mobile experience when you’re on the go, offering a quick and simple way to dive into your data. With Copilot in Power BI mobile apps, you no longer need to analyze data yourself. Copilot provides report summaries and insights that allow you to make informed decisions anytime and anywhere. Imagine a sales manager effortlessly pulling up an executive summary of the latest sales report with a single tap, or a maintenance technician getting real-time machine-performance insights while on the factory floor. To start using Copilot on your mobile app, simply tap the Copilot button located in the report header (that meets Copilot requirements in Power BI) and choose whether you want to get a summary or to look into insights. Copilot will deliver a response based on your request. You can then copy and share the response or keep interacting with Copilot by choosing from the suggestions at the bottom. These suggestions can help you tweak the response or create new requests. For more details about Copilot in Power BI mobile apps, check out our full blog post. Copilot summaries in subscriptions (Preview) Do you need to extract insights from Power BI report images in your email or quickly digest a summary of your Power BI report? Subscribe to Copilot summaries for Power BI reports. This feature is available with Standard subscriptions and for reports in a copilot-eligible capacity. Learn more about using Copilot in Power BI and Fabric. Set up the copilot summaries for Power BI reports that you subscribe to as follows: Select the ‘Subscribe’ option from the ribbon for the Power BI report that you are interested in. 2. Select ‘Standard Subscription’. 3. You can subscribe to the report. Learn more about creating report subscriptions. 4. Add a copilot summary to the email delivered by the subscription. If you are eligible, your subscription will receive the Copilot summary by default. Learn more about setting up Copilot summaries for subscriptions. 5. You can ‘Preview summary’ to view a sample of what the summary might look like. 6. Test your subscription by selecting ‘Send Now’ after you Save the subscription. Note: ‘Send Now’ will deliver the email with the copilot summary to all recipients. Email Sample: Learn more about Copilot summaries in subscriptions from our documentation. This feature will roll out gradually over the next few weeks and is not available in Gov clouds. Copilot and AI demo https://youtu.be/eyjVj-k8m1M?si=MYJkFPH5FFdNtEBT&t=160 Reporting Path Layer for the Azure Map visual This month we’re introducing a new feature to the Azure Map visual that takes geospatial analytics to the next level -- the Path layer. The Path layer provides users with the ability to visualize geographic connections between multiple points. Whether you’re managing logistics, analyzing network traffic, or tracking asset shipment across the globe, this feature allows you to visualize connections between multiple geographic points in an intuitive and interactive way. The Path layer is ideal for several key scenarios, for example: Network Analysis: For industries like telecommunications, the Path Layer enables you to map intricate network connections. It helps identify inefficiencies, monitor data flow, and strengthen critical infrastructure. Flight Path Analysis: Airlines can leverage the Path Layer to visualize and analyze flight routes, improving air traffic management. It helps identify new route opportunities and enhances the overall passenger experience by optimizing existing routes. To get started, add the location for each point using either a geocoded location field, such as city names, or latitude and longitude. Then, differentiate between the paths by adding a field to the Path ID field well and indicate the order of connection through the Point Order field well. For example, you could create a map showing the path of two ships with the latitude and longitude of their positions for each point, a path field with a unique identifier for each ship, and a timestamp for each location to make sure the points are connected in the correct order. You can also format the visual by controlling the color, transparency and width of the lines, and even turning off the bubble markers for each point. If you turn off the bubble layer, you’ll only see a bubble on hover showing you the closest point to your pointer location. Paths are interactive as well, so you’ll get tooltips on hover and be able to cross-highlight other visuals by clicking on points of the lines. There are a couple of unique behaviors to be aware of with this new layer. First, when using a drill hierarchy with the path layer, the visual will automatically drill down to the lowest level and will not allow you to drill up, as points in the path would be aggregated at higher drill levels. Next, if you have a location that’s part of multiple paths, the bubbles for that location show up on top of each other. If you want to click on the bubbles underneath, just hover on the line associated with its path, and it will float to the top and be selectable. Lastly, you can further break down the paths by adding a legend, which will create unique lines for each legend value of a given path ID. An additional point to consider: Currently, the path layer operates mainly in conjunction with the bubble layer. Once you add a path to your map, you’ll see that the filled, cluster bubbles, heat map, and 3D column layers are all disabled. Additionally, while you can use the path layer in conjunction with reference layers, the reference layer will be static. It’s currently unsupported to mix data bound reference layers with the path layer. The path layer is still actively rolling out to all regions. Depending on what region your tenant is in, you might not see the path layer in the Power BI service through the weekend. Be sure to check the report after publishing, and if you don't see the layer, it should be accessible within a week. We’re excited to see what you create with this new path layer. Give it a try and let us know what features you’d like to see next! Visual calculations (Preview) The work on visual calculations continues as usual and this month we are adding a highly requested item: support for exporting! You can now export data from visuals that contain one or more visual calculations or hidden fields. If you export data, hidden fields on a visual are not included in the export, unless you export the underlying data. The results of visual calculations are always included in the export, except when exporting underlying data, since visual calculations are not part of the underlying data as they only exist on the visual. Learn more about visual calculations in our documentation. ‘Set Alert’ with Activator and Real-Time Intelligence (Generally Available) Back in December we announced the preview of alerting capabilities within Power BI reports using Real-Time Intelligence Activator, part of Microsoft Fabric. We are excited to announce that this capability is now generally available! With GA, you’ll be able to: Stay on top of your critical metrics by monitoring your business objects. You can track and analyze key business objects such as individual packages, households, refrigerators, and more in real-time, ensuring you have the insight needed to make informed decisions. Whether it’s understanding how individual instances of your business objects impact sales figures, inventory levels, or customer interactions, our monitoring system provides detailed insights, helping you stay proactive and responsive to changes in your business environment at a fine-tuned level of granularity. Unlock the full potential of creating business rules on your data with advanced data filtering and monitoring capabilities. This update offers a wide array of options for filtering, summarizing, and scoping your data, allowing you to tailor your analysis to your specific needs. You can set up complex conditions to track when data values change, exceed certain thresholds, or when no new data has arrived within a specified timeframe. Ensure your communications are perfect before hitting send by seeing a preview of your Email and Teams messages. This will allow you to see a preview of your message exactly as it will appear to the recipient. Review your content, check formatting, and make any necessary adjustments to ensure clarity. With this feature, you can confidently have Data Activator send messages on your behalf knowing they look just the way you intended. Set up rules that trigger automatically with every new event that comes in on your stream of data. Whether you need to send notifications or initiate workflows, this feature ensures that your processes are always up-to-date and responsive. We renamed our feature to help create clarity about what it is and what it does. If you are used to seeing Reflex, please note that it is now called Activator. The items you create to set up rules and actions are, therefore, activators. We also enabled capacity usage reporting, to help you better understand your capacity consumption and future charges. Our billing is based on Storage used for events retention, and Compute resources: the number of rules running, the number of events per second ingested, rules evaluation and activation. For more on Activator meters and billing stay tuned for the detailed RTI Billing Blog post coming soon. You can learn more about the updates in GA through our blog. We continue to improve our capabilities, and we’d love to hear your feedback. Please share your ideas or suggestions. Small multiples for the new card visual (Preview) With this month’s update, we’re enhancing the Card visual with a new version that retains all familiar features and updates, while adding advanced functionality and an improved user experience with small multiples. This new feature is currently in preview with the new Card visual, offering an excellent opportunity to experience the capabilities of the feature. Small multiples are a series of similar card tiles displayed together in a grid format, each representing a different category or dimension of data, allowing for easy OKR comparison and analysis across multiple fields. This newly added feature enhances data organization, visual clarity, and performance, making it easier to analyze and present data effectively. To try it, navigate to Options and settings > Options > Preview features > New card visual, and make sure it’s enabled. Another advantage of the new Small multiples feature is the extensive customization it offers, including: Small multiples layout: Choose from single column, single row, or grid, and customize the number of small multiples, rows, or columns displayed. Advanced formatting options: Enhanced features such as font styles, color-coding, and conditional formatting. Border and gridlines: When enabled, individual controls for borders and gridlines permit the customization of style, width, color, and transparency. Overflow style: Options include continuous scroll or paginated, to smoothly navigate through multiple cards without overwhelming visual space. Headers: Choose from horizontal or vertical orientation, top or left position, customizable alignment, font, color, transparency, padding, plus background color or image. To create a card visual with Small multiples, first select the Card (new) icon from the visual gallery on the Build visual tab in the Visualizations pane, then select some data fields from the data model to add them to the data field well. To categorize your cards using small multiples, choose a data field from the data model and add it to the Small multiples data field well. This new feature provides extensive customization options, such as layout, advanced formatting options, conditional formatting, borders and gridlines, overflow style, and customizable headers. Small multiples for the Card visual in Power BI offer another great enhancement that significantly improves data organization, visual clarity, and performance. The Core Visuals team continues to add new features and greater functionality, and we’re committed to improving our capabilities. We invite you to explore this new feature and share your feedback with us in the comment section below as we continue to improve our Card visual capabilities. For more information we encourage you visit the Core Visuals blog on LinkedIn. New visual - text slicer (Preview) Introducing the new text slicer, now available in our core visuals gallery. This month brings the arrival of the new text slicer in Power BI offering new possibilities for both users and the organization. Enable the new text slicer by navigating to Options and settings > Options > Preview features > text slicer visual to ensure its selected, and restart Power BI. The text slicer works by allowing users to input specific text that acts as a filter, targeting a designated data field. By entering the desired text in the slicer’s input box, the slicer effectively narrows down the dataset to display only the relevant information that contains the entered text. This functionality is particularly useful for handling large datasets, where quick and precise filtering is essential for efficient data analysis and presentation. To create a text slicer visual, select the text slicer icon from the visual gallery on the Build visual tab in the Visualizations pane. This adds a visual placeholder to the report canvas. To filter a dataset, add a text field from the data model to the Field well to establish the text slicer's functionality, allowing it to filter the dataset based on user input. Simply add text to the slicer’s input box, select the apply icon, or press enter, and the slicer immediately filters the dataset, displaying results on the visual. As shown here, the new text slicer introduces a powerful and customizable filtering tool in Power BI: Improved user experience: The text slicer provides users with a straightforward and efficient method to filter input. Unmatched customization: It offers numerous options for users to tailor their filter experience to their needs and preferences. The Core Visuals team is dedicated to enhancing our features and functionality continuously. We are committed to advancing our capabilities and highly value your feedback. Kindly share your insights regarding this capability in the comments section below. For more information we encourage you visit the Core Visuals blog on LinkedIn. Reporting demos https://youtu.be/eyjVj-k8m1M?si=kH1kS1oSDCy5_xZK&t=516 Modeling Define new measure in DAX query view quick queries Creating measures in DAX query view just became even easier. The quick queries option available from the context menu of tables, columns, or other items in the Data pane, now includes Define new measure. This will create a new query table with the syntax started for you to create a query-scoped measure DAX formula ready for you to add your own DAX formula and then run when ready. Learn more about DAX query view and the other quick queries available at DAX query view - Power BI | Microsoft Learn. Metric sets: a new era of metric management in Fabric (Preview) The preview of metric sets is now officially available for both service and desktop. This is a transformative new feature designed to redefine how organizations manage and consume metrics. The Fabric Metric Layer’s home base is the Metrics Hub in Power BI and brings powerful capabilities to streamline metric management, ensure consistency, and foster trust in data across your organization. Metric sets will be available for both consumers to browse, and creators to use in reporting. A service experience consisting of visualized metrics and data exploration will allow end users to answer their data questions. The desktop experience will allow creators to connect to the most authoritative metrics to visualize in reports. Key Features: Curated Collection of Metrics: metric sets will serve as a collection of measure pointers to source semantic models and include key dimensions so end users and authors alike can unambiguously understand how a metric should be grouped or used. Rich Consumption Experiences: Users can explore and consume metrics from the metric set itself, allowing for deep insights and understanding. Copilot summaries and multiple visuals will be available for users to scroll through and go from data to insights in seconds. Efficiency: Consumers no longer need to rely on report creators to answer questions or build custom reports for specific needs. Consumers can leverage the Explore dialog to dig deeper into a given metric in an environment where everything in the data pane ‘just works’ because the dimensions have been curated specifically for the metric. Discoverability and Reuse: Consumers - Metrics are discoverable via search, and metric sets can be promoted, endorsed, certified just like any artifact so that users trust it. Consumers can also leverage the Explore dialog to dig deeper into a given metric in a safe environment where everything in the data pane ‘just works’ because the dimensions have been curated specifically for the metric. Authors- Metrics in Desktop: In the November release of desktop, metric sets will be available to connect to and use in desktop reporting. You can access the metric you want to include in your model via OneLake datahub / data catalog and connect there. This ensures your reports use the most up to date and authoritative measures available. Stay tuned for the upcoming milestones and get ready to transform your metric management experience with Metrics Hub! Performance improvements for models with calculation groups and format strings in Excel We’re excited to announce significant performance improvements for MDX queries on models with Calculation Groups and Format Strings! The latest changes should greatly improve the performance and reliability of operations in Analyze in Excel on models that include one or both of: Dynamic Format Strings for Measures. Calculated Items with Format Strings. This extends to other MDX scenarios as well, so all client applications that use MDX to query semantic models with the above will experience the same performance benefits. DLP policies restrict access action for semantic models (Preview) Purview data loss prevention policies for Fabric now enable admins to restrict access based on the sensitive information detected within their semantic models’ data. When Purview compliance admins configure DLP policies for Fabric, they now have the option to decide if upon detecting sensitive information they would like to block access to the data. They have the option to prevent guest users from accessing the data or to restrict access for all users except the data administrator. In Fabric, data admins will see an indication that their data is restricted, and can act, such as reporting an issue to the compliance admin or override the policy rule. Consumers, such as guest users who have been now restricted from seeing this information, also see an indication letting them know that an organization policy revoked their access, and if they attempt to see its content, they will not be able to. With restrict access action, compliance admins get further control and enforcement when uncovering sensitive data in their Fabric tenant. Semantic modeling in Visual Studio Code with the new TMDL extension (Preview) Power BI developers, the new TMDL Extension for Visual Studio Code in public preview enhances your TMDL editing experience, boosting semantic model development. The Tabular Model Definition Language (TMDL) is designed to make model representations readable, editable, collaborative, and reusable. The TMDL Extension builds on these strengths of TMDL with several key features that create a rich development experience: Semantic Highlighting: Improves readability by applying different colors to parts of your code based on meaning, making it easier to understand the structure and functionality of your TMDL briefly. Error Diagnostics: Helps you identify and fix issues in your code by clearly highlighting errors and providing you with detailed messages that guide you on how to resolve them. Autocomplete: Offers intelligent suggestions while you type to speed up your workflow, reduce the chance of errors, and help you understand your code options More features on the way! By working in Visual Studio Code, you can also take advantage of other fantastic tools on the platform such as: Source Control: Seamless integration with Git, allowing you to track changes, collaborate with team members, and version control your semantic models. GitHub Copilot: An AI coding assistant that will help you write code faster, generate TMDL from natural language, and quickly apply advanced bulk edits to your models. Download the TMDL Extension on the Visual Studio Marketplace and see how you can accelerate your semantic model development today! Modeling demos https://youtu.be/eyjVj-k8m1M?si=IvBsLYzRyRaOZKI2&t=1104 Developers + APIs Fabric Git: TMDL format for semantic model export As part of our commitment to providing a developer-friendly experience that enhances team collaboration, Fabric Git integration will begin exporting semantic model definitions as Tabular Model Definition Language (TMDL) in January 2025. This change will replace the use of a single JSON file (model.bim) with Tabular Model Scripting Language (TMSL). Due to its folder representation and readable format, TMDL offers a significantly improved source control experience. This enhancement facilitates tracking commit history and simplifies the resolution of merge conflicts, particularly when compared with TMSL. If necessary, you can continue to obtain the TMSL representation of your semantic model by using the Get Semantic Model Definition REST API or XMLA Endpoint. Semantic model client library updates Client applications, such as Excel or Power BI Desktop, connecting to Power BI semantic models now benefit from better performance due to an automatic conversion of legacy connection strings (e.g. pbiazure://*) to the XMLA endpoint. Requests are routed directly through the XMLA endpoint, reducing intermediary steps, speeding up request processing, and decreasing the likelihood of errors. You may need to update your firewall rules. See the troubleshooting document for details. Please ensure that you are using the latest Analysis Services client libraries for optimal performance when connecting to Power BI semantic models. Visualizations KPI by Powerviz KPI by Powerviz (Power-BI Certified) is a powerful custom visual for Power BI that allows users to visualize and create eye-catching and advanced Key Performance Indicators (KPIs). Key Features: 100+ Prebuilt KPI templates within visual and option to create own templates. Design: 16 layers and 40+ chart variations to create infographic designs. Rich customization, formatting options, and color styles. Create KPI objects in layers, combining charts, metrics, and icons. Analytical: Data Visualization Types: Categorical: Compare values across categories. Comparison: Analyze differences between values. Composition: Show parts of a whole. Progression: Display trends over time. Actual vs Target: Compare actual against targets. Formatting Features: Configure the Ranking, Sorting, Axis, Number-Formatting, Tooltip, Gridlines, Data Labels and Series Labels for visuals. IBCS Theme Support: Includes deviation bars, series labels, and consistent color scheme. Small Multiples: Support for all chart types – Fixed/Fluid with change chart feature. Other features include multi-categories comparison, Highlight values, Layer Flexibility, and more. Business Use Cases: Sales Performance, Financial Health, Customer Satisfaction. Try KPI Visual for FREE! Check out all features of the visual Step-by-step instructions YouTube Video Link Learn more about visuals Follow Powerviz Zebra BI Tables 7.3 With Zebra BI Tables 7.3, users can harness the power of a rich text editor to create and update visual comments with remarkable efficiency. This feature empowers you to style and format your text, add bullet points, and insert hyperlinks, making your report a one-stop shop for the entire team by just leaving links to reports and documentation different stakeholders might have an interest in. Well-structured comments can streamline communication within your reports, enabling readers to quickly grasp essential insights. By emphasizing what’s important and explaining why it matters, you guide your audience towards critical information and promote clarity and understanding. This clarity is crucial in any business environment, where time is often limited, and strategic decisions must be made swiftly. Effective comments reduce the time and effort required to generate actionable insights, which ultimately improves report quality and effectiveness. Incorporating thoughtful commentary can transform a standard report into a powerful tool for decision-making. With Zebra BI Tables, enhancing your reports with meaningful comments has never been easier -- all so you can communicate your message more effectively and engage your audience better. Learn more from our video example of the rich text editor in Zebra BI Tables 7.3. Waterfall PRO by ZoomCharts: the most interactive waterfall visual for financial data Waterfall PRO by ZoomCharts is the most user-friendly and insightful way to visualize financial data, combining incredible user experience with customizability and powerful features. It also seamlessly cross-filters data across multiple visuals, allowing you to create truly interactive Power BI reports. Main Features: Custom Sequence: Have full control over the column order with the Sequence field. Drill Down: Use multiple categories to enable drill down directly on the waterfall chart. Automatic Subtotal Calculation: Display subtotals even if you don’t have them in your data. Rich Customization: Customize X and Y axes, legends, tooltip content, and adjust the appearance settings for positive, negative and total columns separately Thresholds: Display up to four constant or dynamic thresholds as lines or areas. Cross-chart filtering: Dynamically filter data across multiple visuals. Get Drill Down Waterfall PRO on AppSource Visit Product Page Lollipop bar chart by Nova Silva We’re thrilled to continue receiving your valuable feedback, and we appreciate your contributions in helping us improve our visuals. In our latest Lollipop Bar Chart release for Power BI, we’ve added a much-requested feature: secondary markers. This allows you to display not only the primary value but also add context by including a secondary value marker. This new feature integrates seamlessly with all other Lollipop Bar Chart functionalities, such as transforming the Lollipop Bar Chart into a dot plot by removing the connecting bars, as shown in the second image. This also removes the requirement to start your numeric scale at 0, allowing you to have a closer look at the values and their differences. While standard bar charts are great for comparing a single measure across categories, they can become cluttered with larger datasets (>10 categories). The colored bars may fill too much of the chart space. To address this, the Lollipop Bar Chart offers a cleaner, more efficient alternative, minimizing clutter without sacrificing clarity. Try the Lollipop Bar Chart for FREE now on your own data by downloading it from the AppSource. Questions or remarks? Visit us at: https://visuals.novasilva.com/. Sales velocity chart The Sales Velocity chart is a unique tool for analyzing product sales and profitability in specific countries. It uses a combination of pie charts, needles, and color coding to visually represent key metrics. Key Features: Needles: Length indicates sales percentage; width reflects profit margin. Pie & Circle Size: Reflects overall current sales in a country. Color Coding: Green (high profit), Yellow (moderate), Red (low profit). Sales Trend Dot: Gray (no data), Red (decreasing sales), Green (increasing sales). Benefits: Visual Clarity: Easy to understand data representation. Dynamic & Scalable: Handles large datasets and adapts to screen size. Interactive Features: Tooltip displays details, premium options offer filtering and logo removal. Use Cases: Businesses can identify top sales regions and areas needing improvement. Financial analysts can pinpoint high and low profit contributors. Note: For more information, visit our website. Watch a short video. Read this Sales Velocity Chart documentation. For any queries, questions, or requests, please write to us. Donut Chart by JTA An innovative visualization tool that segments data into three clear categories: Positive, Neutral, and Negative. This format is particularly effective for sentiment analysis, offering clear insights into the overall distribution of opinions or data points. Enhance your data visualization effortlessly with this versatile tool. Key Features: Personalize Colors: Tailor the look of your chart by adjusting the color scheme of each slice to reflect your brand or style. Customize Text: Make it uniquely yours! Modify titles, legends, values, and percentages, adjusting margins, colors, fonts, and alignments to perfectly match your design preferences. Shape the Visual: Personalize the entire chart—adjust the circumference, tweak the colors, and refine the overall look and feel to suit your needs. Target Comparison: Easily compare your metrics against specific targets for clearer insight. Icon Customization: Set your own indicators! Choose custom icons to represent performance below or above your target. Conditional Formatting: Effortlessly apply color-coded formatting to highlight how values measure up against their target. Animation Control: Smooth transitions! Enable or disable animations to enhance or streamline your visual experience. Download Decomposition Tree by JTA for free: AppSource Try Decomposition Tree by JTA: Demo Learn more about us: JTA The Data Scientists New book: Data Visualization with Microsoft Power BI We recommend the new book ‘Data Visualization with Microsoft Power BI’ by Alex Kolokolov & Maxim Zelensky, the first book that delivers DataViz best practices for Power BI! 25 chapters about different chart types. 40 visuals: from default to advanced from the AppSource gallery. 400 color pages of an exceptional quality. The book is suitable for non-technical professionals as well as for experienced data analysts, it consists of 3 parts: Classic Visuals - Authors explain how to choose charts for basic types of analysis and avoid common mistakes. How to set up interactions and put visuals together on a dashboard. Trusted Advanced Visuals - Different options and data requirements for waterfall and bullet charts, Gantt, tornado, funnel, Sankey, etc. Risky Advanced Visuals - ‘Eye-catching’ charts that may confuse the average user. We explain use cases and offer simpler alternatives. Book features: Beautiful examples, specific use cases for charts. Step-by-step guides on how to set it up in the app. Data preparation tips and tricks. Quizzes to consolidate the learning material. “I want to inspire people to use Power BI for more than just reporting. I want them to create brilliant dashboards and tell interactive data stories!” - Alex Kolokolov The book is now available on Amazon. Other Support for Power BI language settings when a paginated report is viewed on the Power BI service When a localized paginated report is published to the Power BI service, the viewer of the report will now see the report in the preferred language that they have selected in the Power BI/ Fabric Settings page. Previously, the rendering of the report was determined by the server settings. Learn more about viewing localized paginated reports on the Power BI service. Platform Introducing OneLake catalog OneLake catalog is the next evolution of the OneLake data hub. Providing a unified experience, where data engineers, data scientists, analysts, and decision-makers can browse, manage, and govern all their data from a single, intuitive location. The OneLake catalog now includes various item types in Fabric, such as dashboards and reports (available by the end of November), dataflows, pipelines, and more. Streamlined for collaboration OneLake catalog offers filtering capabilities to help users find specific items efficiently. Business users can uncover reports and dashboards to answer their questions, while analysts can explore data items and processes for deeper analysis. In-place data management OneLake catalog allows you to view and manage any item directly within the catalog itself, simplifying navigation and enhancing efficiency. This contextual management ensures that you can handle your data ecosystem more effectively. In-depth item metadata Clicking on any item in the OneLake catalog reveals relevant metadata, including descriptions, tags, endorsement and sensitivity labels. The catalog also provides a granular view of your data items schemas and objects (e.g. warehouse tables and views), enabling better insight and control. Unified management and governance OneLake catalog combines crucial functionalities such as cross-workspace item lineage, access permissions, and real-time activity monitoring within a single interface. This unified approach makes governance and management tasks more accessible for every user. Explore the catalog today Explore OneLake catalog to experience the future of data management in Fabric. OneLake catalog is available in more than 40 scenarios where users connect to data within Fabric. It is also accessible in services and applications outside of the Fabric service application, such as Power BI Desktop and Azure Ibiza, with plans to expand to Excel soon. To learn more about this update, find detailed information that covers all the features and benefits of the catalog in depth. Tenant switcher control The tenant switcher is now available in the Fabric portal. Users with access to more than one Fabric tenant can easily switch between tenants directly from the account manager in the top right corner of the Fabric portal. This is in addition to the existing From External Orgs tab that can be found in the home page of the Power BI experience. Automate GitHub integration with Microsoft Fabric REST APIs Introducing the new REST APIs for Git integration with GitHub! These APIs enable you to automate Git integration tasks, such as connecting to GitHub, retrieving connection details, committing changes to your connected GitHub repository, updating from the repository, and more. For more information about the APIs and find available code samples. Switch branches from the source control pane Switching connected Git branches is now available directly through the source control pane. All branching actions can now be accessed in one place within the branches tab. This allows you to: Branch out to new workspace to create a new workspace with a new connected branch. Checkout branch to create a new branch while keeping the current workspace state, useful for resolving conflicts. Switch branch to replace the current workspace content with another branch, new or existing. Learn more details for these actions. Announcing general availability of the Fabric Workload Development Kit The Microsoft Fabric Workload Development Kit is now generally available. This feature allows Fabric to extend additional workloads and offers a robust developer toolkit for designing, developing, and interoperating with Microsoft Fabric using frontend SDKs (Software Development Kits) and backend RESTful APIs (Application Programming Interfaces). See feature blog to learn more. This release includes new features and enables users to start using Partner Workloads, which will be available in the workload hub in the coming weeks. Go to the Workload hub OneLake External data sharing is now generally available The external data sharing feature announced earlier this year is now generally available. External data sharing enables the sharing of OneLake tables and folders across tenant boundaries. In the current release, each share may include a single folder or a table from a Lakehouse. In the coming releases, support will be added for multiple folders and tables in a single share as well as sharing from Warehouses and Eventhouses. For more information check out the documentation. Mirroring Introducing Open Mirroring Introducing Open Mirroring, our new Mirroring capability. When we created Microsoft Fabric, we designed our platform to be extensible, customizable, and open. With that in mind, Open Mirroring, now in preview, is a powerful feature that enhances Fabric’s extensibility by allowing any application or data provider to bring their data estate directly into OneLake with minimal effort. By enabling data providers and applications to write change data directly into a mirrored database within Fabric, Open Mirroring simplifies the handling of complex data changes, ensuring that all mirrored data is continuously up-to-date and ready for analysis. For those looking to expand their data processing and analytics capabilities within Microsoft Fabric, Open Mirroring brings a flexible and powerful solution to ensure your data remains in sync, accessible, and analytics-ready within OneLake. Our Fabric partners such as Striim, OCI Golden Gate, and MongoDB already have capabilities to integrate with Open Mirroring, with DataStax integration coming soon. This enables any organizations to leverage a broader ecosystem of tools, enriching their data processing and analytics within the Fabric environment. Learn more about Open Mirroring in the Introducing Open Mirroring in Microsoft Fabric blog post. Fabric Database Mirroring Public REST APIs are now generally available Announcing the general availability of Fabric Databese Mirroring Public Rest APIs. Users can now utilize Microsoft Fabric REST APIs to perform CRUDLE operations: Create a new Mirrored database in your Fabric workspace. Read existing Mirrored database to get the definition of the item. Update Mirrored database with changes to the definition. Delete existing Mirrored database to clean up your workspace. List all Mirrored database in a workspace to get all available mirrored database in your workspace. With a Mirrored database ID, you can also get additional status for the Mirrored database and its tables mirroring status. In addition, you can start and stop existing mirrored databases with public REST APIs as well. To learn more read mirrored database REST API. Mirroring for Azure SQL Database now Generally Available Mirroring for Azure SQL Database is now generally available. Mirroring is a simple, free and frictionless way to replicate a snapshot and incremental data changes from Azure SQL database to Fabric OneLake with data sync in near-real time. With the GA release, the following new features are now available: Support for Truncate Table in source database when Mirroring is active Address issues related to schema hierarchy and column mapping in Data Warehouse and Lake House experience To learn more Announcing the general availability (GA) of Fabric Mirroring for Azure SQL Database. Introducing Mirroring for Azure SQL Managed Instance (Preview) The Preview of Mirroring for Azure SQL Managed Instance is now available. Mirroring is a simple, free and frictionless way to replicate a snapshot and incremental data changes from Azure SQL Managed Instance to Fabric OneLake with data sync in near-real time. Fabric Mirroring offers a great alternative to running a project to set up an ETL process to enable insights into an operational database that is the subject of an analytics scenario. You can set up Fabric mirroring in just a few steps, choose tables to mirror and the data will start flowing. To make changes to which tables are mirrored, you just need to make a few clicks. At any point, it is easy to see the status of replication for all mirrored tables. All this setup, management and monitoring is integrated directly into Fabric UI. Before mirroring, ETL setups would require additional tooling for data replication, people expertise to set up, configure, monitor and maintain the ETL- and this is just to keep the replication going. Any changes to replication would again require queuing up and waiting for ETL experts to modify your pipelines. To learn more about this new and exciting capability of Mirroring for Azure SQL Managed Instance in Microsoft Fabric, please read more in the blog. Databases Introducing Fabric SQL database (Preview) SQL database is now available as a native solution in Microsoft Fabric (Public Preview) and is the first database offering to land in the new databases workload. This new offering is seamlessly integrated with the Fabric platform and includes unified billing through the Capacity units (CU) model. Whether you are working on small or large analytics projects, we have heard your feedback: you need database support in Microsoft Fabric. With the addition of Fabric databases, we are evolving Microsoft Fabric from an analytics platform into a data platform. Fabric now has everything you need for your GenAI apps: operational database support, analytical storage, real-time intelligence for data in motion, and top-tier visualization. This week, we also announced the preview of a new vector type and functions in Fabric SQL database and Azure SQL Database, making building AI apps much simpler. We have samples for how you can easily integrate with frameworks like LangChain, Semantic Kernel, and more. You can get started with SQL database in Fabric today. For more information, please see the Announcing Fabric SQL database Preview. Data Warehouse Cold query performance improvement Running a query with a cold cache presents several challenges. When data is not cached, it must be fetched from OneLake and transcoded from parquet file format structures into in-memory structures for query processing. This process can be time-consuming and impact overall performance. With our latest improvement, we have optimized both fetching data from the storage and the transcoding process, observing median cold query overhead reduction of 40%. Service principal support for Fabric Data Warehouse We’ve made a major enhancement in the way you can authenticate and manage your Fabric Data Warehouses: the introduction of service principal (SPN) support. This new feature empowers developers and administrators to automate processes, streamline operations, and increase security for their data workflows. Earlier we launched service principal support for various Microsoft Fabric items, including Lakehouses and Eventhouses. Now, this support extends to Fabric Data Warehouses, making it easier to connect, manage, and deploy warehouse solutions in a secure, scalable way without needing to rely on user identities. The feature provides the following benefits: Automation-friendly API Access: You can now create, update, read, and delete Warehouse items via Fabric REST APIs using service principals Seamless Integration with Client Tools: You can use tools like SQL Server Management Studio (SSMS) to connect to your Fabric Data Warehouses using service principals and run TSQL features like COPY INTO. Granular Access Control: Ability to provide granular-level access by using T-SQL commands like GRANT, administrators can assign specific permissions to service principals to control precisely which data and operations an SPN has access to. Improved DevOps and CI/CD Integration: By using service principals, developers can automate the deployment and management of data warehouse resources in their DevOps and Continuous Integration/Continuous Deployment (CI/CD) pipelines to ensure rapid and reliable delivery of data solutions. Data Engineering Notebook display chart upgrade The new and improved chart view is the latest enhancement to our notebook display. This update is designed to provide a more intuitive and powerful experience for visualizing your data by leveraging the built-in visualization tool on Fabric Notebook. Key Features: Multiple charts view: Now you can add up to 5 charts in one display() output widget, allowing you to create multiple charts based on different columns, and compare charts easily! Rich chart recommendation: Get chart suggestions when creating new charts or clicking the suggestion button, easy to get started with the rich chart template and summarized title and insightful recommendations of key-value pairs. Advanced Chart Editing: You can add, rename, delete charts, and configure chart options. A lot of new configurations are provided in this upgrade, like chart title and subtitles, legend, theme, label etc. All your configurations are saved immediately. Global Configuration: Easily filter and apply custom ranges to your data. These settings will be applied to both tables and charts. Interactive Toolbar: Hover over a chart to access a toolbar for exploring the chart, like zoom in, zoom out, select to zoom, reset, panning, etc. Toolbar settings won't be saved, allowing for temporary adjustments. Benefits of the Enhanced Chart View: Improved Data Visualization: The new chart view offers a more dynamic and interactive way to visualize your data, making it easier to identify trends and insights. User-Friendly Interface: The enhancements provide a seamless experience, allowing you to switch between table and chart views effortlessly. Customization Options: With the ability to configure chart options and apply global settings, you can tailor the visualizations to meet your specific needs. We’ll gradually add more advanced chart types based on the new UX framework, stay tuned! Getting Started: To access the Enhanced Chart View, just open your Fabric notebook and run the display(df) statement. If you're seeing the legacy UX, use the switch to go to the new UX. Fabric API for GraphQL is now generally available with exciting new features The Microsoft Fabric API for GraphQL is now generally available, marking a significant milestone in providing powerful, flexible, and efficient data access APIs in Fabric. In addition to important features made available last month (Service Principals support and code generation from the API editor), this release introduces several new capabilities aimed at enhancing your experience and expanding the possibilities of what can be achieved with your GraphQL API in Fabric, making it easier to harness the power of Fabric data in your business applications. New data sources: Azure SQL and Fabric SQL DB (Preview) integration for seamless data access. Access data sources with connections and saved credentials: Enhanced security and simplified access management. Logging and Monitoring Dashboard: Visual insights into API activity and detailed logging for better performance monitoring and troubleshooting. CI/CD Support: Git Integration and Deployment Pipelines for consistent and automated deployments. You can find more information about these exciting new features in our GA announcement blog. Esri’s ArcGIS GeoAnalytics integration with Fabric Spark (Preview) Esri is recognized as the global market leader in geographic information system (GIS) technology, location intelligence, and mapping, primarily through its flagship software, ArcGIS. Esri empowers businesses, governments, and communities to tackle the world's most pressing challenges through spatial analysis and location insight. Microsoft and Esri have collaborated to integrate spatial analytics in Fabric, with a preview set to launch soon. Our collaboration with Esri will introduce cutting-edge visual spatial analytics right within Microsoft Fabric Spark notebooks and Spark job definitions (across both Data Engineering and Data Science experiences). With its integrated product experience, it empowers Spark developers or data scientists to natively use ArcGIS capabilities to run GeoAnalytics functions and tools within Fabric Spark for transformation, enrichment, and pattern / trend analysis of data across different use cases without any need for separate installation and configuration. Here is an example to transform the data with ArcGIS spatial function to uncover the pattern of interest, for instance summarizing the total number of policies of insured properties by hexagonal bins. Here is another example to understand the impact of natural hazards or current events on insured properties by bringing a dataset with probabilities of hurricane force winds and spatially joining it with insured properties. Spatial join links insured properties with wind speed probabilities, and with that for each property we would know the likelihood of hurricane force winds and can run predictive models to assess potential insurance claims. To learn more about ArcGIS GeoAnalytics integration within Microsoft Fabric Spark, please refer to the documentation. Jar libraries are now supported in Fabric Environments Java Archive (JAR) files are a popular packaging format used in the Java ecosystem. They allow developers to bundle multiple files—such as Java class files, metadata, and resources—into a single, compressed archive for distribution. JAR files simplify the distribution and execution of Java applications and libraries by consolidating everything into one file, which can be easily shared and managed. Previously, integrating JAR files into Fabric requires inline commands within notebooks. This approach, while functional, posed a challenge of reproducibility. And now, you can upload your JAR files as the custom libraries to the Environment. These custom libraries will be effective in the Notebooks and Spark jobs once attached to the Environment. Embracing JAR files within Fabric Environments can streamline your development and deployment processes, enhancing the overall efficiency and scalability of your applications. Support of spaces and special characters in Delta table names Support of spaces and special characters in Delta table names in Microsoft Fabric is now available! This is a highly desired enhancement requested by Fabric customers. Now, you can name Delta tables using spaces, special characters and the encoding of your natural language in all Fabric experiences. Everything will work: Spark, Lakehouse, Notebooks, Warehouse, Power BI, Shortcuts creation, Metadata discovery, etc. Some restrictions apply, learn all about it in the documentation. The feature will be available over the next weeks across all Fabric regions worldwide. Data Engineering demos https://youtu.be/eyjVj-k8m1M?si=CPy-GGf3uHHGks4a&t=2175 Data Science Introducing low code AutoML AutoML, or Automated Machine Learning, is a process that automates the time-consuming and complex tasks of developing machine learning models. It simplifies the workflow by handling data preprocessing, feature engineering, model selection, and hyperparameter tuning, allowing users to focus on interpreting results and making decisions. We are introducing the new low code AutoML user experience in Fabric, designed to empower analysts and data scientists to quickly prototype and build machine learning models with ease. This innovative interface supports a variety of tasks, including regression, forecasting, classification, and multi-class classification. Getting started with the AutoML user experience is incredibly simple. Users can begin with an existing experiment, model, or notebook. All it takes is selecting the relevant files or tables from your lakehouse and specifying the desired ML task. For those who want more control, there are optional configurations available. You can choose your parallelization mode, deciding whether to train one Spark-based model at a time or to parallelize trials with Pandas by distributing them across all nodes on your Spark cluster. Additionally, the auto-features setting enables us to generate useful features for model training. One of the key features of the AutoML experience is its integration with MLflow. All generated models are tracked using MLflow and the existing Experiment items. This allows users to monitor all the details, such as metrics, parameters, model types, and model files, making it easy to compare different models generated from the AutoML trial. Learn more about Automated Machine Learning in Fabric. Data Science demo https://youtu.be/eyjVj-k8m1M?si=KqDXTvHWwHw9LqML&t=2602 Real-Time Intelligence Real-Time Intelligence is now generally available (GA)! Announced at Build 2024, Real-Time Intelligence includes a wide range of capabilities across ingestion, processing, analysis, transformation, visualization and taking action. All of this is supported by the Real-Time hub, the central place to discover and manage streaming data and start all related tasks. This month includes a wide range of improvements, read on for more information on each capability and stay tuned for a series of blogs describing the features in more detail. Please submit any feedback on our features at RTI ideas. Ingest & Process Announcing the general availability of Real-Time Hub Fabric Real-Time Hub is now generally available! The one enterprise-wide catalog that enables users to discover, connect to, explore and act upon streaming data & events from anywhere. Seamless integration with all Real-Time Intelligence services like Fabric Eventstreams, Eventhouse and Activator greatly accelerates time to insights. Fabric Real-time Hub was originally released to Public Preview at //Build 2024 and has since become one of the most broadly adopted features within Real-time Intelligence suite. At the same time, our customers continue to give us valuable feedback to make Real-Time Hub even better. And we are listening! Here are some of the recent improvements: What’s new? Simplify Azure Event Hubs source connection: We have simplified the experience when connecting to an existing Azure Event Hub. For users with the right permission to access the available Azure Event Hubs, a single click is all it takes for Fabric to automatically establish the connection to the source New Sources: Azure Service Bus, Apache Kafka, CDC from SQL Server on VM DB and CDC from Azure SQL Managed Instance are added to the “Connect data source” options. Rich Sample Scenarios: For users who are new to Fabric Real-Time Intelligence, we provide three streaming data samples for you to get started. Streams and KQL tables with read (or higher) permission: Users can discover streams and KQL tables that they have read (or higher) access to within Real-Time Hub, which allows them to discover more data streams shared with them. Generate Real-time Dashboards (preview): Users can now quickly and automatically create real-time dashboards by selecting ‘Create Real-Time dashboards’ on KQL Tables. This CoPilot assisted feature can take users input to generate the most common real-time dashboards within seconds. Fabric Events (preview): Customers will soon be able to build event-driven applications, trigger Notebooks and workflows or send emails and Teams IM when OneLake files/tables are created, deleted or renamed (OneLake events) and Jobs are started or completed (Job Events). Explore Data action on KQL tables (coming soon): customers will soon be able to explore the data of their KQL tables with no-code experience, to allow them to interact with the data without leaving the context that they are in Azure Data Explorer (ADX) Database Shortcut (coming soon): customers will soon be able to create database shortcut for their ADX clusters. This will allow customers to manage their ADX clusters more efficiently directly from Fabric. Real-Time Hub serves as the starting point for your Real-Time Intelligence journey. Please feel free to try it out and give us feedback through Ask Fabric Real-time Hub [email protected]. Announcing the general availability of Enhanced Eventstream Enhanced Eventstream is now generally available! This offers new features that improve your experience in building stream flows within Fabric Real-Time Intelligence. The enhancements include Edit and Live View modes, Default and Derived Streams, and Smart Routing, transforming how data engineers handle real-time data streams with greater flexibility and efficiency. Edit Mode and Live View: Eventstream now offers two separate modes, Edit Mode and Live View, to give you flexibility and control over your data streams. Edit mode lets you design and modify your data streaming flow without interrupting the active data streams. Live View gives real-time insight into the data flow, allowing you to monitor the ingestion, processing, and distribution of data streams within the Fabric. You can switch between the two modes using the button in the top-right corner. To learn more, visit: Edit and publish Microsoft Fabric eventstreams - Microsoft Fabric | Microsoft Learn Default and Derived Streams: Data stream is a dynamic and continuous flow of data, allowing you to set up real-time alerts, and feed into different types of data stores. A data stream is a continuous flow of dynamic data that allows for real-time alerts and diverse data storage options. Default stream is created automatically when a streaming source is added to Eventstream, capturing raw event data directly from the source and preparing it for transformation or analysis. Derived stream is a specialized stream that users can set up as a destination within Eventstream. After performing operations such as filtering and aggregating, the derived stream is ready for further analysis or consumption by other organization members through the Real-Time Hub. To learn more, visit: Create default and derived Fabric eventstreams - Microsoft Fabric | Microsoft Learn Content-based Routing: Customers are now able to design stream operations directly within Eventstream’s Edit mode, transforming and routing of real-time data streams. It lets you create stream processing logic and direct data streams based on their content right in the Eventstream editor. To learn more, visit: Route events based on content in Fabric event streams - Microsoft Fabric | Microsoft Learn Announcing the general availability of connector sources in Eventstream Connector sources in enhanced Eventstream are now generally available! This feature enables seamless connection of external real-time data streams to Fabric, allowing for an optimal out-of-the-box experience and more choices for real-time insights from a variety of sources. It supports well-known cloud services like Google Cloud and Amazon Kinesis, as well as database change data capture (CDC) streams through our new messaging connectors. These connectors utilize Kafka connect and Camel Kafka connectors for a flexible approach to data integration, ensuring broad connectivity across leading platforms. Additionally, Debezium is integrated for precise CDC stream capture. Below is the list of generally available connector sources: Confluent Cloud Kafka Amazon Kinesis Data Streams Google Cloud Pub/Sub Amazon MSK Kafka Azure SQL Database Change Data Capture (CDC) Azure SQL Managed Instance (CDC) SQL Server on VM DB (CDC) PostgreSQL DB (CDC) Azure Cosmos DB (CDC) MySQL DB (CDC) To learn more about sources regarding the details of the configuration, visit: Add and manage eventstream sources - Microsoft Fabric | Microsoft Learn. To ask for new connector sources, please contact [email protected]. Introducing Azure Service Bus Connector for Eventstream (Preview) Many enterprise customers rely on Azure Service Bus as a key message broker for managing queues and publishing subscribe topics. They want to integrate their messaging infrastructure with Fabric to enable seamless data streaming, high-performance processing, and real-time dashboards. Now, we're introducing the Azure Service Bus Connector for Eventstream! This connector allows you to stream messages from Azure Service Bus topics and queues directly into Eventstream. Once the messages are in Eventstream, you can process them in real time and route them to multiple destinations within Fabric. This new connector simplifies the integration process and empowers real-time, scalable data streaming from your Azure messaging sources. Below, you will find how to add an Azure Service Bus source in Eventstream's edit mode. New Fabric events (Preview) New Fabric event categories, namely OneLake events and job events, will be available at the end of November, in Preview in Real-Time Hub. These events can be used for real-time alerting and data processing through Reflex triggers and sending them to other destinations via Eventstreams. OneLake events allow you to get alerted when changes occur in your OneLake. For example, when new files or folders are created or deleted. Users can use these events to automate workflows such as triggering a Data pipeline via the Reflex. Job events provide detailed information about various job activities and statuses within Fabric. For example, status when a Data pipeline or Notebook is run. These events can include updates on job initiation, completion, failures, and any intermediary states or changes. Learn more about Fabric events. You can incorporate the two new Fabric events into Eventstream as a source if you want to direct these events to various destinations, including Eventhouse, Lakehouse, or your custom application via Eventstream's custom destination endpoint. To learn more about how to add and configure these sources, please visit Add and manage eventstream sources - Microsoft Fabric | Microsoft Learn. Eventstream Data Preview on database CDC sources The enhanced Eventstream now includes Data Preview for database CDC sources. This feature allows you to view a snapshot of your data from the source in both Edit mode and Live View mode. In Eventstream's Edit mode, the data preview captures a snapshot from the CDC source you configured, enabling you to infer the schema for configuring subsequent operators or destinations without having to publish it first and then return to Edit mode. You can access the data preview in the 'Test result' tab at the bottom pane by selecting the source node on the canvas in Edit mode. Similarly in Live View mode, the data preview provides you with a snapshot from your sources so that you can understand what the data looks like inside your sources. The supported connector sources are: Azure SQL Database Change Data Capture (CDC) Azure SQL Managed Instance (CDC) SQL Server on VM DB (CDC) PostgreSQL DB (CDC) Azure Cosmos DB (CDC) MySQL DB (CDC) Monitoring experience on connector sources with Runtime Logs and Data Insights in Eventstream Eventstream now offers Runtime Logs and Data Insights for the connector sources in Live View mode. With Runtime Logs, you can examine detailed logs generated by the connector engines for the specific connector, which assist in identifying failure causes or warnings. You can access this feature in the bottom pane of Eventstream by selecting the relevant connector source node on the canvas in Live View mode. Data Insights provides metrics that are from the connector engine, aiding users in monitoring the connector sources' status and performance. The Source Incoming/Outgoing Events display the number of records polled or produced by the task assigned to the specified source connector in the worker. To learn more, please visit: Monitoring status and performance of an Eventstream item Processing and routing events to Activator with Eventstream (Preview) Fabric Activator (previously Data Activator) is a no-code experience for automatically taking action when patterns or conditions are detected in data. You use the activator item (previously reflex item) to manage the rules and actions. Fabric event streams under Real-time Intelligence, represented by 'Eventstream' as a Fabric item, aims to establish a centralized place on the Fabric platform for seamlessly capturing real-time events from diverse sources, transforming it, and routing it to various destinations. Now, Eventstream supports processing and transforming events with business requirements before routing the events to the destination: Activator. When these transformed events reach Activator, you can establish rules or conditions for your alerts to monitor the events. To add this destination, simply choose Activator from the Destination menu in the ribbon while in Edit mode. To learn more about sources regarding the details of the configuration, visit: Add an Activator destination to an eventstream Introducing Eventstream’s CI/CD support Collaborating on data streaming solutions can be challenging, especially when multiple developers work in the same environment. Conflicts, versioning issues, and deployment inefficiencies often arise. The integration of Fabric CI/CD tools for Eventstream in Microsoft Fabric has been developed to address these challenges and improve team collaboration. Fabric offers complete CI/CD experience with a variety of tools, including Git integration and Deployment pipelines. By integrating Eventstream with these tools, developers can efficiently build and maintain Eventstream items from end-to-end in a web-based environment, while ensuring source control and smooth versioning across projects. Key features include: Git Integration for Eventstream: Developers can collaborate freely using versioning and branching with their favorite git tools e.g., GitHub and Azure DevOps, preventing conflicts and enabling seamless teamwork. Deployment Pipeline for Eventstream: Accelerate and standardize Eventstream deployments to various stages, such as testing and production, with minimal manual effort in the Fabric UI. Below, you will find how to commit an Eventstream change to a git repository: With these powerful CI/CD capabilities, you can streamline your development workflow for Eventstream, isolate your development environments, and collaborate effortlessly with your team. Experience faster, more reliable development with Eventstream's CI/CD support. Automate Eventstream Item Operations with Eventstream REST APIs Introducing Eventstream REST APIs, these APIs allow you to automate and manage Eventstream items programmatically, simplifying CI/CD workflows and making it easier to integrate Eventstream with external applications. With Eventstream REST APIs, you can: Automate Eventstream deployments within your CI/CD pipeline. Perform full CRUD (Create, Read, Update, Delete) operations on Eventstream items programmatically. Seamlessly integrate Fabric Eventstream into external applications. Scale your streaming solutions quickly and efficiently. By leveraging these REST APIs, you can create fully automated workflows that enhance the quality, reliability, and productivity of your Eventstream items. Stream Data to Eventstream Securely using Entra ID Authentication (Coming Soon) Introducing Entra ID authentication for Eventstream’s Custom Endpoint! This feature enhances security by allowing users to stream data to Eventstream without relying on SAS keys or connection strings, reducing the risk of unauthorized access. Entra ID authentication ties user permissions directly to the Fabric workspace access, ensuring that only authorized users can access the workspace and stream data to Eventstream. Check out the screenshot below to see how this feature appears in Eventstream’s Custom Endpoint! Additionally, Tenant Admins now have the option to disable Eventstream’s key-based authentication in tenant settings, further securing the eventstream by enforcing the use of Entra ID authentication only. Analyze & Transform Eventhouse monitoring (Preview) Fabric workspace monitoring is the centralized logging solution of Fabric. Workspace monitoring is designed to provide a seamless and consistent monitoring experience with end-to-end visibility across all Fabric items. Workspace monitoring is based on the Real-time Intelligence Eventhouse KQL database. Once Fabric Monitoring is enabled, a KQL database is created to store all the workspace items event logs. KQL databases are ideal for time series logs and metrics monitoring solutions. For each one of the supported items, one or more events or metrics tables are created. Here you can see the tables supporting Eventhouse query, command and ingestion monitoring, and semantic model query logs. The Eventhouse Monitoring offers 5 events and metrics tables: EventhouseQueryLogs – logs all Eventhouse KQL queries. EventhouseCommandLogs- logs all Eventhouse commands. EventhouseDataOperations – logs all successful data operations including Batch ingestions, Streaming seal operations (operations that store streaming data to database extents), Materialized views updates, and Update policy table updates. EventhouseIngestionResultLogs – logs all successful and failed ingestions. EventhouseMetrics- set of metrics that provide in depth monitoring of ingestions, materialized views, and continuous exports. Users can explore and directly query workspace monitoring tables using KQL or SQL, with example queries available in the documentation. Here is an example of monitoring queries stored in an Eventhouse KQL QuerySet: The workspace monitoring solution centrally monitors all the Power BI reports, semantic models and Eventhouses items created in the workspace. In some cases, the Power BI semantic models read data from KQL database sources, however semantic models queries are logged in workspace monitoring solution regardless of the Power BI report data source. Realtime Dashboards can be created on top of the workspace monitoring KQL database, providing an easy graphical monitoring user experience. Real time dashboard templates can be imported to provide an out of the box monitoring experience. In this example real time dashboard, you can see Semantic Model CPU usage monitoring, with a direct link to the underling KQL queries being run by the Power BI report. Users can troubleshoot issues by correlating events across semantic models and their underlying databases. Users can troubleshoot spikes and activities and investigate who is consuming these resources. The user can easily zoom into the exact spike time and determine which user, or application generated the usage peak. They can then drill down to the specific query log record. In summary, workspace Monitoring offers a centralized monitoring solution, allowing users to efficiently monitor and troubleshoot their workspace items. Specifically, for Eventhouse, query, command and ingestions events and metrics logging enable advanced Eventhouse monitoring capabilities. Learn more about Eventhouse Monitoring Eventhouse Query Acceleration for Shortcuts (Preview) Shortcuts are embedded references within OneLake that point to other files' store locations without moving the original data. Previously, you could create a shortcut to OneLake delta tables using Eventhouse and query the data, but performance lagged direct ingestion in Eventhouse, as shortcut queries lacked the powerful indexing and caching capabilities of Eventhouse. Query acceleration indexes and caches data landing in OneLake on the fly, allowing customers to run performant queries on large volumes of data. Customers can use this capability to analyze real-time streams coming directly into Eventhouse and combine it with data landing in OneLake either coming from mirrored databases, Warehouses, Lakehouses or Spark. Customers can expect significant improvements by enabling this capability, in some cases up to 50x and beyond. How to enable Query Acceleration? You will now see an option to enable Acceleration while creating a new shortcut from Eventhouse. Learn more about Real-Time Intelligence Synapse Data Explorer to Eventhouse migration (Preview) Synapse Data Explorer (SDX), part of Azure Synapse Analytics, is an enterprise analytics service that enables you to explore, analyze, and visualize large volumes of data using the familiar Kusto Query Language (KQL). SDX has been in public preview since 2019. The next generation of SDX offering is evolving to become Eventhouse, part of Fabric Real-Time Intelligence. Eventhouse offers the same powerful features and capabilities as SDX, but with enhanced scalability, performance, and security. For customers looking to migrate to Eventhouse from SDX, we are happy to announce a seamless migration capability. Customers can use the migration API to seamlessly move their SDX cluster to Eventhouse with minimal disruption, learn more. New explorer for database objects in KQL Queryset Effortlessly browse through the database your current Queryset is connected to, viewing tables, functions, materialized views, and more. Double-click any object to instantly copy its name to the query editor, making query writing easier than ever. When opening the data source switcher, you can easily refresh the data or disconnect it from the KQL Queryset if it’s no longer needed: In addition, you can now apply actions directly from the explorer. Simply click the ellipsis next to any object to access a menu with options tailored to your selection. Entity Diagram view in KQL Database (Coming Soon) A new feature in the KQL Database page enables you to visually explore relationships between database entities—such as tables, functions, materialized views, update policies, external tables, and continuous exports—through an interactive graph visualization. This helps you efficiently manage your database and gain a clearer understanding of how these entities interact. Sample Scenarios Proactively manage dependencies With this visual representation, you can easily manage dependencies between entities such as tables and functions. For instance, when renaming a table or modifying its schema, you can immediately see which functions are using that table as part of their KQL body. This proactive approach helps prevent unintended consequences and ensures smoother updates to your database structure. Track data sources behind Materialized Views The new feature also lets you trace the relationships between materialized views and their underlying source tables. This makes it simple to identify original data sources, allowing you to track and troubleshoot data flow more effectively. Interact with elements and act You can click on any element in the graph to see its related items, while the rest of the graph is greyed out, making it easier to focus on specific relationships. For tables and external tables, additional options become available, such as querying the table, creating a Power BI report based on the table, and more. Track record ingestion Additionally, you can easily track how many records have been ingested into each table and materialized view. This clear view of data flows helps you stay on top of ingestion size and volume, ensuring your database processes data correctly. Summary This visual enhancement simplifies database management and helps you optimize your data structures, making it easier to track dependencies and take actions quickly. Visualize & Act Easily share Real-Time Dashboards with others Microsoft Fabric’s new real-time dashboard permissions feature brings granular control to how users interact with real-time analytics. With the introduction of separate permissions for dashboards and underlying data, administrators now have the flexibility to allow users to view dashboards without giving access to the raw data. This separation is key for organizations that need to ensure data security while providing actionable insights to a broader audience. Fabric permissions focus on how users interact with the dashboard itself, determining who can view, edit, or share the dashboard. Meanwhile, data source permissions ensure that only authorized users can access the raw data that powers these visualizations. This division improves the overall security posture by ensuring users have only the necessary level of access. An added benefit of this feature is the option to choose between pass-through and editor’s identity for handling data access. Pass-through allows users to access data using their own credentials, while editor’s identity uses the dashboard editor’s permissions. This ensures that the system is adaptable to different collaboration scenarios and aligns with organizational needs. Overall, these enhancements to real-time dashboard permissions in Microsoft Fabric promote secure, efficient, and tailored access to data and dashboards. This flexibility empowers teams to collaborate more effectively while maintaining strict control over data access, making it a valuable addition for organizations leveraging real-time intelligence. Learn more about Real-Time Dashboards permissions (Preview). Announcing the general availability of Real-Time Dashboards Real -Time Dashboards is now generally available in Microsoft Fabric, bringing fast, actionable insights to your fingertips. Real-Time Dashboards make it easier than ever for organizations to track and act on key metrics in real-time, empowering faster decisions and deeper insights without the need for complex coding. Unlocking the Power of Live Insights With Real-Time Dashboards, you can monitor critical data events as they happen. Users can now set auto-refresh rates as low as 10 seconds or even continuous updates for real-time data streams, ensuring you stay up to date on every key metric. Flexible, Secure Data Sharing A new feature announced as part of the general availability of Real-Time Dashboards is the separation of permissions for dashboards and underlying data. Administrators can now grant dashboard access without exposing raw data, allowing teams to make data-driven decisions while maintaining strict data security. This separation of permissions is particularly valuable for organizations that need to ensure compliance and protect sensitive information while still sharing key insights broadly. Effortless, No-Code Data Exploration Our no-code ‘Explore Data’ functionality empowers users of all technical backgrounds to go beyond the dashboard’s insights. Now, anyone can dive deeper into metrics, explore underlying data, and analyze trends—all without needing to know KQL or write queries. With "Explore Data," you can filter, drill down, and interact with data using a user-friendly UI, gaining a clear understanding of what’s driving changes or fluctuations. Start Gaining Real-Time Insights Today Real-Time Dashboards in Microsoft Fabric are designed for users who want to transform data into action, faster. By delivering continuous updates, enhancing security, and offering intuitive data exploration, Real-Time Dashboards provide everything you need to harness the power of live data. Try it today and start unlocking the full potential of your data! Announcing the general availability of Activator Real-Time Intelligence Activator is now generally available! We would like to extend our gratitude for your invaluable partnership and feedback throughout Data Activator’s development as we help your organizations go from insights to action. With GA, you’ll be able to: Stay on top of your critical metrics by monitoring your business objects. You can track and analyze key business objects such as individual packages, households, refrigerators, and more in real-time, ensuring you have the insight needed to make informed decisions. Whether it’s understanding how individual instances of your business objects impact sales figures, inventory levels, or customer interactions, our monitoring system provides detailed insights, helping you stay proactive and responsive to changes in your business environment at a fine-tuned level of granularity. Unlock the full potential of creating business rules on your data with advanced data filtering and monitoring capabilities. This update offers a wide array of options for filtering, summarizing, and scoping your data, allowing you to tailor your analysis to your specific needs. You can set up complex conditions to track when data values change, exceed certain thresholds, or when no new data has arrived within a specified timeframe. Ensure your communications are perfect before hitting send by seeing a preview of your Email and Teams messages. This will allow you to see a preview of your message exactly as it will appear to the recipient. Review your content, check formatting, and make any necessary adjustments to ensure clarity. With this feature, you can confidently have Data Activator send messages on your behalf knowing they look just the way you intended. Set up rules that trigger automatically with every new event that comes in on your stream of data. Whether you need to send notifications or initiate workflows, this feature ensures that your processes are always up-to-date and responsive. We renamed our feature to help create clarity about what it is and what it does and simplify the way you discover Data Activator and create actionable rules. If you are used to seeing Reflex, please note that it is now called Activator. The items you create to set up rules and actions are, therefore, activators. You can find Activator tile in Fabric Real-Time Intelligence section. We hope that, however small, these changes simplify the process of getting started with Data Activator. Activator billing is now enabled, and it is based on the following four meters: Compute resources The number of rules running, and the duration of time these rules have been up. Each active rule comes with a uniform ‘uptime’ cost. The number of events per second ingested. Evaluating the rules and, when the conditions are met, triggering the defined action. Storage The Fabric storage charges are based on the storage consumed for events and activation retention. The retention policy by default is set to 30 days. Should your account run out of capacity, we will show you an in-product banner and send email notification. You can always track and review your Activator capacity usage and, if needed, update it to fit your business needs. The formal billing for Activator usage will begin with our GA announcement on Nov 18. To learn more, find our documentation Activator. As always, we’d love to hear your feedback and look forward to hearing from you. Stay tuned for the detailed RTI Billing Blog post for more details. Real-Time Intelligence demos https://youtu.be/eyjVj-k8m1M?si=X3w7sqJV7sSsAtR8&t=3416 Data Factory Table and partition refreshes added to semantic model refresh One of the most popular features that we built in Fabric Data Factory came from our customer patterns that we observed being used in ADF and from our community. That is the semantic model refresh activity. After first releasing this pipeline activity, we heard your request to improve your ELT pipeline processing by including an option to refresh specific tables and partitions in your semantic models. We are super pleased to announce that we’ve now enabled this feature making the pipeline activity the most effective way to refresh your Fabric semantic models. Learn more about Semantic model refresh activity Import and export your Fabric Data Factory pipelines As a Data Factory pipeline developer, you will often want to export your pipeline definition to share it with other developers or to reuse it in other workspaces. We’ve now added the capability to export and import your Data Factory pipelines from your Fabric workspace. This powerful feature will enable even more collaborative capabilities and will be invaluable when you troubleshoot your pipelines with our support teams. New connectors available In the Data Factory, both data pipeline and dataflow gen 2 now natively support the Fabric SQL Database connector as source and destination. Additionally, data pipeline expands its connectivity to include the ServiceNow connector (source) and MariaDB connector (source). Worth mentioning, Iceberg format is newly introduced in data pipeline first. As the first click-stop, you can now use data pipeline to write data as iceberg format via Azure Data Lake gen2 connector. Alongside these new connectors, there’re numerous feature enhancements to existing connectors. Highlights include continued improvements to the following connectors: The Snowflake connector with added support for the China domain in data pipeline and dataflow gen2. The Dataverse connector with enrichment on the authentication type support in data pipeline. Both PostgreSQL connector and Azure PostgreSQL connector with the capability to customize query timeout in data pipeline. Simplify data ingestion with Copy Job - CI/CD upsert & overwrite Copy Job simplifies data ingestion, providing a seamless experience from any source to any destination. Whether you need batch or incremental copying, Copy Job provides the flexibility to meet your data needs while keeping things simple and intuitive. Since the Public Preview launch at FabCon Europe in late September, we've been rapidly enhancing Copy Job with powerful new features. Here’s our latest update: Copy Job now supports CI/CD capabilities in Fabric, including Git integration for source control and ALM Deployment Pipelines. Check out the details in CI/CD for copy job in Data Factory - Microsoft Fabric | Microsoft Learn. Copy Job now also offers expanded writing options: Upsert functionality for SQL DB and SQL Server, and an Overwrite option for Fabric Lakehouse—bringing added flexibility and control to data movement. Check out the details in What is Copy job (preview) in Data Factory. New capabilities in Copilot for Data Factory to efficiently build and maintain your Data pipelines The new Data pipeline capabilities in Copilot for Data Factory are now available. The new capabilities are now in preview and serve as an AI assistant to help users to effortlessly create data integration solutions with data pipelines, to easily understand complex data pipelines and to efficiently troubleshoot data pipeline error messages. Create a new data pipeline and click on the Copilot button on the home tab to get started. Easily get started to build the pipeline with three starter options and clear guidance. Copilot for Data Factory can easily understand your intent and business requests to transform them into data integration solutions. You can either easily set up your data pipeline with pre-filled prompt step by step or you can efficiently create your data pipeline with a comprehensive prompt. Copilot for Data Factory also improves Data pipeline troubleshooting error messages experience. It provides clear explanations and actionable recommendations for you to identify and resolve the Data pipeline errors easily. Copilot for Data Factory can quickly summarize your pipeline for better understanding, which is extremely useful in collaborative scenarios involving complex data pipelines. You can get the complex pipeline summary either by clicking on ‘Summarize this pipeline’ option or sending ‘Summarize this pipeline’ prompt. Then you will get a very clear explanation of the complex pipeline developed by other team members. OneLake datahub is now the OneLake catalog in Modern Get Data experience We are pleased to announce that the OneLake datahub has been rebranded as the OneLake catalog in Modern Get Data. When you use Get data inside Pipeline, Copy job, Mirroring and Dataflow Gen2, you will find the OneLake datahub has been renamed to OneLake catalog. The OneLake catalog represents the next evolution of OneLake data hub, offering a cohesive platform. In the current Modern Get Data, the OneLake catalog will keep the same functionality as previous OneLake Datahub. In the future, we will expand the OneLake catalog functionality to allow data engineers, scientists, analysts, and decision-makers to seamlessly explore, organize, and oversee their data in one comprehensive and user-friendly location. Dataflows now support CI/CD (Preview) With Dataflows Gen2, you can now leverage the benefits of GIT integration and CI/CD support. By enabling GIT integration within your workspace, you can store your dataflow definitions into git and branch to other workspaces, collaborating on the same dataflow. This improves your end-to-end experience, especially when working across for dev, test, and prod workspaces. Get started today! We look forward to learning more from your feedback to improve the experience. Data Factory demos https://youtu.be/eyjVj-k8m1M?si=6TQuAgbOyT45idII&t=5141
Microsoft Fabric January 2025 update
We’ve got a lot of exciting updates this month. To name a few, NotebookUtils session management utilities, Enhancing COPY INTO operations with Granular Permissions in Data Warehouse, Application Lifecycle Management (ALM) and Fabric REST APIs. Keep reading to hear about everything we have in store for you this month. Microsoft Fabric Community Conference 2025 After 2 consecutive sold-out events, FabCon returns bigger than ever to Las Vegas from March 31 to April 2, 2025. 215+ sessions, 4 keynotes, and 20 workshops to take you from Power BI to Fabric to AI and help you make the most of Copilot and SQL in Fabric. Plus, one-on-one time with Microsoft experts and community legends, a FREE pre-day for partners, and the famous Power Hour. Prices go up on February 11, register ASAP and use MSCUST to get $150 off. First-ever Power BI DataViz World Championships Are you ready? The first-ever Power BI Data Visualization World Championships are coming to FabCon Vegas! Participate to learn and win a chance to compete live on-stage! Stay tuned to the World Championships blog for more details. Free training and discount certification vouchers for DP-700! The Fabric Data Engineer Certification is now generally available! The best way to prepare for Exam DP-700 is to join Microsoft Fabric experts for live and on-demand sessions. Sessions start this week. Register now and save your spot! Ready to take the exam now? Head over to the community to request your discount voucher for Exam DP-700. Contents Microsoft Fabric Community Conference 2025 Free training and discount certification vouchers for DP-700 First-ever Power BI DataViz World Championships Power BI Copilot and AI Unlock suggested questions from standard prompts in Copilot Reporting Explore this data: new entry point from a visual Storytelling in PowerPoint - New reset behavior Storytelling in PowerPoint - Supporting page up & down Save to OneDrive and SharePoint: updated file picker (Preview) Enhancement to Text slicer (preview) Enhancements to Treemap visual Modeling Semantic model version history (Preview) Edit your data model in the Power BI Service – updates (Preview) Live edit of semantic models in Direct Lake mode with Power BI Desktop – updates (Preview) TMDL scripting experience (Preview) Data connectivity New Snowflake connector implementation (Preview) Visualizations Drill Down Scatter PRO by ZoomCharts: The All-in-One Scatter Visual Lollipop Chart by Powerviz Other Now in Power BI Desktop - OneLake catalog Platform Folder support in Git OneLake OneLake Catalog - Semantic model table & column description Filtering workspaces in OneLake Catalog Data Engineering Python notebook (Preview) Notebook live versioning NotebookUtils session management utilities Native Execution Engine on Runtime 1.3: simplified enablement and transition from Runtime 1.2 Legacy timestamp support in Native Execution Engine on Runtime 1.3 Notebook and Spark Job definition execution with service principal Lineage Enhancement to Spark Notebook Data Warehouse COPY INTO column count check Enhancing COPY INTO operations with Granular Permissions in Data Warehouse Introducing default schema changes in Data Warehouse Enhanced performance metrics in Query Insights Previewing estimated Query Plan available via SHOWPLAN_XML Query Hints in Fabric Data Warehouse Simplifying search & introducing Filter in Object Explorer Open from SSMS & VS Code Git Status Bar to Fabric Warehouse Artifact Tooltip support for built-in functions Stay updated with IntelliSense JSON aggregates (Preview) Spatial analytic functions SQL analytics endpoint performance improvement Databases Tenant Level Private Link (Preview) Copilot for SQL database in Fabric Region Availability Real-time Intelligence Override late arrival tolerance in Activator Create new Activator items RTI ALM & APIs GA Data Factory Mirroring Copy Job Dataflow Gen2 Conclusion Power BI Copilot and AI Unlock suggested questions from standard prompts in Copilot When launching Copilot or using the prompt guide, you can select from standard prompts. In preview mode, a new prompt ‘Answer a question about the data’ will be available coming end of January. Selecting this prompt will unlock 3 suggested questions to help you explore your data. You can continue to select the prompt by scrolling up or from the prompt guide (book icon) to generate 3 more suggested questions if none of the first set are interesting to you. Authors can also personalize these suggested questions using Q&A setup in Desktop for a particular semantic model. With this feature, suggested questions will now show up in both Copilot and Q&A visual. You can continue to select the prompt by scrolling up or from the prompt guide (book icon) to generate 3 more suggested questions if none of the first set are interesting to you. Authors can also personalize these suggested questions using Q&A setup in Desktop for a particular semantic model. With this feature, suggested questions will now show up in both Copilot and Q&A visual. Reporting Explore this data: new entry point from a visual Exploring your data is easier than ever, now that we’ve added an Explore this data option to the visual options menu. This lightweight and focused experience allows users to launch Explore and easily tweak their visual (change chart type, add new data, filter, and more!) and see the underlying data, making it easy to get the answers they need without all the distractions and extra complexity of reports. Learn more about how to use Explore here! Simply select ‘Explore this data’ in the ‘More options’ menu and start exploring the visual. Storytelling in PowerPoint - New reset behavior When integrating a report into your presentation, it is important to ensure that it remains stable and unaltered. The add-in refreshes data from Power BI without modifying the report definition. However, since Power BI reports are dynamic, sometimes you may want PowerPoint to get the latest changes done in the report in Power BI service. Previously, you had to remove and re-embed the report to achieve this. Now, with the improved ‘Reset’ command, you can choose either to reset the add-in to its original state as initially added to the presentation or to reset and update it with the current view from Power BI. Storytelling in PowerPoint - Supporting page up & down You can now use the Page Down/Up keys on your keyboard to quickly navigate between slides when using the Power BI add-in. This is especially useful when the add-in captures the entire slide, and you want to advance the slide rather than trigger a Power BI event. Save to OneDrive and SharePoint: updated file picker (Preview) Updates have been made to the Power BI file picker to simplify navigation and file-saving processes. We have considered your feedback regarding the current file picker and have made significant improvements in this update to ensure the Power BI file picker aligns more closely with the Office experiences you are familiar with. New updates include: Improved experience when opening and saving files in One Drive and SharePoint. Easy access to reports in OneDrive and SharePoint. Navigating between folders in various workspaces. Adding new folders to existing workspaces. Pinning folders and files in the file picker. We acknowledge that these updates may disrupt your workflow; therefore, they are not enabled by default. To access the updates, open the desktop app, go to Options and settings (under the File Menu) > Options > Preview features, select the checkbox to enable ‘Show the new file saving and open experience’ then select ‘OK’ to accept the setting. By early next year, these settings will be on by default and will no longer require an opt-in. We hope you enjoy these new updates, and we’d love to hear any feedback you may have via our feedback forum. When submitting feedback, be sure to include ‘OneDrive and SharePoint’ and/or ‘Updated File Picker’ in the title. For more information, please check out our documentation on the Power BI Desktop and OneDrive + SharePoint integrations. Enhancement to Text slicer (preview) Following the November 2024 Text slicer release, this update enhances functionality and user experience by allowing multiple text selections. The Text slicer is currently in preview. To enable the Text slicer, go to Options and settings > Options > Preview features > Text slicer visual to make sure it is selected, then restart Power BI. This month’s enhancement adds a new Slicer settings control with an on/off toggle allowing the slicer to Accept multiple values. All other existing formatting options for the Input text, Apply button, and Input text box from our November update remain the same in the Format pane. After creating a Text slicer visual and adding a text field from the data model, users can filter the dataset based on user input. Simply select the slicer input box, type your text, and apply the filter either by selecting the apply icon, pressing enter, or selecting outside the visual. The slicer immediately filters and displays the results, and you can repeat these steps to add more text selections. When the Accept multiple values option is enabled, additional text can be added to the slicer by repeating these steps, thereby allowing multiple selections for filtering the dataset. Keep in mind that switching the toggle on or off will clear any previous text selections. Adding filtering with multiple values brings more control to data slicing, and we encourage users to explore this new feature and provide feedback. Future enhancements are still planned as we continue to improve Power BI's visualization capabilities with the Text slicer. The addition of filtering with multiple values offers enhanced control over data slicing, and users are invited to explore this feature and provide feedback. Further improvements are planned for Power BI's visualization capabilities with the Text slicer. Share your comments and suggestions in the comments section below and stay connected with us through our dedicated Core Visuals LinkedIn blog where we announce new features, updates, and engage with our community. Learn about our new Core Visuals Vision Board, where customers can now explore, vote, and comment on the Epic Ideas that will shape the future of Power BI Core Visuals. The Power BI community can instantly see what features are already completed, what is currently in development, and the new features and enhancements that are upcoming. Enhancements to Treemap visual This month's update includes significant enhancements to the Treemap visual, with three new tiling methods that improve layout options, plus new spacing controls to enhance the visual's appearance and usability. These features offer richer control and customization, resulting in more precise and aesthetically pleasing treemaps in Power BI. Treemap visuals are powerful tools for data visualization that allow users to represent hierarchical data through nested rectangles. Each branch of the hierarchy is represented by a rectangle, which is then tiled with smaller rectangles representing sub-branches. This structure allows for quick comparison of different category proportions. To generate two-level Treemap visuals, ensure that both the Category and Details fields are enabled. This allows you to visualize the hierarchical relationships between various categories and their subcategories in a clear and organized manner. Three new Tiling methods: Squarified: This method uses a squarified treemap algorithm to prevent elongated rectangles, creating a balanced layout. It arranges rectangles so their aspect ratios are close to squares, making size comparisons potentially easier. Binary: This method continuously divides the chart area into two sections while incrementally adding new rectangles/nodes creating a balanced and visually appealing treemap. Each hierarchy level further splits the space, resulting in an organized treemap that adapts to the dataset's structure. It may produce different visual characteristics compared to squarified algorithm depending on the dataset. Alternating (Columns, Rows): The Alternating method clearly distinguishes categories by first splitting them by columns and then within each column by rows. This method effectively organizes datasets with numerous hierarchical levels. This month's update also introduces new spacing options to enhance the readability and appearance of the Treemap visual: Space between all nodes: This setting introduces gaps between adjacent nodes at all hierarchy levels, reducing clutter and improving clarity. Space between groups: By adding extra space around each node group, this option visually separates different hierarchical groups, which helps to visually distinguish categories within the hierarchy. This update to our Treemap visual has brought improvements that reflect the commitment of the Core Visuals team to delivering the tools and features most requested by our users. Your feedback helps us refine and expand the capabilities of core visuals. Test these Treemap enhancements and share your thoughts in the comments section below or visit our Core Visuals LinkedIn blog, to leave comments, and find up-to-date news, developments, and announcements. Learn about our new Core Visuals Vision Board, where you users can vote on upcoming features and see what is in the pipeline. Together, we can continue to innovate and improve the tools that help our community to visualize data with Power BI. Modeling Semantic model version history (Preview) Announcing the public preview of semantic model version history coming this month. This feature aims to empower self-service users by providing confidence to recover from critical mistakes when editing semantic models on the web. In this preview, versions will be automatically captured in an Office-like history pane for your web-edited Premium semantic models. You can easily select and restore any of these previous versions of your semantic model. Additionally, you have the option to manually save versions to the version history for your semantic model. Stay tuned as we continue to roll out updates to this experience, including future support for semantic models in Pro workspaces. We highly value your feedback, so please share your thoughts using the feedback forum. For more details on this feature, including limitations, please refer to the documentation. Edit your data model in the Power BI Service – updates (Preview) The following improvements to the data model editing in the Service preview will be introduced this month: On by default preview for Premium workspaces With the release of semantic model version history, we will start enabling the workspace-level preview feature for editing data models in the service. The users can edit data models workspace setting will be turned on by default for Premium workspaces. If you prefer, you can still disable the workspace preview for your workspace, but we recommend keeping it enabled! Power BI administrators will still have the ability to enable or disable data model editing in the service for the entire organization or specific security groups through the admin portal. Viewing mode Now, when you open your semantic models on the web, it will default to Viewing mode. This allows you to easily view the model in a safe environment, preventing any accidental edits. When you're ready to make changes, simply toggle to Editing mode to make your modifications directly on the web. For more details on the subject, reference the documentation. Please continue to submit your feedback directly in the comments of this blog post or in the feedback forum. Live edit of semantic models in Direct Lake mode with Power BI Desktop – updates (Preview) On by default preview Live editing semantic models in Direct Lake mode with Power BI Desktop is now enabled by default, allowing you to use this feature immediately without needing to turn on the preview feature. If you prefer, you can still disable this feature by turning off the live edit of Power BI semantic models in Direct Lake mode preview in Options and Settings > Options > Preview features. More details on the feature, including requirements, considerations, and limitations can be found in the documentation. We highly value your feedback on this feature and encourage you to share it through our feedback form or the Power BI Community. TMDL scripting experience (Preview) TMDL view is a new view in Power BI Desktop that lets you script, modify, and apply changes to the semantic model being edited in Desktop with a modern code editor using Tabular Model Definition Language (TMDL), improving development efficiency, and providing complete visibility over the semantic model metadata. TMDL view offers an alternative experience to semantic modeling using code instead of a graphical user interface like Model view. Enhance development efficiency with a rich code editor that includes search-and-replace, keyboard shortcuts, multi-line edits, and more. Increase Reusability by easily script, share and reuse TMDL scripts among semantic model developers. For example, use a centralized SharePoint site to easily share reusable semantic model objects such as calendar tables or time intelligence calculation groups. Get more control and transparency, showing all semantic model objects and properties, and allowing changes to items not available in Desktop GUI, such as IsAvailableInMDX or DetailRowsDefinition. Script any semantic model object such as table, measure, column or perspective by selecting the objects from Data pane and dragging them into the code editor: TMDL view will script the selected objects as a TMDL script and just like TMDL in VS Code you get an enriched code experience with features such as semantic highlighting, error diagnostics and autocomplete. You may change any valid property or object within the semantic model. For instance, the example below demonstrates how to modify the displayFolder property and detail rows definition of multiple measures: When ready you can hit the Apply button to execute the TMDL script against the semantic model to get your changes applied: When successful, an instant notification will be displayed, and your modeling change will be applied to the semantic model. In the event of a failure, your modeling changes will not be applied to the semantic model, and you can view more information about the error by selecting on show details, which expands the Output pane with the error details. Get started today by turning on this public preview feature, go to File > Options and settings > Options > Preview features and check the box next to TMDL View. To learn more about TMDL View refer to our documentation. Data connectivity New Snowflake connector implementation (Preview) We continue to enhance the integration with Snowflake. This month, we are introducing a new implementation for Snowflake connector, currently available in preview. To access this feature, in Power BI Desktop, navigate to Options and settings (under the File Menu) > Options > Preview features, select the checkbox to enable the ‘Use new Snowflake connector implementation’ option. Once the option is on, all the newly created connections will automatically use the new connector implementation. Your existing connections remain unchanged. You can also test the new feature by editing the queries. Learn more about the Snowflake connector from this documentation article. If you're using On-prem Data Gateway to refresh your semantic model, make sure you have the latest version to use this feature. We highly value your feedback on this feature and encourage you to share the feedback with us. Visualizations Drill Down Scatter PRO by ZoomCharts: The All-in-One Scatter Visual The latest ZoomCharts visual, Drill Down Scatter PRO, is now available on AppSource! Just like all ZoomCharts visuals, Scatter PRO combines powerful data visualization features with an intuitive and user-friendly user experience. It is designed for fully interactive Power BI reports that deliver quick insights and foster a decision-centric culture. Scatter PRO makes data exploration seamless and enjoyable with user interactions like panning, zoom-in, and rectangular or lasso selection. You can also create a multi-level hierarchy, which will allow users to drill down by simply selecting on a data point marker. You can learn more in our blog post, but here are the main features of Scatter PRO: Drill Down: Create a multi-level category hierarchy and drill down with just a select. Customization: Configure marker colors, shapes, outlines, labels, threshold lines/areas, X & Y axes, and more. Data-Driven Formatting: Apply marker colors, shapes, and even images directly from data. Area Shading: Highlight areas that need attention with up to 8 shapes at custom coordinates. Dynamic Regression Line: Show a linear or polynomial regression line. It will automatically recalculate upon any changes in the chart. Get on AppSource Lollipop Chart by Powerviz The Powerviz Lollipop chart is a variation of bar chart that uses lines and dots to represent data points. It is perfect for highlighting specific trends to help stakeholders make informed decisions. Key Features: Chart Options: Switch easily between vertical/horizontal chart. Marker Style: Choose from Shapes, Charts, Icons, Images, or Upload custom image. Small Multiples: Split your visual into multiple smaller visuals. Error Bars: Add error bars to show data variability, improving analysis accuracy. Race Chart: Enhance the chart by adding animations to show data changes over time. Cut/Clip Axis: Trim/Adjust the axis to accommodate the outliers. Dynamic Deviation: Analyze the deviation between two bars in a glance. Preview Slider: Easily explore various sections of a chart in large datasets using a slider. Conditional Formatting: Easily find outliers by using rules for measures or categories based on rules. Other features included are Templates, Import/Export Themes, Data Colors, Ranking and more. Business Use Cases: Sales Analysis, Financial Reporting, Market Research. Try Lollipop Chart visual for FREE from AppSource Check out all features of the visual: Demo_file Step-by-step instructions: Documentation YouTube Video: Video_Link Learn more about visuals: https://powerviz.ai/ Follow Powerviz: https://lnkd.in/gN_9Sa6U Other Now in Power BI Desktop - OneLake catalog The OneLake catalog is now part of the Power BI Desktop experience, providing a consistent and seamless way to discover and explore data. This update ensures alignment with the broader Fabric ecosystem, offering users a unified and familiar experience across tools. Platform Folder support in Git Timeline update - Folder support is planned to roll out to all customers by mid-April. Thank you for your patience! This update ensures that the folder structure in your Fabric workspace is seamlessly mirrored in your connected Git branch, providing an organized and consistent experience across both platforms. New features: Folder Structure Mirroring: The entire folder hierarchy in Fabric is reflected in Git and vice versa, enabling a more intuitive and organized collaboration process. Nested Folders Are Synced: Fabric items located within nested folders will now be included in the sync, and their folder structure will be preserved. Item Updates as Commits: Changes to an item's folder (e.g., moving an item to another folder or reorganizing folders) will now appear as updates or commits in Fabric. Subfolder support is enabled by default as soon as the feature is live. This means any folder differences between Fabric and Git will automatically show up as updates or commits. Handling Folder Changes Safely If changes to the connected branch cannot be made directly due to branch policy or permissions, we recommend using the ‘Checkout Branch’ option. Guidelines for managing this: Checkout a New Branch: Use the checkout branch feature to create a branch with the updated state of your Fabric workspace. Commit Folder Changes: Any workspace folder changes can then be committed to this new branch. Merge Changes: Use your regular pull request (PR) and merge processes to integrate these updates back into the original branch. OneLake OneLake Catalog - Semantic model table & column description We are expanding the details view of Semantic Models, to also include table and column descriptions which were set in the data model editor in the service or in Power BI Desktop. The goal is to provide consumers with multiple trust signals regarding an artifact, thereby enabling them to make swift and well-informed decisions. This improvement provides additional name and type details for tables and columns in semantic models, helping data consumers identify relevant tables more efficiently and encouraging data producers to document organizational knowledge. We are planning to expand this ability to other data items in the future. Filtering workspaces in OneLake Catalog You now have a dedicated filter for workspace names, which will allow you to quickly locate the required workspace. This makes it easier for users with access to multiple workspaces, and makes finding the relevant workspace difficult if it isn't featured at the top of the list. Data Engineering Python notebook (Preview) Announcing the preview of the highly anticipated Python Notebook! This new feature is designed to enhance the experience of BI developers and data scientists working with smaller datasets using Python as their primary language. Key features: Native Python support: Enjoy the full power of Python with native features and libraries right out of the box, like ipywidget, magic commands. Version flexibility: Easily switch between different Python versions (initially supporting Python 3.11 and 3.10). Optimized resource utilization: Benefit from better resource utilization with a smaller 2vCore/16G memory compute, real-time resource utilization monitor is available. Lakehouse & resources natively available: Leveraging the Fabric Lakehouse capabilities seamlessly, with built-in Resource folder to store your modules, libs and files. Mix programming with T-SQL: You can interact with data warehouses and SQL endpoints on Python notebook, with the built-in notebookutils connector. Superior Python intellisense: Powerful Pylance are natively integrated to provide smoother coding experience. Popular libraries are pre-installed: Using duckdb, polars and other popular 3 rd party libraries on Python notebook conveniently. Fabric utilities like Semantic Link and NotebookUtils are also natively supported. Seamless integration with Fabric ecosystem: All the advantages of Fabric notebook like sharing, CI/CD, schedule run, data pipeline integration, OrgAPP integration, are available for Python experience. Getting Started: Access the Notebook: You can access the Python Notebook from the Notebook language dropdown menu. Comprehensive Guide: A detailed guide is available to help you get started. Please refer to the public document to find more details. Your feedback is crucial in shaping the future of our product. We look forward to your active participation and valuable insights. Thank you for being a part of this exciting journey with us! Notebook live versioning Announcing the launch of the Fabric notebook version history feature. This new feature is designed to significantly improve your experience in developing and managing notebooks by providing robust built-in version control capabilities. Highlights: Automatic checkpoints: These checkpoints are created automatically every 5 minutes, ensuring that your work is consistently saved and versioned. Manual checkpoints: You can also manually create checkpoints to record your development milestones, providing flexibility in how you manage your notebook versions. Track history of changes: Users can now view a list of previous notebook versions, see what changes were made, contributed by whom, and when. Compare different versions: Easily compare different versions of a notebook through a diff view to understanding the evolution of your work. Restore previous versions: If you make a mistake or want to explore a different approach, you can restore previous versions of your notebook or save a new copy of it. NotebookUtils session management utilities Introducing a new utility in NotebookUtils- session management utilities, including a list of APIs that can help you manage your session and interpreter status. notebookutils.session.stop(): Support stopping the interactive session via code, it's available for Scala and PySpark. notebookutils.session.restartPython(): Support restarting the Python interpreter in PySpark notebook. For more details, please refer to the documentation. Native Execution Engine on Runtime 1.3: simplified enablement and transition from Runtime 1.2 Introducing a new update that simplifies enabling the Native Execution Engine. Now, activating it is as easy as toggling a switch! You’ll find the new toggle button in the Acceleration tab within your environment settings. If you were previously using the Native Execution Engine, please navigate to the Acceleration tab and re-enable it using the new toggle. This updated UI control now takes precedence over any previous configurations in Spark settings, meaning prior setups will remain inactive until re-enabled with the toggle. Additionally, the Native Execution Engine now fully supports our latest runtime version, Runtime 1.3 (Apache Spark 3.5, Delta Lake 3.2). As a result, support for Native Execution Engine on Runtime 1.2 is ending. We recommend upgrading to Runtime 1.3 to maintain support, as native acceleration will soon be unavailable on Runtime 1.2. Legacy timestamp support in Native Execution Engine on Runtime 1.3 The latest Native Execution Engine on Fabric Runtime 1.3 introduces legacy timestamp handling, ensuring compatibility across Spark versions. This feature addresses timestamp issues caused by Spark 3.0’s shift to the Java 8 date/time API (Proleptic Gregorian calendar) from the previous hybrid Julian-Gregorian calendar. With the configuration spark.gluten.legacy.timestamp.rebase.enabled, the Native Execution Engine auto-adjusts for calendar differences in Parquet files and Delta Tables, handling dates seamlessly across Spark versions. Dates post-1970 are unaffected, ensuring consistency without extra steps. To activate this feature, add the following to your Spark session: SET spark.gluten.legacy.timestamp.rebase.enabled = true; Notebook and Spark Job definition execution with service principal The service principal support for Fabric API was announced back in September. Today, we unblock another key scenario, enabling run the Notebook/Spark Job Definition execution under SP (service principal) for Data Engineering experience. Using the Fabric Job Scheduler API, users can trigger the execution of either a Notebook or Spark Job Definition (SJD) and monitor its execution status. By utilizing the same API with a service principal's access token, the Spark Job associated with the notebook/SJD will run within the security context of that service principal. This update enhances SP support for the Data Engineering experience, expanding its capabilities beyond current CRUD operations to include comprehensive job execution coverage. To make sure the SP does have the privilege to run the job, you need to add that SP as Admin/Contributor/Member into the workspace which hosts the Notebook/SJD. If the notebook/SJD has some code related with Data Science scenario such as Model/Experiment, the SP triggered execution could fail, this is something we are working on to unblock them later. Lineage Enhancement to Spark Notebook A new improvement to the lineage related to Spark Notebooks has been introduced to enhance data exploration effectiveness. You can now view all Lakehouses connected to your notebook, including pinned and additional Lakehouses. This update helps you: Perform Impact Analysis: Easily assess how changes affect data workflows by identifying which Lakehouses are being used. Document Data Pathways: Streamline collaboration and audits with clear visibility into data relationships. Experience smarter data management today with this lineage enhancement to your Spark Notebooks! Data Warehouse COPY INTO column count check The COPY statement offers flexible, high-throughput data ingestion from an external Azure storage account into Fabric Data Warehouse tables. When there is a column count mismatch between rows in source files and the target table, COPY INTO has the following behavior: If a row within the source files has less columns than the target table, COPY INTO inserts columns with missing values as NULL. If there’s no corresponding value for a non-nullable column in the source data, then COPY INTO fails. If a row within the source files has more columns than the target table, any excess columns from source files are ignored in the target table. We’re introducing a new option for COPY INTO that allows you to control the behavior of your data ingestion jobs by checking if the count of columns in the source data matches the count of columns on your target table. The following syntax should be used for the column count check option: COPY INTO FactSale FROM '<external_location>' WITH ( FILE_TYPE = 'CSV', [ , MATCH_COLUMN_COUNT = { 'ON' | 'OFF’ } ] ) MATCH_COLUMN_COUNT checks the column count on each row of each source file for a match against the target table specified in the COPY INTO statement. This option is available only for CSV file type sources now, with support for Parquet coming soon. The default behavior of COPY INTO remains unchanged and is equivalent to using MATCH_COLUMN_COUNT = ‘OFF’. Learn more about COPY INTO and this new option, refer to the documentation. Enhancing COPY INTO operations with Granular Permissions in Data Warehouse One of the challenges our customers shared is that executing COPY INTO command requires users to have at least the Contributor role at the workspace level, granting broad permissions that may exceed what is necessary for specific tasks. We are excited to announce that now a user with minimum ‘read’ permissions on the control plane will be allowed to execute write operations at the Data Warehouse level. The benefits of this change include reducing the need for broad workspace roles, and it also works seamlessly even when storage account is protected behind a firewall. Learn more about COPY INTO in our documentation and to know more about this new option, check out this blog post COPY INTO support for secure storage with granular permissions. Introducing default schema changes in Data Warehouse We are happy to announce the ability to change the default schema in Fabric Data Warehouse. With this improvement, which has been highly requested by our customers, we strive to make database management and enhanced security more straightforward. This is done using the ALTER USER statement, ensuring that every user has a predefined schema context when they connect to the database. ALTER USER [username] WITH DEFAULT_SCHEMA = [schema_name]; By allowing administrators to assign default schemas to users, we ensure that users operate within their designated schemas, reducing the risk of unauthorized access and simplifying permissions management. For more information, check our documentation: What is data warehousing in Microsoft Fabric? ALTER USER (Transact-SQL) Create a Database Schema Enhanced performance metrics in Query Insights New features have been introduced to provide deeper insights into query performance. With the introduction of Data Scanned Analysis, you can now determine if large data scans are contributing to slower query execution. This feature allows you to compare similar queries, pinpoint fluctuations caused by changes in data scanned, and even identify when cache was utilized. Additionally, we've introduced allocated CPU time as a key performance metric. This enables you to understand the resources consumed by your queries and workloads. High CPU time often correlates with higher costs, making it easier to identify and address resource-intensive queries. These enhancements empower you to optimize performance and manage costs effectively. These columns are available in queryinsights.exec_requests_history: Column name Data type Description allocated_cpu_time_ms Bigint Shows the total time of CPU(s) that was allocated for a query’s execution. data_scanned_remote_storage_mb Bigint Shows how much data was scanned/read from remote storage (One Lake). data_scanned_memory_mb Bigint Shows how much data was scanned from local memory. Data scanned from disk and memory together indicates how much data was read from cache. data_scanned_disk_mb Bigint Shows how much data was scanned/read from local disk. Data scanned from disk and memory together indicates how much data was read from cache. For more information, check out our documentation: Query insights queryinsights.exec_requests_history (Transact-SQL) Previewing estimated Query Plan available via SHOWPLAN_XML The Preview for SHOWPLAN_XML in Microsoft Fabric Data Warehouse is now available. This capability allows users to generate and view the estimated query execution plan in XML format, a tool for analyzing and optimizing SQL queries. Whether you're troubleshooting performance bottlenecks or refining query strategies during development, SHOWPLAN_XML offers a granular, detailed view of how the database engine plans to execute your queries. By providing insights into operations like joins, data movements, etc. it helps pinpoint inefficiencies and identify opportunities to enhance performance. How can you use SHOWPLAN_XML? Enabling SHOWPLAN_XML - To enable SHOWPLAN_XML, execute the following SQL command: SET SHOWPLAN_XML ON;This command instructs Fabric DW to return execution plans in XML format for all subsequent queries. Running Queries - After enabling SHOWPLAN_XML, run the queries that you wish to analyze.The execution plan for this query will be returned in XML format. Capturing the Output - Capture the SHOWPLAN_XML output by saving the result set to a file or copying it to an XML viewer. Ensure that the entire XML content is preserved for accurate analysis. When running SHOWPLAN_XML in Fabric UI, copy results and save them as a .sqlplan file. Open this file in SSMS to view the graphical plan. If you run in SSMS, use the SET SHOWPLAN_XML syntax as explained above. You can also use the plan Display Estimated Plan button to see the graph. Turn OFF SHOWPLAN_XML - Run SET SHOWPLAN_XML OFF to receive results instead of the execution plan when you run queries. For more details, check out our documentation: SET SHOWPLAN_XML (Transact-SQL) Query Hints in Fabric Data Warehouse Along with SHOWPLAN_XML, we are announcing support for some query hints. Query hints in Fabric SQL are optional keywords that can be added to SQL statements to provide additional information or instructions to the query optimizer. These hints can improve the performance, scalability, or consistency of queries by overriding the default behavior of the query optimizer. To use a query hint, the OPTION clause is added at the end of the query, followed by the name of the query hint and its optional parameters in parentheses. For instance, if you want to instruct the query optimizer to use a hash-based algorithm for the GROUP BY operation, you can use the HASH GROUP query hint. SELECT band_id, SUM(ticket_cost) FROM gigs GROUP BY band_id OPTION (HASH GROUP) Fabric SQL supports a variety of query hints, including HASH GROUP, ORDER GROUP, MERGE UNION, HASH UNION, CONCAT UNION, FORCE ORDER, LOOP JOIN, HASH JOIN and REPLICATE. Each of these hints serves a specific purpose, such as improving the performance of GROUP BY operations, UNION operations, or join operations. However, query hints should be used with caution and tested thoroughly, as they can have negative effects on the performance, scalability, or consistency of queries if used incorrectly or unnecessarily. It is essential to monitor and evaluate the impact of query hints on queries and adjust them as needed. For more details, refer to our documentation: Join hints (Transact-SQL) Query hints (Transact-SQL) Simplifying search & introducing Filter in Object Explorer The search and filter features in Fabric Data Warehouse empower users to efficiently navigate and manage their data. The new search feature is designed for ease of discovery and intuitive use, allowing users to locate objects in the object explorer by entering keywords. The search function quickly highlights matching objects and highlights the results within the object explorer for the user. The filtering feature in Object Explorer is an essential tool for managing large data warehouses and simplifying navigation within your warehouse environment. When dealing with numerous objects, such as schemas, tables, or stored procedures, finding specific items can be challenging. The filtering capability allows you to streamline this process effectively. The filtering options allow for precise object selection based on various criteria such as object type, created date, last updated, enabling you to focus on the most relevant information for your exploration in object explorer. By leveraging this combination, users can significantly reduce the time spent searching or filtering data, allowing for more focus on troubleshooting, generating scripts for development and documenting objects in Object Explorer. Open from SSMS & VS Code Developers now can easily access their Fabric Warehouse through their preferred client tools. With a renewed focus on integrating with widely used developer tools, this enhancement prioritizes flexibility and convenience, enabling seamless connections with SQL Server Management Studio and Visual Studio Code. This means you can dive right into your data analysis and management without any hassle, using the tools you know and love. Developers have the option to open Fabric Warehouse in SQL Server Management Studio (SSMS) or Visual Studio Code, either from a workspace or within the warehouse itself. You can open the warehouse in VS Code or by downloading VS Code. Note that Visual Studio code will install ms-sql for you and pre-populate Server and Database name in the connection to get started. You can also open or download SSMS to begin using your preferred tool. Git Status Bar to Fabric Warehouse Artifact The Git artifact status bar offers a comparable experience to the status bar in the workspace. When accessing the DW item page, you can view the details of the connection between the workspace and the Git repository, such as: The name of the branch to which the workspace is connected The time of the last sync event between the workspace and the repository A hyperlink to the most recent commit on the branch. The Git status bar is useful for following scenarios It offers a user-friendly interface for developers, like that of Visual Studio Code and other applications that display the status of the connection to the remote repository at the artifact level. More features, such as the ability to commit directly from the artifact page, will be added soon. Tooltip support for built-in functions Fabric Web editor provides robust tooltip support for built-in functions, enhancing the development experience by offering quick access to function details. When you hover over a built-in function in your query, web editor displays an interactive tooltip. This tooltip includes the function’s name, possible parameters, and a brief description. For example, if you hover over the COUNT() function, the tooltip will show its syntax and a short description of what the function does. This feature is particularly useful for quickly referencing function parameters and understanding their usage without leaving the query window. It helps streamline the coding process and reduces the need to manually look up function details. Stay updated with IntelliSense Our goal is to facilitate Fabric developers in writing queries by enabling new T-SQL statements in Fabric Warehouse. Recently, we have integrated IntelliSense support for the following newly released features. FOR JSON - Announcing improved JSON support in Fabric DW COPY INTO – Column count check: COPY INTO (Transact-SQL) - Azure Synapse Analytics and Microsoft Fabric JSON aggregates (Preview) As part of our ongoing enhancements to JSON functionalities in Fabric DW, we are excited to announce the preview of two new JSON aggregate functions: JSON_OBJECTAGG JSON_ARRAYAGG These aggregate functions simplify the process of concatenating columns within a GROUP BY operation and formatting them as JSON text. Previously, achieving this required the use of the generic STRING_AGG() aggregate combined with manually concatenated and escaped column values to produce a valid JSON string. Now, with JSON_OBJECTAGG and JSON_ARRAYAGG, this process is streamlined and more efficient. These functions are already in public preview in Azure SQL Database and Azure SQL Managed Instance, and Fabric DW is now joining this preview. They will become generally available across all SQL flavors simultaneously. Spatial analytic functions Spatial analytics functions are now fully supported in Fabric DW and SQL endpoints. Spatial functions enable you to perform complex calculations on the geographical and geometrical shapes, such as determining the distance between points, checking whether a point is within a polygon, or whether the shapes intersect. Previously, these functions were not fully supported. Comprehensive support commenced in December 2024. Fabric DW supports both geography objects and functions for simple 2D geometries, as well as geography objects for more realistic shapes represented on the Earth's surface. This includes all spatial reference systems available in SQL Server and Azure SQL databases. While Fabric DW does not support storing spatial types directly, you can represent your spatial objects as float columns representing the (latitude, longitude) pairs or store complex shapes in VARBINARY columns in Well-Known Binary (WKB) format. You can then use spatial functions to convert them to spatial objects and apply spatial operations. For example, the following query finds the number of trips starting near the Empire State Building in New York (40.748817, -73.985428): Be aware that the spatial functions are among the most complex calculations you can perform in your DW, which might impact performance. To improve the query performance, ensure you are physically storing a geoindex column that can approximately determine the location of an object or shape (e.g., using a bounding box, quadkey, H3 index, geohash, or something similar) and prefilter data based on this index column instead of applying a direct spatial filter. Since spatial filters and joins are the most resource-consuming operations, they might impact performance if rely only on them without additional indexing columns. Once you reduce your data set using geo index columns, you can perform complex spatial analytics using geography and geometry methods. SQL analytics endpoint performance improvement We released an update to the SQL analytics endpoint that improves query performance and data freshness. Previously you might have encountered slow SELECT statements and stale data in your tables. With this update, metadata changes are synced more efficiently to the SQL analytics endpoint, resulting in faster SELECT query execution and data updates. This improvement ensures a more responsive and reliable experience for our users. We'll continue to enhance the SQL analytics endpoint based on your feedback, so make sure to comment or vote on Ideas. Databases Tenant Level Private Link (Preview) We are excited to announce the preview of Tenant Level Private Link for SQL database in Fabric! This new feature enhances the security and privacy of your data by allowing you to connect to your SQL database through a private endpoint within your virtual network. With Private Link, you can now ensure that your data traffic remains within the Microsoft network, reducing exposure to the public internet and minimizing potential security risks. This integration simplifies network architecture and provides seamless and secure connection experience for your SQL database in Fabric. To enable Private Link in Fabric, start by creating a private endpoint within your virtual network (VNet) to securely connect to the Fabric service using a private IP address as outlined here Set up and use private links for secure access to Fabric Next, enable the Private Link toggle in the Fabric Admin Portal for your tenant to allow VNet requests to access Fabric resources. Additionally, you can choose to completely disallow any connections other than via Private Link by enabling the Block Public Internet Access toggle. For more information check out SQL database Overview (Preview). Copilot for SQL database in Fabric Region Availability Copilot for SQL database is now available in all supported regions listed in Fabric region availability! Today, we are offering three key features: Inline Code Completion for faster, smarter query writing. ‘Explain the Query’ & ‘Fix Query Error’ Quick Actions to simplify complex tasks. Sidecar Q&A Chat for answers and deeper understanding. Before your business can start using Copilot capabilities in Microsoft Fabric, please make sure to enable Copilot in the tenant settings. For more information, refer to these resources: Overview of Copilot for SQL database (Preview) Introducing Copilot for SQL database in Microsoft Fabric Introducing Copilot for SQL database in Microsoft Fabric | Data Exposed Real-time Intelligence Override late arrival tolerance in Activator Late arrival tolerance refers to how long Activator waits for an event to arrive and be acknowledged and processed. This setting ensures that late events and events that arrive out of order have an opportunity to be included in the rule evaluation. The consideration is to tradeoff on getting more ‘accurate’ rule evaluations by waiting longer for late data points to arrive or run your rule on potentially incomplete data, so the rule is activated sooner. The default setting is 2 min, but now you can set the late arrival tolerance to a longer or shorter period. Please note that this setting will not be shown for rules that are built on Power BI or KQL data. Create new Activator items As of November 2024, Real-time Intelligence and Activator are now Generally Available (GA)! You may need to work with your admin to ensure that you have all the capabilities available to you in Activator GA. If your tenant is using the preview version of Data Activator but does not have Fabric enabled, you will no longer be able to create new Activator items. To keep using Activator and create new Activator items, enable Fabric for your tenant. To enable Fabric, go to the admin portal and make sure that users are allowed to create Fabric items. Please note that if you have delegated settings to other admins, you should also allow capacity admins to enable/disable. RTI ALM & APIs GA Application Lifecycle Management (ALM) and Fabric REST APIs are now available for all RTI items: Eventstream, Eventhouse, KQL Database, Realtime dashboard, Query set and Data Activator. ALM includes both deployment pipelines and Git integration, both allow you to manage change within your workspaces. This enables multiple scenarios, from simply ‘recording’ changes in Git to have an audit trail, to deploying your development workspace, with all its dependencies to a staging and production workspace, to introducing changes via feature branch connected to feature workspace. REST APIs enables more control over changes by allowing you to programmatically create / read / update / delete (CRUD) your artifacts. Data Factory Mirroring Mirroring now supports replicating source schemas Mirroring in Fabric now supports replicating source schemas. When data is mirrored from various types of sources, your source schema hierarchy is preserved in the mirrored database. This ensures that your data remains consistently organized across different services, allowing you to consume it using the same logic in SQL analytics endpoint, Spark Notebooks, semantic models, and other references to the data. The existing mirrored databases remain unchanged to maintain backward compatibility and avoid affecting downstream workload. If you want to reorganize your tables with schemas, please recreate the mirrored database. Please refer to schema support with Mirroring in Microsoft Fabric. Delta column mapping support for Mirroring is now available Mirroring in Fabric now supports Delta column mapping. Column mapping is a feature of Delta tables that allows users to include spaces and special characters such as ',;{}()\n\t=.' in the table's column names. With this new capability in mirroring, columns containing spaces or special characters in names can now be replicated from your source databases to the mirrored databases. For tables with special characters in column names that are already under replication, you can update the mirrored database settings by removing and re-adding them to include those columns. To learn more, refer to Delta column mapping support with Mirroring in Microsoft Fabric. Mirroring now supports CI/CD (Preview) Mirroring in Fabric now supports CI/CD capabilities, enhancing the efficiency and reliability of your development workflows. Users can integrate Git for source control and utilize ALM Deployment Pipelines, streamlining the deployment process and ensuring seamless updates to mirrored databases. To learn more about these new capabilities, refer to CI/CD for mirrored databases in Fabric (Preview). Integrating SAP data into Open Mirroring dab (dab – We are your company for SAP data analytics) is the first partner to announce support for Open Mirroring from our SAP ecosystem. With over 20 years of experience in SAP analytics, dab offers a variety of analytic solutions covering multiple lines of business including accounting and procurement. Dab Nexus now integrates with Open Mirroring to synchronize data from various SAP sources including SAP S/4HANA (on-premises and Private Cloud Edition), SAP ECC, CRM, SRM, SCM and EWM. For more information on Open Mirroring with dab Nexus, refer to: Microsoft Fabric Open Mirroring: Efficient and innovative. To learn more about Open Mirroring, please refer to the documentation. Copy Job Simplify data ingestion with Copy Job: more connectors and better usability Copy Job simplifies data ingestion, providing a seamless experience from any source to any destination. Whether you need batch or incremental copying, Copy Job provides the flexibility to meet your data needs while keeping things simple and intuitive. Since the Public Preview launch at FabCon Europe in late September, we’ve quickly enhanced Copy Job with new features. We’re excited to announce that Copy Job now supports more connectors, including Snowflake and Azure SQL Managed Instance, for easier integration with your data sources. More connectors are coming soon! We’re committed to making Copy Job as simple and intuitive as possible, and your feedback is key to achieving that goal. You can now easily configure the update method and schedule before creating a copy job, giving you greater control and flexibility right from the start. Check out the details in What is Copy job (preview) in Data Factory. Dataflow Gen2 CI/CD support for Dataflows in Fabric We are delighted to share that CI/CD and GIT integration support for Dataflow Gen2 is now available in preview! You can opt into enabling these capabilities when creating a new Dataflow Gen2. With this new set of features, you will be able to seamlessly integrate your Dataflow Gen2 artifacts with your existing CI/CD pipelines and version control of your workspace in Fabric. This integration allows for better collaboration, versioning, and automation of your deployment process across dev, test, and production environments. Learn more about these new capabilities: Dataflow Gen2 CI/CD and GIT source control integration are now in preview! Single-line ribbon in Power Query editor The default experience for Dataflows Gen2 in Fabric now uses the ribbon in its single line mode. This brings consistency against other experiences that you will find within Microsoft Fabric. The full ribbon mode will remain available. You can access the complete ribbon experience by selecting the expand button. Conclusion We hope that you enjoy the update! Be sure to join the conversation in the Fabric Community As always, keep voting on Ideas to help us determine what to build next. We are looking forward to hearing from you!
