Fabric July 2026 Feature Summary
Welcome to the July 2026 Fabric update! This month brings new capabilities across the Fabric experience, from improved deployment and governance experiences to expanded Spark, Eventstream, and Real-Time Intelligence functionality. Whether you're building data pipelines, managing analytics workloads, or monitoring real-time operations, these updates help you work more efficiently and get more value from your data. _______________________________ Events and Announcements Get Fabric certified for FREE This is your chance to take the DP-600 (Fabric Analytics Engineer) or DP-700 (Fabric Data Engineer) certification exams for free. As part of Data Days, we have over 100 live sessions, more than 5 contests and challenges, and dozens of study groups and learning opportunities. And free Fabric exam vouchers. Available now through August 10, 2026. Request your voucher. Join us for FABCON and SQLCON in Barcelona, September 28 – October 1, 2026 Explore what’s possible with Microsoft Fabric and get up to speed on the latest in SQL, analytics, and AI. From 130 sessions and 4 keynotes to workshops, the expo, community spaces, and the Power BI DataViz World Championships, this is where the data community comes together. Learn directly from Microsoft and community experts shaping the future of Fabric and SQL. Register now and save €200 with code FABCMTY200. Fabric Platform Fabric-CICD tool v1.2.0 – new bulk publish mode (Preview) The June release of the fabric-cicd Python library, v1.2.0 introduces bulk publish mode. It lets fabric-cicd publish multiple items in a single API call using the Fabric bulk import API instead of publishing each item individually through a separate API call. This can make deployments more efficient. Why this matters Because dependencies are managed by the API during publication, bpublishinglish reduces the rigidity of item-type-based staging and better supports cross-item dependencies when logical ID references are used. It can reduce the number of item-specific parameter values you need to configure. For unsupported scenarios, fabric-cicd automatically falls back to the standard publishing flow, so you can try bulk publishing without manually switching deployment paths. To learn more, refer to fabric-cicd bulk option documentation and the v1.2.0 release notes. Change Git branch with at least the contributor role (Preview) Fabric Git integration now let’s any workspace member with at least the Contributor role switch the workspace's connected Git branch. Previously, this action required the workspace Admin role, which forced developers to either be over-permissioned or wait on an admin every time they moved between branches. Why this matters Developers no longer need Admin rights to switch branches. Removes a key blocker in the branch-out to existing workspace flow. Keeps developers as Contributors, aligned with least privilege governance. Fewer hand-offs to admins mean faster time to code. You can find this new capability under the Git integration settings. To learn more about Fabric Git integration new setting, refer to the Allow Contributors and Members to switch branches documentation. We refreshed the Fabric CI/CD documentation this month to make it easier to get started and to follow best practices: New CI/CD intro page — a reworked Introduction to CI/CD in Microsoft Fabric that walks through the platform layer by layer with a new enterprise reference architecture. New best practices guide — Understand the best practices for Fabric CI/CD brings together practical guidance for structuring workspaces, branching, and promoting content safely across environments. Auto-bind for Git integration — new guidance on cross-workspace dependency binding, covering how item dependencies automatically rebind when you branch out or update from Git, and which dependency types are supported. Actionable recommended actions for data owners in the OneLake catalog The Govern tab in the OneLake catalog gives data owners a health view of their data estate, along with recommended actions to improve it, such as increasing sensitivity label coverage, removing items that are no longer in use, handle failed refresh, and more. Until now, these recommendations have told data owners what to improve, but not which items were affected, leaving them to track the relevant entities down manually before they could act. That gap turned a clear recommendation into a manual investigation, and the action often stalled before it started. This release closes that gap. When you open a recommended action, you now see the specific items behind the recommendation, with everything you need to act on them in one place, and a direct path to open item details and resolve the issue at the source. Selecting a recommended action now opens a dedicated view that includes: A breakdown of the affected items — a visual summary of how many of your items are impacted. Why it matters — a short explanation of the governance impact. How to fix it — clear, step-by-step instructions for resolving the recommendation. Newly added A complete list of affected items — displayed in a table with key details, including Name, Last Refreshed, Owner, Location, Endorsement, and Sensitivity, making it easy to review and prioritize actions. Filter and search — narrow the list by keyword or filters to focus on the items you want to handle first. Open item details — jump straight to any affected item to make the change, instead of searching for it across your workspaces. The same enriched experience applies across the recommended actions in OneLake catalog Govern, helping you act on each one without leaving the catalog: Increase Sensitivity label coverage — find and label items that create potential security risks while unlabeled. Remove unused items — review items that weren't accessed or refreshed recently to keep your estate organized and reduce costs. Investigate items that failed to refresh — see which items failed to refresh so your data stays current and reliable. Add descriptions to your endorsed items — surface endorsed items without a description so people can better understand and use them. Apply relevant tags to your items — make items more discoverable by tagging the ones that are missing tags. Why it matters Governance only improves when recommendations turn into action. By bringing the affected items and a direct way to reach them into every recommended action, OneLake catalog Govern removes the guesswork from following through. Data owners can move from understanding a recommendation to resolving it in just a few steps, keeping their data estate secure, organized, discoverable, and trustworthy with far less effort. Learn more about the OneLake catalog Govern tab recommended actions. Data Engineering Microsoft Fabric Runtime 2.0 (Preview) Based on feedback received directly from our customers and partners, we have upgraded Fabric Runtime 2.0 to the latest and compatible stack: Apache Spark 4.1 Delta Lake 4.2 Python 3.13 These upgrades bring access to the newest enhancements, performance improvements, and ecosystem innovations, while also providing a longer support window for enterprise customers to build large-scale analytics and AI workloads on Fabric. With Runtime 2.0, customers can take advantage of: Latest Spark and Delta Lake capabilities. Improved compatibility across the modern data ecosystem. Better developer productivity and runtime performance. Future-ready platform investments aligned with long-term supportability. Learn more about Fabric Runtime 2.0. Fabric Runtime Release Channels (Preview) Fabric Runtime Release Channels provide a structured and transparent way for customers to test upcoming runtime changes before they become the default. This feature helps organizations validate their production workloads early with these new changes in early access, avoid unexpected disruptions, and gain better control over Spark runtime upgrades. Instead of receiving silent updates that might break your production workloads, you can opt in to an early access release channel, test your workloads in a development or staging environment, and confirm compatibility before the update becomes default. How release channels work Each Spark runtime has at least two public release channels: Default channel — This production-grade channel runs the default version of the runtime. All users automatically use this channel unless they opt in to early access. Early access channel — This production-grade channel includes upcoming updates and library changes that are scheduled to become the next default channel. You can opt in to test your workloads against upcoming changes. Once the designated validation window ends, the early access release channel automatically gets promoted to become the new default, and a fresh early access channel is introduced with another set of new changes — continuing the cycle. This model gives you a predictable testing window before changes become default for everyone. Why release channels matter Spark runtime updates can include library upgrades, security patches, dependency changes, or even operating system upgrades. While all updates pass internal quality checks before release, those checks can't capture all customer-specific variations and use cases. Early access channels let you identify potential issues early and work with Microsoft by creating a support ticket to address them before updates affect your production environment. Benefit Description ✔ Predictable updates Customers know exactly when a new runtime becomes available and have time to validate against it. ✔ Reduced risk Testing workloads on early access ensures compatibility before changes reach production. ✔ Better visibility Customers can easily tell which runtime version they're running, reference release notes, and verify upgrade timing. ✔ Improved quality and security You receive well-tested builds with security patches applied faster, giving you confidence in runtime stability. Learn more about Fabric Runtime Release Channels: Fabric Runtime Release Channels. Spark Diagnostic Emitter: Spark 4.1 Runtime Support and New Log Ingestion API The Fabric Apache Spark Diagnostic Emitter is now supported on the Fabric Runtime with Apache Spark 4.1. Customers can collect driver logs, executor logs, Spark event logs, and metrics from workloads running on the latest runtime and route them to Azure Event Hubs, Azure Blob Storage, or Azure Log Analytics — using the same spark.synapse.diagnostic.emitter.* configuration model, so existing emitter setups carry forward as customers upgrade. The emitter also now supports the Azure Monitor Log Ingestion API for sending diagnostics to Log Analytics, available on both Spark 3.5 and Spark 4.1 runtimes. The new AzureLogIngestion emitter type replaces the legacy HTTP Data Collector API path, providing a structured ingestion model with DCR/DCE-based authentication, schema definition, and routing into custom Log Analytics tables. Customers currently on the legacy AzureLogAnalytics type are encouraged to migrate — migration involves creating Data Collection Rule and Data Collection Endpoint resources and updating the Spark properties in the Fabric Environment. To learn more, refer to the Collect logs and metrics with Azure Log Analytics for migration guidance and Spark Diagnostic Emitter documentation. The Fabric Spark Operations Skill — AI-Assisted Spark Diagnostics, Open Source (Preview) The Fabric Spark Operations Skill (spark-operations-cli) is now available in the open-source Skills for Fabric library on GitHub (microsoft/skills-for-fabric). The skill brings AI-assisted, read-only diagnostics to Spark workloads in Fabric — troubleshoot failed notebooks and Spark jobs, stuck Livy sessions, and performance bottlenecks (OOM, shuffle, skew) in plain English from GitHub Copilot CLI, Claude Code, VS Code, Cursor, or other compatible AI tools. It returns severity-ranked findings with root cause analysis and fix recommendations, and includes an automated diagnostic workflow spanning job triage, log mining, Spark Advisor findings, and mitigations. Setup takes minutes: install the skill, run az login, and ask, "Why did my notebook fail last night?" Get started with Skills for Fabric in GitHub. Faster Python UDFs, Scala UDFs, and complex data types in the native execution engine (Generally Available) The native execution engine in Microsoft Fabric, which is generally available, now accelerates Python user-defined functions (UDFs), Scala UDFs, and complex data types such as arrays, maps, and structs. You get faster Spark processing for expressive code without changing your existing notebooks or jobs. Python UDFs have historically carried serialization overhead as data moves between the JVM and Python worker processes. The native execution engine optimizes that data path and keeps vectorized processing intact, so scalar and Pandas (vectorized) UDFs run faster automatically. Scala UDFs and queries that work with nested, complex data types benefit from the same native acceleration. Why this matters Faster UDF execution with no code changes. Vectorized Pandas UDFs see the largest gains. Complex types (arrays, maps, structs) run natively. Existing notebooks and jobs benefit automatically. To use Efficient Scaledown, enable the native execution engine on your Fabric Spark pool or environment. Existing Python and Scala UDFs and queries over complex types are accelerated without any code changes. To learn more, refer to the Python UDFs, Scala UDFs, and complex data types in the native execution engine documentation. Efficient Scaledown with the remote shuffle manager (Preview) Efficient Scaledown decouples Spark shuffle data from executor lifetime in Microsoft Fabric. Instead of pinning shuffle output to local executor disks, Fabric Spark routes large shuffles to Azure Blob Storage and migrates blocks off executors before they're released. Clusters scale down faster, compute costs drop, and jobs become more resilient — with no changes to your queries, notebooks, or pipelines. The feature combines four cooperating capabilities: the Remote Shuffle Manager writes and reads shuffle data to Azure Blob Storage; Shuffle Migration moves blocks off an executor before decommissioning instead of dropping them; a Decision Layer routes small shuffles to local disk and large shuffles to remote storage per stage; and AQE Shuffle Write lets Adaptive Query Execution shape partitioning the first time. Why this matters Clusters scale down faster after demand drops. Lower compute cost from quicker executor release. More resilient jobs with fewer stage retries. No changes to queries, notebooks, or pipelines. To use it, enable the native execution engine and run on Runtime 1.3 (Apache Spark 3.5) or later; autoscale is recommended. Remote Shuffle Manager spark.conf.set("spark.remote.shuffle.enabled", "true") Decision Layer — per-stage routing of local vs. remote shuffle spark.conf.set("spark.sql.rsm.decisionlayer.enabled.level", "stage") AQE participates in shuffle write spark.conf.set("spark.sql.adaptive.shuffleWrite.enabled", "true") Shuffle Migration on executor decommission spark.conf.set("spark.storage.decommission.shuffleBlocks.enabled", "true") spark.conf.set("spark.storage.decommission.shuffleBlocks.cleanup", "true") spark.conf.set("spark.storage.decommission.shuffleBlocks.migrateToFallbackStorage", "true") spark.conf.set("spark.storage.decommission.fallbackStorage.cleanUp", "true") To learn more, refer to the Efficient Scaledown and remote shuffle manager in Microsoft Fabric documentation. Customer-managed key encryption for Spark shuffle data on disk (Generally Available) Microsoft Fabric Spark now generally supports customer-managed keys for Spark jobs through a disk encryption set. Shuffle data written to cluster disks during a Spark job is encrypted with a key you supply and control, giving you ownership of the encryption material that protects intermediate data at rest. With a disk encryption set configured, the cluster disks that hold Spark shuffle data are encrypted using your customer-managed key rather than a platform-managed key. You manage the key lifecycle — including rotation and access — in your own key vault, so encryption of intermediate Spark data follows your organization's key management policies. Why this matters You control the key protecting shuffle data. Intermediate Spark data on disk is encrypted at rest. Key lifecycle and rotation stay in your control. Encryption aligns with your key management policies. To use it, configure a disk encryption set backed by your customer-managed key and associate it with your Fabric Spark configuration. Spark jobs then encrypt shuffle data on cluster disks with your key. To learn more, refer to the Customer-managed key encryption for Fabric Spark documentation. Query your data instantly with the Lakehouse Query Explorer (Generally Available) The Lakehouse Query Explorer is a new, fully integrated query editor built directly into the Lakehouse experience in Microsoft Fabric. You can now write and run Spark SQL queries right where your data lives — no need to switch to a SQL endpoint or spin up a notebook for quick exploration. Whether you’re validating datasets, iterating logic, or exploring patterns, Query Explorer keeps you in flow. orer with the new integrated Query Explorer. Key capabilities Fast, lightweight Spark execution powered by the Lakehouse Livy endpoint. IntelliSense + rich editing experience for faster query authoring. Query across schemas and lakehouses in a single tab. Built-in results grid + inline charts to explore results instantly. Multiple dynamic tabs to analyze different slices of data side-by-side. From quick lookups to multi-table exploration, Query Explorer makes working with Lakehouse data faster and more intuitive—right from the explorer. Learn more and get started with the Lakehouse Query Explorer documentation. Analytics and Insights for Materialized lake views (Generally Available) Analytics and Insights bring continuous refresh intelligence to your lakehouse — two new tabs beside Recent run(s) that shift you from reacting to individual failures to staying ahead of performance drift, rising costs, and silent inefficiencies across your entire materialized lake view estate. The Recent run(s) page tells you what happened in a single execution, including which views succeeded, which failed, and how long the job took. That's valuable when something breaks, but it doesn't answer the questions that matter most for day-to-day operations. Are my durations stable or slowly climbing? Is a new error class appearing and spreading across views? Are my schedules still aligned with how often upstream data actually lands? Am I paying for full refreshes on views that could run incrementally? These are the questions that separate a well-tuned deployment from one that quietly accumulates cost and risk until something finally breaks loudly enough to notice. The Recent run(s) page answers all of them. The Analytics tab transforms your run history into trend lines, distributions, and comparisons you can read at a glance: duration trajectories, success-rate shifts, error-class frequency over time. The Insights tab goes further by watching those same patterns the way a seasoned reliability engineer would, recognizing the signatures behind slow, failing, or wasteful runs, and providing a prioritized list of worthwhile changes. Each recommendation names the affected view, explains why it's flagged, and estimates the payoff in runtime savings, so you can act in seconds rather than investigate for hours. Together, they move refresh management from a reactive, break-fix posture to a proactive, continuously improving one — keeping your materialized lake views fast, healthy, and cost-efficient as your estate grows. To learn more, refer to analytics charts, insight categories in Materialized Lake Views. Introducing Event-Driven Refresh for Materialized Lake Views (Preview) Event-driven refresh brings responsive refresh intelligence to your lakehouse — a new scheduling mode alongside time-based schedules that shifts you from refreshing on the clock to refreshing the moment your data is ready, so your materialized lake views reflect reality instead of an arbitrary calendar. Time-based schedules tell your views when to run: every hour, every morning, every night. That's dependable when upstream data lands like clockwork, but it doesn't answer the questions that matter most for day-to-day operations. Did the ingestion pipeline that feeds this view finish before I refreshed it? Am I recomputing gold-layer views on a fixed cadence while the source data only changes twice a day? Am I paying for refreshes that run before new data has even arrived — or worse, serving stale results because the next scheduled slot is still hours away? These are the questions that separate a refresh strategy tuned to your data from one that quietly burns compute on empty runs and lags the moments that matter. Event-driven refresh answers all of them. You bind a view or a lineage sub-chain to the events that should drive it, and we support two types of events: OneLake events — all file and folder events are supported, so a refresh can fire the moment data lands in OneLake (file or folder creation, update, and more). Job events — Pipeline and Notebook events are supported, so a refresh can fire when the ingestion job that feeds your views completes. The moment an event fires, Fabric resolves the dependency chain and refreshes exactly the views that depend on it. Each trigger names the source event, scopes precisely to the affected lineage, and can react to success or failure, so a stalled upstream job never silently cascades into stale downstream reports. Together with multi-schedule support, event-driven refresh moves refresh management from a fixed-clock, guess-the-cadence posture to a responsive, data-driven one — keeping your materialized lake views fresh the instant new data arrives, and idle when it hasn't, as your estate grows. Learn more and get started with the Schedule a Materialized Lake View Refresh documentation. Data Science AI functions: new models, no package dependency, better usage stats (Generally Available) Fabric AI Functions now use gpt-5-mini as the default model, with “low” reasoning enabled. This powers AI Functions across pandas, PySpark, Data Warehouse, and Dataflows Gen2. For more sophisticated transformations, users may configure gpt-5.1 or tune the reasoning_effort parameter for additional compute and higher-quality results. The gpt-4.1 model has been retired. Pipelines pinned to gpt-4.1 have migrated to gpt-5.1, and those pinned to gpt-4.1-mini migrated to gpt-5-mini. We’ve also simplified PySpark AI Function chaining. The PySpark .ai interface now stays bound to the result schema, so chains like summarize → classify no longer require intermediate DataFrames. In addition, PySpark now supports df.ai.stats for detailed token usage after any AI function call, including reasoning token breakdowns. For pandas, AI Functions no longer require the openai-python package. Capacity-limited rows are surfaced as CapacityExceededResult, enabling clean retries via aifunc.split_results. To learn more, refer to the AI Functions documentation. Data Warehouse Lakehouse table health check (Generally Available) Lakehouse Table Health Check gives you a simple, T-SQL–based way to validate whether your Lakehouse tables are optimized for the SQL analytics endpoint. A single stored procedure surfaces common layout issues, such as small files and fragmentation, offering the insights you need to determine if your tables need to be optimized. You can integrate health checks into pipelines and operational workflows, enabling proactive, at-scale optimization instead of reactive troubleshooting. -- Run a health check on a Lakehouse table from the SQL analytics endpoint EXEC sp_get_table_health_metrics 'dbo.FactSales'; aluate table health. To learn more about sp_get_table_health_metrics and how to integrate it into your Pipelines to optimize your table only if anomalies are detected, refer to sys.sp_get_table_health_metrics (Transact-SQL). Scalar User‑Defined Functions - Procedural computation for analytical SQL (Preview) Scalar User-Defined Functions now support procedural computation — including loops, multiple return paths, and rich IF/THEN/ELSE branching — running natively within the warehouse engine. Computation-based Scalar UDFs are designed for analytical query shapes, integrating naturally with CTEs, GROUP BY, HAVING, and ORDER BY, and executing efficiently at data warehouse scale. Defining business rules once, in SQL, and reuse them across queries, reports, and pipelines. To learn more, refer to the Create Function documentation. Usage-based resource estimations (Generally Available) The Query Optimizer uses learned resource estimation to correct underestimation in T-SQL query plans. It saves actual cardinalities from past executions and automatically adjusts row count estimates in subsequent runs. With this release, the optimizer will also begin correcting overestimations — closing the loop in accurate cardinalities, leading to more efficient resource requests and improved concurrency. Real-Time Intelligence Improved tile error experience in Real-Time Dashboard (Generally Available) We improved the way tile errors appear in Real-Time Dashboards to make them clearer, calmer, and easier to act on. Instead of showing a disruptive red error state, tiles now use a neutral grey error state with a short category header, such as Syntax error, Semantic error, Data source issue, Network error, or Something went wrong. Users can select Details directly from the tile to open a popover with the full engine error message, shown exactly as received. This keeps the dashboard readable while still making the technical details easy to access, copy, and share when needed. This update helps users understand what went wrong faster, while ensuring that one failed tile does not block or visually overwhelm the rest of the dashboard. To learn more, refer to the Troubleshoot Real-Time Dashboard Tile Errors documentation. Folders in Eventhouse tree (Generally Available) Folder support in the Eventhouse tree is now available through the UI. Previously, folders could only be created and managed via code. With this update, you can now organize your Eventhouse directly from the tree, making it easier to manage assets at scale. You can group tables, shortcuts, materialized views, functions, and data streams into a structured hierarchy, improving navigation and reducing clutter. Key capabilities Create, rename, and delete folders from the UI. Organize assets into folders. Move items via the context menu (⋯ → Move to). Create folders inline while moving items. This allows for a simplified and more intuitive way of managing growing Eventhouses. To learn more, refer to the Manage and monitor a KQL database table documentation. Eventstream connector private network support (Generally Available) Data is a critical asset for organizations, and access to real-time data is increasingly essential. However, many high-value data sources reside in private network environments — cloud virtual networks or on-premises infrastructure — particularly in highly regulated industries such as banking, finance, and telecommunications, where strict security and compliance requirements are mandatory. Eventstream's private network support establishes a secure, managed bridge using your Azure virtual network and VNet injection, allowing Eventstream streaming connectors to run inside your virtual network and reach private sources without opening them to the public internet. Whether your data sources reside in on-premises networks, private networks on third-party cloud services, or private networks on Azure, you can connect the bridge Azure virtual network to your source's private network using suitable connectivity options — such as VPN or ExpressRoute for on-premises environments, and private endpoints or network peering for Azure-based sources — enabling Eventstream connectors to securely ingest real-time data from these protected environments into Fabric. The solution leverages a new concept — Streaming virtual network data gateways — which abstracts the bridge Azure virtual network and subnet resource within Fabric. By creating a connector with a streaming virtual network data gateway associating it with your connection, the Eventstream connector is provisioned within your virtual network, ensuring secure communication with your private data sources. Once real-time data from your private network source is securely brought into Fabric Eventstream, you can fully leverage the comprehensive analytics tools in Fabric Real-Time Intelligence to power your real-time scenarios with enterprise-grade security. To learn more about configuration and advanced scenarios, refer to the Eventstream private network streaming guide. Azure Event Hubs source in Eventstream now supports workspace identity authentication (Preview) Currently, when configuring an Azure Event Hubs source in Eventstream, the only supported authentication method is Shared Access Key — a connection string containing static credentials. While simple to set up, shared access keys present several security risks in production environments: they have unlimited lifetime unless manually rotated, provide no per-user or per-application identity, and if accidentally leaked through source code, configuration files, or third-party sharing, grant full access to anyone who obtains them. Revoking a compromised key requires regeneration, which disrupts all services depending on it — with no straightforward way to audit which clients used the key. To address these challenges, we're introducing Workspace Identity as a new authentication option for the Azure Event Hubs source connector (Extended features) in Eventstream (Preview). A Fabric workspace identity is an automatically managed service principal associated with your workspace. Fabric manages the credentials entirely — there are no secrets to store, rotate, or risk leaking. It integrates with Microsoft Entra ID, providing identity-based access with full audit trails, fine-grained role-based access control, and automatic credential lifecycle management. To use workspace identity authentication with your Azure Event Hubs source in Eventstream: Navigate to your workspace settings and create a workspace identity on the Workspace identity tab. In your Azure Event Hub namespace, assign the appropriate role (e.g., Azure Event Hubs Data Receiver) to the workspace identity's service principal. When adding an Azure Event Hubs source in Eventstream, select Workspace Identity as the authentication method — no connection string or key is needed. Eventstream will automatically obtain tokens using the workspace identity to securely connect to your Event Hub, eliminating credential management overhead while strengthening your security posture. To learn more about the configuration, refer to the Azure Event Hubs source extended connector configuration documentation. Custom CA and mTLS support in Eventstream streaming connectors (Generally Available) Fabric Eventstream under Real-Time Intelligence provides various streaming connectors, enabling the integration of real-time data from popular sources into Fabric. When the Eventstream connector client establishes a connection with sources, it is required to implement TLS or mTLS encryption to fulfil the necessary security standards. Many organizations use certificates issued by private or internal Certificate Authorities or require mutual TLS (mTLS) authentication where both the client and server verify each other's identity before transmitting data. Without custom CA and mTLS support, Eventstream connectors cannot connect to these secured source systems. The Custom CA and mTLS support feature, is now generally available for MQTT, Apache Kafka, AWS MSK, and Confluent Cloud for Apache Kafka source connectors. Customers can specify their custom CA and client certificates managed in their own Azure Key Vault when configuring their source in Eventstream. Once specified, Eventstream connector will fetch the certificates from the customer's Azure Key Vault and use them to establish a mutually authenticated, encrypted connection — enabling secure, compliant real-time data ingestion across all supported streaming sources. To learn more about the configuration, refer to the Eventstream sources overview page and choose the corresponding source. Introducing the Oracle CDC connector for Eventstream (Preview) Eventstream now introduces the Oracle Database Change Data Capture (CDC) connector, enabling you to stream database change events directly from any Oracle Database — whether running in the cloud or on-premises — into Eventstream for real-time processing and analytics. Many organizations run important operational workloads on Oracle Database and need to react to changes as they happen. With the Oracle CDC connector, you can continuously capture change events from Oracle Database and bring them into Fabric without building custom polling applications or managing separate integration services. With the Oracle CDC connector, you can: Capture and stream databases changes from Oracle Database into Fabric in real time. Connect to Oracle databases running either on-premises or in the cloud. Process incoming change events using Eventstream transformations. Route processed change events to supported destinations such as Eventhouse, Lakehouse, Activator, or custom endpoints. This capability helps you build real-time analytics and event-driven applications from Oracle data. For example, you can route transaction changes to an Eventhouse Kusto table for operational analysis, send selected events to Activator for alerting, or combine Oracle change events with other streaming sources in the same eventstream. To learn more about Eventstream Oracle CDC connector, refer to Add Oracle Database CDC source to an eventstream (preview). Eventhouse update policies now support referencing accelerated shortcuts in update policies (Generally Available) Eventhouse update policies now support accelerated shortcuts in update policy queries, enabling ingestion-time enrichment scenarios. Use this for dimension lookups, such as enriching ingested fact events with customer, device, or product attributes stored in OneLake shortcut data. The shortcut-backed external table must have Query Acceleration Policy enabled, and Hot must cover all data. For update policy scenarios, set: .alter external table DimCustomer policy query_acceleration '{"IsEnabled":true,"Hot":"36500.00:00:00"}' Then join to it from the update policy query: .alter table EnrichedEvents policy update '[{ "IsEnabled":true, "Source":"RawEvents", "Query":"RawEvents | lookup kind=leftouter (external_table(''DimCustomer'')) on CustomerId","IsTransactional":true, "PropagateIngestionProperties":false }]' Processing uses the authorization context captured in the system-populated OwnerPrincipalDetails property: the user who creates or alters the update policy must have access to the shortcut data. This enables ingestion-time enrichment with governed shortcut data without separate orchestration. Shortcut Tables in Eventhouse Now Automatically Synchronize Schema Changes To help maintain consistency between source data and shortcut tables, the default schema synchronization behavior for Eventhouse shortcut tables is changing. Previously, schema changes made to the source table were not automatically propagated to the shortcut table unless schema synchronization was explicitly enabled. As a result, source and shortcut schemas could diverge over time. With this update, all new and existing shortcut tables in Eventhouse automatically synchronize schema changes from their source table by default, including: Adding new columns Changing column data types Renaming columns Deleting columns Automatic schema synchronization helps ensure that shortcut tables remain aligned with the source schema, preserving the latest business context and reducing manual maintenance. Customers who prefer to disable automatic schema synchronization can continue to control this behavior using the existing KQL management command: .create-or-alter external table ExternalTable kind=delta ( h@'https://storageaccount.blob.core.windows.net/container1;secretKey' ) with (AutoUpdateSchema=false) Note: Because schema changes are now automatically propagated, queries, dashboards, and downstream workloads may require updates if they reference columns that are renamed, removed, or otherwise modified in the source table. Investigator Insights in Operations Agent (Preview) When an anomaly is detected, understanding what caused it is often the hardest part. Investigator insights are designed to make that easier by analyzing the surrounding data and surfacing relevant context. When the operations agent detects that a rule has been met, there is an option to run an investigation in the background to identify correlated signals and patterns. Instead of manually digging through telemetry, you get a guided view into what changed, what stood out, and what may have contributed to the issue. This helps you move more quickly from detection to understanding. You can access these insights directly from Teams. Open the agent’s message and select Investigate further to generate a detailed analysis. The investigation provides a structured view of what happened: Investigation scope shows which tables were analyzed, whether from a single source or across related datasets. Key observations highlight the most important findings, including actual values, deviations from baseline, and notable trends or outliers. Pattern analysis surfaces meaningful changes around the time of the anomaly, such as dimensions with significant shifts, and clearly call out when no strong patterns are identified. Together, these insights help you quickly understand not just that something went wrong, but why it happened. To learn more, refer to the Operations Agent Actions documentation. Anomaly Detector Configurations Pane As you build on top of your data, it is often just as important to understand what already exists as it is to create something new. This update makes it easier to discover and build existing anomaly detection configurations. You can now view all anomaly detection configurations that have already been created for a given data source in one place. This lightweight experience gives you quick visibility into how anomaly detection is currently set up, helping you avoid duplicate work and better understand how others are using the data. From this view, you can explore existing configurations or create a new one if your use case is not yet covered. This makes it simple to extend existing setups or start fresh when needed, all without leaving the context of your data source. By making configurations easier to discover and reuse, this experience helps streamline workflows and ensures you can move quickly from exploration to action. To learn more, refer to the Anomaly Detection in Real-Time Intelligence documentation. Ingestion time stamp in Anomaly Detector (Preview) Working with anomaly detection often assumes your data already includes a clean, reliable timestamp. With this update, you can now use the system-generated ingestion time as the timestamp for anomaly detection. This means that even if your dataset does not include a dedicated timestamp column, you can still run analysis without needing to modify or preprocess your data. This is especially helpful for scenarios where events are ingested in real time or where timestamps are missing, inconsistent, or not trustworthy. Instead of blocking data preparation, you can rely on ingestion time to move forward with detection and start identifying meaningful patterns right away. By reducing setup requirements and removing a common dependency on source data quality, this capability makes it easier to apply anomaly detection across a wider range of use cases. To learn more, refer to the Anomaly Detection in Real-Time Intelligence documentation. Configuring Anomaly Detection without a Group by Column (Preview) With this update, Anomaly Detection now supports scenarios where you choose not to use a group by column. This provides an additional configuration option for datasets that already represent a single stream of data, allowing you to apply anomaly detection directly to the metric without first identifying a grouping dimension. This added flexibility helps the configuration experience better align with how your data is structured. Whether you are monitoring a single device, tracking a specific service, or analyzing a focused dataset, you can now create anomaly detectors without requiring a group-by column. At the same time, grouping remains available for scenarios where you want to monitor and compare multiple entities within the same dataset. To learn more, refer to the Anomaly Detection in Real-Time Intelligence documentation. Introducing Fabric Maps Tilesets: High-Performance Visualization for Large Geospatial Datasets (Generally Available) Have location-based data sitting in OneLake but no simple way to see it come alive on a map? With Fabric Maps Tilesets, you can now turn that data into fast, interactive map experiences directly inside Microsoft Fabric. The Tileset Builder lets you create map-ready PMTiles from OneLake data without custom code, manual exports, or separate geospatial infrastructure making it easier for teams to explore large geospatial datasets, uncover patterns, and bring location intelligence into the analytics workflows they already use. Organizations can rely on Microsoft Fabric to manage operational data, analytics workflows, and business reporting. When that data includes locations, routes, assets, boundaries, or events with a location context, teams need a simple way to visualize it on a map. Traditionally, this required separate geospatial pipelines, custom polling services, or manual exports. Leveraging a Fabric Map removes complexity by allowing organizations to create and refresh map-ready tilesets from data already stored in OneLake. Try creating your own Tileset using the following steps: Connect to a lakehouse and select source files Configure tileset metadata Configure layer settings Tileset schedule (Preview) Review and create tileset What is a Tileset? Tilesets are map-optimized representations of geospatial data. Instead of trying to load a large geospatial data file all at once, data is divided into small tiles that are loaded and rendered as needed while users zoom and pan across the map. This makes tilesets especially useful for large datasets, such as infrastructure networks, delivery routes, asset locations, service areas, or operational events. Fabric Maps Tileset Builder Capabilities Build map-ready tilesets directly from OneLake data. Visualize large geospatial datasets with high performance. Enable smooth, interactive map experiences at scale. Keep map content synchronized with source data. Eliminate manual exports and external geospatial processing workflows. Integrate native geospatial visualization into existing Fabric data workflows. Common scenarios Tilesets help teams turn large location-based datasets into fast, interactive map experiences. Utility and energy companies can visualize nationwide power line grid systems as a single dataset to monitor field assets and service coverage. Supply chain and logistics teams can explore daily routes, regional boundaries, and operational areas. Retailers can analyze store territories, expansion opportunities, and new developments. Because the data stays connected to Fabric, these map experiences become part of the broader analytics workflow — not a separate geospatial process. To get started, refer to How to create tilesets. Cross-domain intelligence with Azure Monitor data in Microsoft Fabric (Preview) Have you ever detected an issue in your systems but struggled to understand what it meant for the business? As systems grow more complex, this gap becomes harder to bridge. Incidents no longer affect just systems; they affect customers, revenue, and operations in real time. Azure Monitor Logs mirroring into Microsoft Fabric helps close that gap. In just a few steps, telemetry from Log Analytics workspaces becomes available in OneLake alongside business and operational data — without duplication and with near real-time availability. This creates the foundation for Cross-domain insights and actions: Bring observability, operational, and business data together in Eventhouse for real-time analysis. The same unified data can be used by Real-Time Dashboards for investigation and by operations agents to recommend and drive actions informed by both business and observability context. For example, an operations team can identify that a check-in kiosk outage is impacting high-value customers and act before customer impact grows. Advanced Fabric analytics Apply tools like Spark and Power BI for long-term analysis, machine learning, and a wide range of analytical scenarios. For example, an operations team can create a Power BI report showing trends in customer impact and cost, helping management make informed decisions on resource allocation and understand the true cost of application failures. Together, these capabilities help organizations move from isolated technical signals to business-aware insights, decisions, and actions. To learn more, refer to Cross-domain intelligence with Azure Monitor data in Microsoft Fabric (Preview). Until next month That's a wrap for the July 2026 Microsoft Fabric Monthly Update. As always, we'll continue sharing new capabilities, enhancements, and improvements across Microsoft Fabric in future monthly updates. Thank you for being part of the Fabric community!
Fabric August 2026 Feature Summary
Welcome to the August 2026 Fabric update! Microsoft Fabric continues to evolve with new capabilities that help organizations build, manage, and scale their data and AI solutions more efficiently. This month's updates introduce enhancements across Fabric Platform, OneLake, Data Engineering, Conversational Analytics, Data Warehouse, Real-Time Intelligence, and Data Factory. Whether you're strengthening governance, improving performance, streamlining development workflows, or expanding AI-powered experiences, these updates are designed to help you get more value from your data while simplifying day-to-day operations. Explore the following highlights to see what's new in Microsoft Fabric this month. Events and Announcements Join us for FABCON and SQLCON in Barcelona, September 28 – October 1, 2026 Explore what’s possible with Microsoft Fabric and get up to speed on the latest in SQL, analytics, and AI. From 130 sessions and 4 keynotes to workshops, the expo, community spaces, and the Power BI DataViz World Championships, this is where the data community comes together. Learn directly from Microsoft and community experts shaping the future of Fabric and SQL. Register now and save €200 with code FABCMTY200. Fabric Platform KQL-Dashboard Embed in Fabric (Preview) A new capability that lets you add interactive KQL-Dashboard content directly into your own browser-based web applications is available. Now, you can bring Fabric analytics into the apps, portals, and workflows your users already use. Analytics are most useful when they are available where decisions happen. With Fabric Embed, you can place Fabric content inside a custom application or internal portal instead of requiring users to switch to the Fabric portal. Fabric Embed can help you: Bring interactive analytics into an existing business workflow. Explore Fabric analytics without switching between your application and the Fabric portal. Keep Fabric workspace permissions and Microsoft Entra ID identities at the center of access control. Build user-based embedded experiences for people who already have access to the underlying Fabric item. The embedded experience complements the Fabric portal. Content owners can continue to create and manage analytics in Fabric while application developers present that content in the context most useful to their users. To learn more, refer to the Microsoft Fabric Embed documentation. Git Integration – Workspace Relation API (Preview) Following our recent announcement of branch workspaces and the relationship that’s automatically created when a user performs a branch-out operation, we’re introducing the new Workspace Relations API. These endpoints let you create, query, update, and delete relationships between Fabric workspaces programmatically — and for anyone building automation around Git integration, branched workspaces, and CI/CD, it’s a big deal! The workflow is straightforward. A developer creates a feature branch. An ADO pipeline or GitHub Action then provisions a new feature workspace, configures Git, applies the right settings and permissions, and synchronizes the workspace. Finally, the Workspace Relations API links that feature workspace back to its parent — closing the loop and giving your automation a first-class, queryable connection between parent and branched workspaces. Figure: Branch workspace relation. To learn more, refer to Development process using Branch-Out experience. Git Integration and Deployment Pipeline – Item level permission restriction Starting December 1, 2026, users without read-write permissions on workspace items can't use Git integration and won’t be able to deploy a workspace or assign workspace to a stage via Deployment Pipeline. This restriction can result in loss of access to certain items because of sensitivity labels and protection policies applied to those items. To learn more, refer to Information Protection in Microsoft Fabric. OneLake Resource instance rules for OneLake (Generally Available) Resource instance rules for OneLake are ready for production workloads across enterprise analytics environments. They give workspace admins a precise way to allow access from trusted Azure resource instances while continuing to enforce network and data-level protections. Use resource instance rules when you need to enable secure service-to-service access to OneLake without relying solely on IP allowlists or requiring private networking for every integration. Combined with Private Link, IP firewall rules, and identity-based permissions, they help organizations apply layered security based on the needs of each workspace and workload. Resource instance rules support a broad set of Azure services that can present a verifiable Azure resource identity, including Azure Databricks, Azure SQL Server, Azure Data Factory, Azure Event Grid, Azure Machine Learning, and more. Configuration requires only the Azure resource ID, simplifying setup while maintaining control over which Azure resources can access OneLake. To learn more, refer to Manage inbound access to OneLake with Resource Instance Rules. Data Engineering Native Execution Engine performance improvements This month, we continued to improve the Native Execution Engine (NEE) with a new set of query execution optimizations designed to accelerate Spark workloads while reducing compute consumption. Key enhancements include broadcast joining reuse across queries, native acceleration for ranking window functions such as RANK and DENSE_RANK, and automatic materialization of reused Common Table Expressions (CTEs). Together, these optimizations eliminate redundant computation, keep more processing within the engine's vectorized execution path, and improve performance for common data engineering patterns including large joins, analytical reporting, ranking workloads, and complex transformation pipelines. Because these improvements are enabled automatically when NEE is available, customers benefit from faster execution times, lower Fabric capacity consumption, and improved price-performance without requiring code changes to existing notebooks, Spark Job Definitions, or pipelines. These optimizations are enabled by default once the Native Execution Engine is turned on, allowing customers to realize performance gains immediately without additional configuration or tuning. Customers can enable NEE at the workspace environment level by navigating to Environment > Acceleration and turning on Native Execution Engine, ensuring it is available for all Spark sessions using that environment. It can also be enabled at the session level by setting spark.native.enabled=true in Spark configuration. With NEE enabled, customers can seamlessly take advantage of the latest runtime innovations to process data faster, improve resource efficiency, and maximize the value of their Fabric capacity investments. To learn more about Native Execution Engine explore our documentation Native execution engine for Fabric Data Engineering. Fabric Runtime 2.0 (Generally Available) As the execution foundation for Microsoft Fabric's Data Engineering and Data Science experiences, Runtime 2.0 delivers a modern, high-performance platform built on Apache Spark and deeply integrated across the Fabric ecosystem. Purpose-built for large-scale data processing and analytics workloads, Runtime 2.0 represents a major advancement in performance, reliability, security, and future readiness. Built on the latest open-source innovations, it enables customers to accelerate data processing, simplify operations, and take advantage of the newest capabilities across Microsoft Fabric. This release includes significant platform upgrades, including Apache Spark 4.1, Delta Lake 4.2, Python 3.13, Java 21, Scala 2.13, and Azure Linux 3.0, providing a modern and enterprise-ready foundation for the next generation of data engineering, data science, and analytics workloads. These enhancements enable customers to take advantage of the latest open-source innovations while continuing to benefit from a fully managed, enterprise-grade experience in Microsoft Fabric. Whether you're building data pipelines, developing AI and machine learning solutions, processing streaming workloads, or powering enterprise analytics, Runtime 2.0 provides a more capable, scalable, and performant platform for your workloads. With improved performance, updated open-source foundations, and continued investment in capabilities such as the Native Execution Engine, Runtime 2.0 provides a modern platform for data engineering, data science, and analytics workloads in Microsoft Fabric. Explore the full documentation and start using Runtime 2.0 in production Runtime 2.0 in Fabric. Enhanced Spark Properties Support in Notebook and Spark Job Definition Notebook and Spark Job Definition (SJD) activities now enable users to specify Spark properties directly within the data integration pipeline. This enhancement allows Spark properties to be set inside the activity panel, ensuring that the values provided are used for activity execution. If an Environment item is linked to the Notebook or SJD and both the activity panel and Environment item define the same property, the value specified in the activity panel will take precedence and overwrite the Environment value. In the case of Notebooks, if the %%configure command is used within the notebook code to set a Spark property, the value set using %%configure will be applied for execution. This update offers users maximum flexibility, allowing Spark properties to be defined at different layers based on their specific use cases. By supporting property configuration in the activity panel, Environment item, and notebook code, users can tailor property values to meet the unique requirements of each execution of Notebook/SJD. To learn more on Transform data by running notebook and Transform data by running a Spark Job Definition activity. Conversational Analytics Enhanced Data Agent Visualizations with Fabric Visuals The data agent now uses Fabric visuals to render the charts it returns, bringing higher-quality, more consistent visualizations into your conversations with your data. When you ask a question like "Generate a bar chart of revenue by region" or "Show me my top 10 customers by sales," the data agent responds with an interactive, polished visual alongside its text and table answers, so you can spot trends, comparisons, and outliers. Because the data agent now shares the same visual foundation as Fabric Apps, charts look and behave consistently with AI-generated visuals in Fabric, with refinements to formatting, legends, tooltips, and axis scaling. Supported chart types include line, bar, stacked bar, pie, scatter, and area charts. To learn more, refer to the Get visual responses from a Fabric data agent documentation. Advanced DAX Generation for Semantic Models in Data Agents Advanced DAX generation for Power BI semantic models is now available in Fabric data agents when you use the Preview runtime. Instead of generating a DAX query in a single pass, the new system works iteratively as a specialized sub-agent that can use tools, inspect results, and refine its approach across multiple steps, providing significant improvements in response accuracy. It also uses instance value indexing to resolve values from the semantic model before generating a query, resulting in more accurate and reliable filters. This update is built on the same semantic-model query engine used across Fabric Skills, Power BI, and M365 Copilot, providing more consistent answers across Microsoft experiences. To use the new experience, open the Runtime dropdown in the data agent ribbon and switch from Standard to Preview. More improvements for semantic models in data agents are coming soon, including data source description and instructions, granular schema selection, and example queries. In the Preview runtime, the data agent uses advanced DAX generation to answer a question over a connected semantic model, enabling more accurate DAX generation and responses. To learn more, refer to the Semantic model best practices for data agent documentation. Data Agent orchestrator upgraded to GPT 5.1 The data agent orchestrator now runs on GPT-5.1, across both the standard and preview runtimes. The orchestrator handles how questions are rephrased, how work is planned across your data sources, and how the final answer is composed — so this upgrade changes behavior in all three. Most of what you see should be an improvement in answer quality and planning, but the change is not behavior-neutral: prompts tuned against the previous model may produce different results. We recommend re-running your evaluations, reviewing the results against your saved baselines, and updating your agent instructions and prompts where the new behavior doesn't match what your scripts expect. To learn more, refer to the data agent runtimes documentation. Example Query Usability Improvements in Data Agent We've made several usability improvements, for example queries. Errors now surface inline, directly alongside the query, so you can identify and correct issues without leaving the editor. The editor also resizes automatically based on the length of your query, removing the need to adjust the pane manually as you write. To learn more, refer to the example queries in data agent documentation. Add Schema Descriptions for SQL Sources in Data Agent Users can now provide tailored schema descriptions through the new schema description editor, available for SQL sources on the Preview runtime. Schema descriptions improve query generation and accuracy by giving the agent more context about what everything in your data means — use them to resolve ambiguous columns, or to give a table or field a more precise meaning than its name conveys. Instead of inferring intent from column names alone, the agent works from what your data team documented, so it selects the right tables and interprets fields the way you intended. To learn more, refer to the schema descriptions documentation. Data Agent is migrating from Assistants API to Responses API The OpenAI Assistants API that powers the orchestration layer for the Microsoft Fabric data agent, is currently scheduled to be shut down by OpenAI on August 26, 2026. After that date, direct calls to the Assistants API will stop working. If you connect to a Fabric data agent programmatically through the Assistants API, you need to migrate to the data agent Model Context Protocol (MCP) endpoint. SDK and Fabric portal users require little or no action because Microsoft will migrate those experiences internally, although conversation history may reset once. Existing agent data sources, instructions, and tools remain unchanged. To learn more, refer to Prepare your Fabric Data Agent integrations for Assistants API retirement. Fabric Data Agents in Microsoft Copilot Studio (Generally Available) Now, you can bring governed business data from Microsoft Fabric into Copilot Studio agents, so those agents can answer questions and support business processes using trusted enterprise data. Since preview, the integration has moved to the new tool-based experience: select Add a tool, search for Fabric, and add Fabric IQ Data MCP, and your agent can call the Fabric data agent like any other tool. The Fabric data agent still runs in Fabric and respects permissions on the underlying data sources. You can also publish your agent to Microsoft Teams and Microsoft 365 Copilot, so business users get data-grounded answers where they already work. To learn more, refer to the Fabric Data Agent MCS GitHub documentation for setup steps and join the community discussion to share feedback. Fabric data agents in Microsoft Foundry: Easier to connect, easier to trust Fabric data agents in Microsoft Foundry are now easier to connect and easier to monitor. The integration moves to Model Context Protocol, so your Fabric data agents appear as tools that Foundry agents can invoke when they need enterprise data in OneLake. Connecting them no longer means hunting for workspace and artifact IDs. You add the Fabric IQ (OneLake Catalog) tool, filter for data agents, and pick the ones you want by name. You can also connect more than one Fabric data agent to a single Foundry agent, so an agent can draw on a sales agent, a supply chain agent, and a customer support agent and choose the right one for the question. On the operations side, you can now view logs and traces for Fabric data agents through Foundry Observability. Traces show which tools were invoked, how long each step took, and what came back, which makes it much easier to troubleshoot an answer that looks wrong or a workflow that runs slow. This is the visibility teams need to move agents from experiments to production. The update is rolling out to all regions over the coming days. Add Fabric data agent as part of Fabric IQ to your Foundry agent. To learn more, refer to the Observability for Fabric data agents in Microsoft Foundry documentation. Add co-publishers for data agents in Microsoft 365 Copilot When you publish a Fabric data agent to Microsoft 365 Copilot, the Microsoft 365 agent platform registers the agent and records you as its only owner. That created a problem for teams. Your co-creators could still edit the data agent in Fabric, but when they tried to republish it, the operation failed, because Microsoft 365 only lets registered owners publish. Fabric access and Microsoft 365 ownership are two separate lists, so giving someone edit rights in Fabric was never enough. Now, you can avoid this with co-publishers. After you publish the data agent, open Settings, go to the Publishing pane, and add your Fabric co-creators under Microsoft 365 Copilot co-publishers. Each person you add is registered as a co-owner on the Microsoft 365 agent platform, so anyone on that list can republish the agent. Add co-publishers right after your first publish so no one hits a failure in the meantime. Publishing pane in the Fabric data agent settings, showing where you add Microsoft 365 co-publishers. To learn more, refer to the Consume a data agent from Microsoft 365 Copilot (preview) documentation. Data Warehouse Identity columns with identity insert (Generally Available) Since preview, thousands of customers have adopted IDENTITY to auto-generate surrogate keys and streamline migrations from SQL Server, Azure SQL Database, and Azure Synapse. Now we're introducing support for IDENTITY_INSERT and reseed operations - two highly needed additions - so you can insert explicit key values, migrate data in bulk with COPY INTO, and safely realign identity ranges with DBCC CHECKIDENT. Figure: Using identity insert on Fabric Data Warehouse. IDENTITY columns are available now in every Fabric Data Warehouse. To learn more, check our updated tutorial and documentation. CI/CD 2.0 with DacFx (Preview) Microsoft Fabric Data Warehouse is introducing a major update to the DacFx engine that powers schema comparison, Git integration, and deployment pipelines. DacFx builds a declarative model of your warehouse and determines the schema changes required to move safely between development, test, and production environments. Git-integrated CI/CD workflow for Microsoft Fabric, showing feature workspace synchronization, branch merging, and deployment pipeline promotion across development, test, and production workspaces. With this update, Git integration uses DacFx-based incremental extraction to produce cleaner, more focused commits. Deployment pipelines also use the updated model to generate more accurate comparisons and smarter deployment plans, with settings tuned for schema evolution. The new warehouse item definition version 2.0 updates the SQL project SDK, moves shared queries into a .sharedqueries folder, adds project-level Git configuration, and re-extracts object definitions to support constraints, identity columns, clustering, and consistent formatting. These changes make future commits easier to review and reduce noisy diffs. For more information, refer to the Upgrade Fabric Data Warehouse System File Version in a Git Integrated Fabric workspace documentation. Microsoft Fabric source control notification prompting users to apply the latest Warehouse system update, with a warning that the update will introduce differences between the workspace and its connected Git repository. The update also improves comparison accuracy. Git-connected workspaces can adopt the update when ready through the System update available experience, giving teams control over upgrade timing. Review and commit the generated changes before continuing normal development and deployment workflows. To learn more, refer to the Development and Deployment Overview documentation. Simplify Fabric Warehouse deployments with Schema Compare in VS Code Database deployments should not feel like a guessing game. With Schema Compare in Visual Studio Code, developers can see exactly what changed before those changes reach a Fabric Warehouse—bringing clarity and control to every release. Compare a Fabric Warehouse with another warehouse or a SQL database project, then review differences across tables, views, stored procedures, functions, and other database objects in a clear, object-by-object view. Choose the changes you want, update the project from the warehouse, or deploy selected changes to the target—without manually assembling and reviewing every deployment script. By keeping database projects synchronized in Git, teams gain a reliable source of truth and can bring schema changes into familiar pull-request and CI/CD workflows. The result is a safer, more intentional path from development to production, with fewer surprises at deployment time. Before applying changes, review the generated script for unsupported operations and potential data loss. To learn more, refer to the Develop warehouse projects in Visual Studio Code and Schema Compare in the MSSQL extension documentation. GPU Query Acceleration (Preview) Query Acceleration brings GPU-powered performance directly to Fabric Data Warehouse, enabling eligible analytical queries to run faster without query rewrites, special syntax, or additional systems to manage. Query Acceleration in Fabric Data Warehouse uses GPUs to accelerate the most compute-intensive portions of analytical queries, helping overcome the limits of CPU-only execution. It works transparently with existing T-SQL, Direct Query reports, applications, and tools, automatically offloading eligible operations such as scans, filters, joins, and aggregations to GPUs while the CPU continues to manage the rest of the execution pipeline. Customers can use Query Insights, Data Warehouse Monitoring, and SQL Server Management Studio (SSMS) for query execution plans to identify accelerated queries and understand how Query Acceleration is applied during query execution. Designed for analytical and high-concurrency workloads, Query Acceleration can improve throughput, reduce query latency, and deliver more consistent performance for dashboards and interactive analytics. Acceleration is applied selectively, enabling performance gains even when only part of a query is eligible for GPU execution. The capability is built with reliability in mind. Unsupported operations or runtime constraints can seamlessly fall back to CPU execution without affecting query correctness. Performance improvements depend on workload characteristics, but Microsoft benchmarks have demonstrated gains of up to 7× across reporting, application, and AI-driven analytics scenarios. Query Acceleration builds on Microsoft's Tensor Query Processor research, described in CoddSpeed: Hardware Accelerated Query Processing in Microsoft Fabric, which was selected as the SIGMOD Companion 2026 Best Industry Paper. To sign up for the Preview, please fill out the form. Metadata Sync supports Delta Checkpoint V2 (Generally Available) Metadata Sync (MD Sync) now supports Delta Checkpoint V2, enabling synchronization of modern Delta tables across both MD Sync (Legacy) and MD Sync (New). Delta Checkpoint V2 is a Delta Lake enhancement designed to improve scalability for large tables through a more efficient checkpoint structure. Previously, tables using Checkpoint V2 couldn't be synchronized and were reported as unsupported. With this release, MD Sync can discover and synchronize Delta tables that use the Checkpoint V2 format. This enhancement helps customers: Synchronize Delta tables that use Checkpoint V2. Improve interoperability with Spark, Databricks, and other Delta-based platforms. Support metadata synchronization for large-scale Delta tables more efficiently. Continue using existing checkpoint formats without any changes. MD Sync support for Delta Checkpoint V2 is available in both MD Sync (Legacy) and MD Sync (New), helping ensure consistent access to Delta tables across Fabric experiences. Secure data ingestion with COPY INTO and Workspace Identity (Generally Available) COPY INTO in Fabric Data Warehouse now supports Workspace Identity, enabling users to load approved data from OneLake or ADLS Gen2 without requiring direct access to the source files. Previously, ingestion users often needed permissions to both the target warehouse and the source storage location, or teams relied on SAS tokens, account keys, or service principals. With this release, source access can be centrally assigned to the workspace identity, while users retain only the SQL permissions required to load data into the target table. Key Capabilities: Load approved data without granting users direct access to raw storage. Use managed identity-based authentication for OneLake and ADLS Gen2 sources. Reduce reliance on SAS tokens, shared keys, and service principal secrets. Maintain separate authorization boundaries for source access and target-table permissions. Support least-privilege ingestion and separation of duties between storage and warehouse administrators. Workspace Identity support for COPY INTO is generally available in Fabric Data Warehouse, providing a simpler and more governed approach to secure data ingestion. To learn more, refer to the Ingest Data into Your Warehouse Using the COPY Statement and COPY INTO (Transact-SQL) documentation. SQL Audit Logs: More Signal, Less Noise with Predicate Filtering (Generally Available) SQL Audit Logs in Fabric Data Warehouse and SQL Analytics Endpoint now support identity-based predicate exclusion filtering, enabling administrators to reduce repetitive audit events generated by selected users and service principals. Previously, expected activity from automation identities, scheduled processes, metadata synchronization jobs, and other operational actors could create significant audit noise. With this release, administrators can configure exclusions through the API or SQL Audit Logs user experience, while activity from identities that do not match the exclusion predicate continues to be audited normally. Key Capabilities: Reduce repetitive audit events from known users and service principals. Focus investigations on higher-value and unexpected activity. Lower the storage, processing, export, and query burden associated with low-value events. Manage identity exclusions through either automated APIs or the user experience. Apply a governed audit policy aligned with organizational monitoring and compliance requirements. Identity-based predicate exclusion filtering is generally available for SQL Audit Logs in Fabric Data Warehouse and SQL Analytics Endpoint, providing a cleaner audit stream, less operational overhead, and more focused investigations. To learn more, refer to the SQL Audit Logs in Fabric Data Warehouse documentation. OneLake security improvements for SQL analytics endpoints (Generally Available) OneLake Security for SQL analytics endpoints now includes improvements for nested groups, shortcut-backed tables, column-level security, and service principals, enabling more consistent enforcement of OneLake security policies across enterprise Fabric environments. Previously, limitations with group expansion, shortcut scenarios, and service principal ownership could make centralized security difficult to apply at scale. With these improvements, customers can define security at the source lakehouse and rely on the SQL analytics endpoint to honor those policies across producer and consumer workspaces. Key Capabilities: Manage access through nested Microsoft Entra group hierarchies. Honor source-side OneLake Security policies for shortcut-backed tables in hub-and-spoke architectures. Apply column-level security consistently when users receive access through groups. Use service principals for automated deployments, pipelines, and application-owned data products, including service principal-owned lakehouses. Define security once in OneLake and reduce the need to duplicate permissions across consumer workspaces and Fabric engines. These OneLake Security improvements help make security synchronization more practical for enterprise architectures while providing consistent access control across lakehouses and SQL analytics endpoints. Microsoft is also continuing to improve security sync notifications, error handling, and permission propagation across Fabric experiences. To learn more, refer to the OneLake Security for SQL analytics endpoints documentation. Real-Time Intelligence Set Alerts Directly from Anomaly Detector (Generally Available) Detecting anomalies becomes more valuable if you can act on them. Previously, after publishing an anomaly detector configuration, you had to leave Anomaly Detector and navigate to Real-Time Hub to create an alert. This added extra steps and interrupted your workflow right after completing your configuration. With this update, you can now create alerts directly from Anomaly Detector. Once you publish a configuration, use the Set alert button in the ribbon to launch the alert creation pane without leaving Anomaly detector. If your configuration hasn't been published yet, you'll be guided through publishing first and then taken directly to the alert setup experience. This helps you move seamlessly from configuring anomaly detection to monitoring it in production. The integrated experience allows you to monitor your anomalies on each event, helping you get notified as soon as anomalies are detected. If you have more complex business logic, select on each event when to add in additional logic to your conditions. Whether you're monitoring operational metrics, business KPIs, or real-time telemetry, you can now complete the entire workflow in one place and start acting on detected anomalies faster with fewer clicks. Create alerts directly from your anomaly detector configuration and continue your workflow without navigating to another experience. Configure notifications for anomaly detector events directly within Anomaly Detector and start monitoring your published configuration immediately. Anomaly detector supports Eventhouse shortcut tables Anomaly Detector now supports Eventhouse shortcut tables, making it possible to analyze data without first copying or moving it into a dedicated Eventhouse table. You can create anomaly detectors directly on supported shortcut tables and use the same analysis, model recommendations, and continuous monitoring experiences available for native Eventhouse data sources. This expands anomaly detection to a broader range of data already connected through Eventhouse shortcuts, helping teams monitor external and federated data sources with less setup and duplication. By enabling anomaly detection directly on shortcut tables, you can move more quickly from connecting data to detecting issues, while continuing to work within a unified Real-Time Intelligence experience. To learn more, refer to the Anomaly Detection in Real-Time Intelligence documentation. Operations Agent Activity Log Understanding what your agent is doing and why is key to building trust and improving outcomes. The activity log is designed to provide that transparency. It gives you a clear view into the agent’s behavior, including the conditions it evaluated, the recommendations it generated, and how those recommendations were handled. Whether you are validating results, troubleshooting unexpected behavior, or refining your configuration, the activity log helps you better understand how decisions are being made. You can access the activity log from the Activity log section in the side navigation. It presents a chronological timeline of events with timestamps and relevant context for each entry. Selecting any event allows you to explore additional details and understand what happened at each step. In the Operation details page, you can view the operation details and status. To learn more, refer to the Create and Configure Operations Agents documentation. Eventstream MQTT connector (Generally Available) It is now easier than ever to ingest real-time data from MQTT brokers directly into Microsoft Fabric Real-Time Intelligence. MQTT is one of the most widely adopted messaging protocols for lightweight, low-bandwidth event driven messaging scenarios. Eventstream MQTT connector simplifies the ingestion of operational and IoT data into Microsoft Fabric, helping organizations turn real-time device events into actionable insights. Key Benefits: Connect to any MQTT broker and ingest messages directly into Fabric Eventstream. Production-ready reliability and support with General Availability readiness. Enterprise-grade security with support for TLS, mutual TLS (mTLS), and custom certificate authorities managed through Azure Key Vault. Private network connectivity through Eventstream's streaming connector virtual network capabilities, enabling secure access to brokers hosted in private and on-premises environments. To learn more, refer to the Add MQTT source to an eventstream documentation. Reference data enrichment in Eventstream (Preview) Eventstream now enables you to enrich real-time event streams with contextual business data using Reference Data Join. Simply add a Reference Data node to your Eventstream, select a Delta table from a Fabric Lakehouse, and use it to enrich streaming events with lookup, metadata, or reference information. You can also leverage Lakehouse shortcuts to access Delta tables across OneLake, making it easy to bring contextual data from anywhere in your Fabric environment into your real-time processing pipelines. Reference Data Join supports both no-code and SQL-based enrichment experiences. Use the built-in Join operator to visually configure INNER and LEFT OUTER joins or use the SQL operator for advanced scenarios. Select only the columns you need from the reference dataset and configure optional refresh intervals to keep slowly changing reference data up to date. This enables Eventstream to continuously use the latest lookup information for real-time enrichment, without requiring additional data movement or downstream processing pipelines. You can easily add multiple reference data sources to a single Eventstream and combine them with streaming data to create richer, more contextual event pipelines. Developers and data engineers can test and validate join conditions, preview join results, and verify SQL-based enrichment queries before deploying them into production, helping ensure accuracy and confidence in real-time data processing workflows. Reference Data Join unlocks powerful real-time enrichment scenarios in Eventstream. Users can enrich IoT telemetry with device metadata, correlate operational events with customer and product information, perform lookups against business reference datasets, and add contextual information to streaming data in flight. By bringing reference data and event processing together in a single experience, Eventstream enables customers to transform raw events into actionable business insights in real time. To learn more, refer to the Reference data join in Eventstream using Lakehouse documentation. Eventstream observability in Workspace Monitoring re-enabled with per-Eventstream control (Preview) Eventstream observability in Workspace Monitoring is back — now with granular control over which Eventstreams emit monitoring data. A new ‘Log Eventstream activity’ toggle in Eventstream Settings lets you enable or disable observability per Eventstream, so you can balance monitoring coverage with capacity consumption. When enabled, your Eventstream emits performance metrics, error counts, and health status to three tables in your Workspace Monitoring Eventhouse: EventStreamMetrics: throughput, backlog, and watermark delay EventStreamErrorMetrics: deserialization, conversion, and runtime error counts EventStreamNodeStatus: node health (Running / Failed) The toggle defaults to OFF for all Eventstreams. To get started, open any Eventstream, go to Settings, and turn on Log Eventstream activity. Your monitoring data will appear in the Workspace Monitoring database within minutes. Eventstream activity" within Monitoring. The panel highlights an active toggle switch, a description explaining that enabling this feature emits performance and error metrics to a monitoring database. To learn more, refer to the Monitor Eventstream data flows in Workspace Monitoring documentation. Eventstream UI editor improvements (Preview) We've redesigned key parts of the Eventstream editor to make building and troubleshooting faster and more intuitive. Always Publish: No more blocked publish buttons. Publish your work at any stage, the editor gives you clear, contextual guidance on what still needs attention instead of preventing you from moving forward. Inline error indicators: Errors now appear directly on the node that needs fixing, with actionable guidance on click. No more hunting through a detached error list to find what's broken. Operator and destination descriptions: Each option now includes an inline description explaining what it does, so you can build confidently without switching to docs. These changes reduce friction during authoring and make it easier to go from idea to running your pipelines. Secure Azure Event Hubs Connections in Eventstream with Workspace Identity (Preview) Bringing real time event data into Microsoft Fabric is now simpler and more secure with Azure Event Hubs integration for Eventstream. Organizations can connect Event Hubs directly to Eventstream and start routing event data to destinations such as Eventhouse for analytics and operational insights. A key capability is Workspace Identity, which removes the need to manage shared access keys. Instead, Eventstream can authenticate to Azure Event Hubs using the Fabric workspace identity. Administrators simply grant the workspace the Azure Event Hubs Data Receiver role, enabling secure access through Microsoft Entra based permissions. This approach improves security, simplifies credential management, and aligns with enterprise governance requirements. The integration supports both public and private network deployments. For Event Hubs hosted in private networks, organizations can connect through a streaming virtual network gateway while continuing to use Workspace Identity for authentication. For advanced event processing scenarios, users can enable schema support, associate schemas from the event schema registry, and route structured events to destinations such as Eventhouse. Combined with Workspace Identity, this provides a secure and scalable foundation for building real time data pipelines without the operational overhead of managing secrets or credentials. Schema Registry and Event SchemaSet Region Availability Previously, Schema Registry feature and the Event SchemaSet artifact was available for preview in 31 regions. Expanded to 10 additional regions: Geography Region Americas Central US Americas Mexico Central Americas West US 3 Europe Italy North Europe Poland Central Europe Spain Central Europe West Europe Asia Pacific Australia Southeast Asia Pacific Israel Central Asia Pacific Japan West If you were previously blocked from trying SchemaSets and schema-based Eventstream data ingestion, you can now do so in these regions. For more information on Event SchemaSets and how you can create and manage them, visit Schema Registry Overview. To learn more about configuring Eventstreams with schema-based sources, visit Use schemas in Eventstreams. Data-driven styling and UX improvements for Maps (Generally Available) Maps become most valuable when they help users understand not just where things are, but what the data means. Now, Data-Driven Styling, along with Markers Rotate by Data, Traffic Flow visualization, additional Map View options, and an improved Layer Settings experience. Together, these enhancements help organizations transform raw geospatial data into intuitive, actionable business insights. Let Your Data Tell the Story Understanding patterns hidden within geographic data can be challenging when every feature on a map looks the same. With Data-Driven Styling, Fabric Maps enables map builders to visually represent business data directly on the map, helping viewers quickly identify trends, hotspots, and outliers without inspecting individual records. The new Color by Value Range capability allows authors to style map layers using numeric measures such as revenue, utilization, sensor readings, environmental measurements, or operational KPIs. Instead of applying a single color to an entire layer, Fabric Maps visualizes value distributions chromatically, making important differences immediately visible. Organizations can choose between two visualization approaches: Gradient Styling uses continuous color transitions to reveal magnitude, trends, and geographic variation across a dataset. Step-Based Styling allows users to define custom value ranges with distinct colors, making it easy to visualize business thresholds, risk levels, performance bands, or service categories. To make these visualizations easier to interpret, Fabric Maps automatically generates corresponding data legends that explain how colors map to underlying values. This helps viewers understand the meaning behind the visualization and make decisions with greater confidence. Fabric Maps also includes thoughtfully designed color palettes, including options that support colorblind-friendly visualization scenarios, helping more users accurately interpret map-based insights. Combined with clear, automatically generated legends, these capabilities improve accessibility and make data-driven maps easier to understand across a wider range of audiences. per street and road and high-value exposure in flood-prone areas. Visualize Direction and Movement with Markers Rotate by Data Many operational scenarios involve not only location but also direction. Fabric Maps now supports Markers Rotate by Data, allowing marker symbols to automatically rotate based on values stored in a data column. This capability is available for Marker layers, enabling builders to visualize directional information directly on the map. Whether visualizing aircraft headings, vehicle movements, equipment orientation, or other operational workflows, map authors can represent direction without requiring custom visualization development. By transforming static points into directional indicators, organizations can add valuable operational context and communicate movement patterns more effectively. Add Real-World Context with Traffic Flow Visualization Location data alone doesn't always tell the full story. Fabric Maps now supports Traffic Flow overlays, allowing map builders to bring current traffic conditions into their existing map experiences. By combining business data with real-world traffic information, organizations can gain additional situational awareness for logistics operations, field service planning, transportation monitoring, and operational decision-making. The added context helps users better understand the environment surrounding their assets and activities without leaving the map experience. Configure the Right Map View for Your Audience Organizations often create maps for users across different regions and business contexts. Fabric Maps introduces additional Map View settings that allow map builders to configure how geographic information is presented, helping ensure maps are displayed in a way that aligns with organizational needs and audience expectations. This flexibility gives authors greater control over creating a consistent and intuitive viewing experience across a variety of business scenarios. A More Discoverable Authoring Experience UX research and user feedback showed that some key layer settings were difficult to discover. We updated the experience by moving geometry and visualization options—including visual type, Data-Driven Styling, and marker rotation—higher in the configuration pane. We also renamed General to Visibility, making the settings clearer and map authoring more intuitive. Turn Location Data into Business Insight Data-Driven Styling, built-in data legends, Markers Rotate by Data, Traffic Flow overlays, enhanced Map View options, and the improved Layer Settings experience help organizations transform location data into meaningful business insight. Together, these capabilities make it easier to uncover patterns, understand operational context, and communicate geospatial insights across teams. Start building with these capabilities today: explore the customization options, apply them to your own geospatial data, and create map experiences that turn location into action. To learn more, refer to the Customize a map in Microsoft Fabric documentation. Workspace Outbound Access Protection (OAP) for Operations Agent (Preview) Workspace Outbound Access Protection (OAP) in Microsoft Fabric helps admins secure outbound connections from workspace items to external resources. Admins can control outbound access by blocking unwanted connections by default and allowing only approved connections through configured rules. As organizations adopt AI-powered operations at scale, governance and security remain critical requirements. With this preview release, Microsoft Fabric introduces Outbound Access Protection (OAP) for Operations Agent, enabling workspace administrators to control the outbound actions an agent can perform. OAP applies workspace-level policies to actions such as Teams notifications, workflow triggers, and cross-workspace operations, helping organizations enforce security and compliance requirements while continuing to benefit from AI-driven automation. When OAP is enabled, Operations Agent continues to perform core functions including reasoning, recommendation generation, rule evaluation, and telemetry collection. However, outbound actions are governed by the workspace's configured access policies. Administrators gain greater visibility through in-product notifications, Teams messaging experiences, and the Operations Agent Activity Log, making it easier to identify and troubleshoot blocked actions What's new with Operations Agent and OAP? Govern outbound agent actions through workspace-level OAP policies. Control whether Operations Agent can send Teams notifications based on allowed connections. Prevent unauthorized cross-workspace actions when OAP policies restrict outbound access. Receive clear visibility when actions are blocked through in-product notifications and Teams messaging experiences. Monitor agent activity and OAP-related outcomes through the Operations Agent Activity Log. During the preview, some limitations apply. Cross-workspace actions are blocked when OAP is enabled. For example, Power Automate actions are not yet supported when OAP is applied, and only connectors that explicitly support OAP policies can be permitted. By extending Fabric's outbound governance framework to Operations Agent, organizations can adopt AI-powered operational automation with greater confidence while maintaining control over how and where agent-initiated actions are executed. Resources Workspace outbound access protection for operations agent (preview) Workspace Outbound Access Protection (OAP) Workspace Outbound Access Protection for Operations Agents Configure and manage Activator rules directly in Eventstream (Generally Available) You can now create and manage rules directly in Eventstream. Previously, setting up an alert required switching from Eventstream to Activator. While powerful, this meant moving between experiences to complete a single workflow. Now, alert creation is embedded directly into Eventstream. Capabilities for building or editing your Eventstream: Select the stream you want to monitor. Choose Set Alert. Define your condition (thresholds, aggregations, patterns). Configure the action. Create the rule. Capabilities for Activator destination created on Eventstream: Select Activator node Select Rule icon Create the rule Once you have the rule(s) created on your Eventstream, you can manage them by editing, deleting or opening in Activator. To learn more, refer to the Add a Fabric activator destination to an eventstream documentation. Data Factory Introducing Hierarchical Navigation in Monitoring Hub for Fabric Pipelines Modern data estates rarely consist of a single job running in isolation. Pipelines trigger notebooks, notebooks invoke other workloads, and business processes span multiple interconnected executions. When troubleshooting a failure or understanding lineage, customers often need visibility into how these executions relate to one another. Hierarchical Navigation in Monitoring Hub, is a new capability that helps you understand the relationships between runs and quickly navigate across upstream and downstream executions. With Hierarchical Navigation enabled, Monitoring Hub can display: Upstream runs that initiated a workload Downstream runs triggered by a workload Execution relationships across supported Fabric artifacts This provides a richer observability experience by helping you move beyond individual run monitoring and understand how your workloads operate together. This enhancement is another step toward a richer observability experience in Fabric, helping customers gain deeper insight into workload execution and dependencies at scale. To learn more, refer to the Hierarchical Navigation for Pipelines in Monitoring Hub documentation. Explore the modern Fabric Pipeline canvas (Preview) The new Fabric Pipeline canvas experience is designed to make pipeline authoring easier than ever. Key capabilities with modern canvas: Better visibility when navigating large pipeline graphs Cleaner, more structured layouts for complex orchestration logic Improved responsiveness when working with enterprise-scale workflows A more intuitive experience for pipelines with many activities and branches The modern Fabric Pipeline canvas, showing the updated node experience and option to disable the preview if needed. If you haven't tried it yet, now's the perfect time. The new experience is rolled out automatically and can be disabled at any time. Whether you're building your first pipeline or managing hundreds of activities across complex workflows, the new canvas is designed to help you stay productive and focused on what matters most. To learn more, refer to the Modern Pipeline Node Experience documentation. Upgrade Dataflow Gen1 to Dataflow Gen2 (CI/CD) using the Upgrade Wizard (Preview) The Dataflows Upgrade Wizard is a guided, end-to-end experience that upgrades your existing Power BI Dataflow Gen1 items to Dataflow Gen2 (CI/CD) in Microsoft Fabric with minimal effort. You can upgrade a single dataflow, or several dataflows from a workspace in a single flow. Previously, bringing a Gen1 dataflow into Fabric meant recreating it and repointing everything that depended on it. The Upgrade Wizard upgrades in place instead. Each dataflow keeps its ID, name, schedule, and connections, so the reports and semantic models that connect to it keep working without any changes. Before anything changes, the wizard assesses every dataflow in the workspace and tells you which ones need attention and why, such as incremental refresh settings to reconfigure or a linked entity to update, so you know what to expect before you upgrade. Why this matters Upgrade in place, with nothing to rebuild and nothing to repoint. Upgrade a whole workspace at once instead of one dataflow at a time. See what needs attention before you upgrade. Unlock Dataflow Gen2 innovations: improved performance, deeper Fabric integration, CI/CD and Git support, data destinations of your choice, richer diagnostics, Copilot-assisted authoring, and a modern data transformation foundation. The wizard is available for Dataflow Gen1 items in Premium or Fabric workspaces and requires Fabric to be enabled. To learn more, refer to the Upgrade Dataflow Gen1 to Dataflow Gen2 (CI/CD) using the Upgrade Wizard documentation. Extended watermark support in Copy job Watermark-based incremental load support enables Copy Job to efficiently ingest only new or changed data from key enterprise and SaaS data sources, avoiding costly full reloads. This reduces source-system impact, network traffic, and runtime while improving scalability for production analytics workloads. Customers can now use incremental loading across more of their critical data sources, including Salesforce, Informix, Cassandra, Greenplum, Presto, and Databricks, by leveraging Copy job’s built-in watermark mechanism and without building custom ingestion logic. To learn more, refer to the Incremental copy in Copy job documentation. Enable Change data feed during Lakehouse table creation in Copy job Copy job can now create Lakehouse tables with Change Data Feed (CDF) enabled automatically. There’s no longer a need to pre-create destination tables or manually configure Delta table properties. Simply select Enable CDF on destination and Copy job takes care of the setup for you. This ensures the table is immediately ready for incremental processing and downstream CDC scenarios, helping reduce data movement and improve replication efficiency. By eliminating manual configuration steps, it makes advanced data integration patterns much easier to adopt and operate at scale. To learn more, refer to the Automatic table creation and truncation on destination documentation. Amazon Redshift as new source in Copy job As part of our mission to enable multi-cloud data movement at petabyte scale with Copy job, we are bringing Amazon Redshift support as a source. This enables customers to seamlessly ingest data from one of AWS's most widely adopted data warehouse platforms directly into Fabric. Redshift support further strengthens Fabric's vision of delivering an open, connected, and multi-cloud data platform. To learn more, refer to the Connectors for Copy Job documentation. Copy job supports timestamps without time zone in Lakehouse Support for timestamps without time zones (timestamp_ntz) allows Fabric Lakehouse tables to preserve date and time values exactly as stored, without applying time zone conversions. Copy job can now automatically map timezone-independent datetime values to Delta Lake timestamp_ntz, ensuring greater compatibility with source systems that allow storing date and time values without time zone information. Migration Assistant for SQL database in Fabric (Generally Available) The guided, Fabric-native wizard takes you from a source SQL Server schema to a running SQL database in Fabric: upload a DACPAC, review compatibility results, deploy the schema with Copilot-assisted fix suggestions, and copy your data using built-in Fabric Copy Jobs. We've added the capability preview customers asked for most: Validate. You can now check out a DACPAC for compatibility before creating a SQL database in Fabric. Upload the file and the assistant reports which schema objects will deploy cleanly, which ones use features that aren't supported, the reason behind each failure, and the dependencies between objects — with nothing provisioned and no capacity consumed. That means you can scope migration effort, plan remediation, and get change-approval sign-off before you commit to a target database. To get started, select Migrate in your Fabric workspace and choose Migrate to SQL database in Fabric. To learn more, refer to the Fabric Migration Assistant documentation. Until next month That's a wrap for the August 2026 Microsoft Fabric Monthly Update. As always, we'll continue sharing new capabilities, enhancements, and improvements across Microsoft Fabric in future monthly updates. Thank you for being part of the Fabric community!Fabric March 2026 Feature Summary
Welcome to the Fabric March 2026 Feature Summary—and welcome to FabCon! As we kick off FabCon, this update captures the momentum we’re seeing across the Fabric platform and the conversations happening with customers and partners right now. March brings a wide range of enhancements across governance, data engineering, real-time intelligence, data science, extensibility, and AI—all designed to help teams build, operate, and scale end‑to‑end data solutions with confidence. Many of the capabilities highlighted here reflect direct feedback from the community and real‑world usage we’ve learned from—including insights shared leading up to (and during) FabCon. We are eager to share what’s new and to continue the conversation throughout the week. If you haven’t already, check out Arun Ulag’s hero blog “FabCon and SQLCon 2026: Unifying databases and Fabric on a single, complete platform” for a complete look at all of our FabCon and SQLCon announcements across both Fabric and our database offerings. Contents https://youtu.be/xhrSMNNX5ho?si=aTWiAMq0PvtTxej8 Events and Announcements Don’t miss the next Monthly Data Days Sessions On March 26 we have a special edition of Fabric Data Days featuring two topics. Join us at 8 AM Pacific for a session on getting started with Fabric IQ. Then at 3 PM Pacific we’ll discuss mapping and spatial analytics in Fabric. Register now! Couldn't make it to Atlanta or just want more FabCon + SQLCon? Join us in Barcelona this September. FabCon Europe is happening again in 2026. Mark your calendars for September 28 – October 1, 2026. Register now to access Super Early Bird pricing! Fabric Platform OneLake Catalog Govern for admins (Generally Available) In today’s data-driven world, effective data governance is crucial to ensure the integrity, security, and usability of data. OneLake catalog is available for Fabric admins, providing tools and insights to govern and secure data estates within Fabric in one place. Figure: OneLake Catalog: Govern for Admin view Figure: OneLake Catalog: Govern for admin—view more report. OneLake Catalog search API and MCP tool (Preview) OneLake Catalog’s Search API brings cross-workspace discovery to code. Instead of traversing workspace-by-workspace and “listing everything,” a single search request can locate matching items across your accessible estate based on catalog metadata and the user’s permissions. Search is designed to help even when the exact name isn’t known. Free-text matching includes the item’s display name and description, so a keyword you remember is often enough to find the right entry. Results can be filtered by the item's type to narrow down the scope of your search. The set of supported metadata signals and filters is expected to grow, enabling richer and more targeted discovery scenarios. The catalog search capability is also included as a built-in tool in the Fabric Core MCP server so AI agents can reliably locate the right Fabric asset as part of a broader workflow, then continue with follow‑up actions using other tools. Workspace tags (Generally Available) Fabric tags add meaningful metadata so people can find the right content faster and organize it consistently. That capability is available for workspaces. Workspace tags add shared context (like team, project, or cost center) at the workspace level, helping teams discover and manage workspaces more efficiently, while also enabling scalable governance through APIs. Figure: Workspace settings screen showing applied workspace tags Workspace tags are built on the existing Fabric tags model: tags are defined once, then applied on items and workspaces. Workspace admins can apply and remove tags in workspace settings, making it easy to add shared context at the workspace level. A workspace can have up to 10 tags applied. Workspace tags are surfaced in key discovery experiences so they’re usable in day-to-day navigation: Workspaces can be filtered by tags both in the workspaces list and in OneLake Catalog Explorer, a tags indicator also appears in the workspaces list and in OneLake Catalog Explorer next to every tagged workspace. Tag names are shown on the workspace screen itself, making the workspace context immediately visible. Tagging can be retrieved and managed at scale using APIs, enabling consistent application and reporting across workspaces. Data loss prevention policies for Fabric—Extending restrict access to structured data in OneLake (Preview) When handling sensitive data, it might be challenging to find the right balance between federating data and keeping it secure and compliant. Data Loss Prevention (DLP) policies enable organizations to detect sensitive data and surface it to users and admins when it is found. The Restrict Access action allows you to restrict access to your data once the sensitive information is detected. DLP Restrict Access reduces the risk of exposure to unauthorized users, without slowing analytics or collaboration. Customers can scale Fabric with confidence, meeting compliance requirements while enabling secure, enterprise-wide data sharing. With this release, you’ll be able to apply access restrictions through DLP on: Warehouses KQL databases SQL databases Lakehouses (previously supported) Semantic models (previously supported) Cosmos DB and mirrored databases are coming soon. Figure: Fabric warehouse with a restrict access indication Admins can ensure that sensitive data is protected consistently wherever it lives and however it is accessed. Learn more about restrict access in DLP. Lakehouse Signals in IRM (Generally Available) Microsoft Purview Insider Risk Management cross-references millions of signals across all your products, to create comprehensive profiles of potentially unethical behavior inside your organization. Using Lakehouse indicators in Insider Risk Management enables security teams to detect and investigate risky data activity in OneLake with greater precision and context. By incorporating Fabric Lakehouse signals directly into insider risk policies, security teams can correlate data access and movement with DLP, labeling, and audit signals in a single investigation experience—reducing blind spots and accelerating response to potential data exfiltration or misuse. This provides stronger protection for high‑value analytics data while maintaining built‑in privacy controls and avoiding the operational overhead of deploying separate monitoring tools. Figure: Lakehouse indicators used within the IRM tool Learn more about Fabric indicators in Insider Risk Management. Quick policy for data theft for Fabric (Generally Available) A new quick policy for the Data Theft rule is available for Fabric. This streamlined experience makes it easier to set up protection against data exfiltration scenarios, helping security teams take action faster when sensitive Fabric data is at risk. Learn more about IRM quick policies. Insider Risk Management PAYG Usage Report (Generally Available) The Microsoft Purview Insider Risk Management pay-as-you-go feature usage report is designed to provide transparency to customers, enabling more accurate budget planning and policy tuning. IRM admins can check the distribution of PAYG processing units billed across workloads (Fabric), sub-workloads (Power BI, Lakehouse), and indicators (downloading Power BI reports, etc.) to fine-tune their policies and plan PAYG budgets accordingly. Figure: Pay-as-you-go Usage Report Purview DSPM for AI for Fabric Copilots and data agents (Preview) As AI adoption accelerates, organizations need built‑in protections to keep data safe. With Purview Data Security Posture Management (DSPM) for AI, customers gain visibility and control over AI interactions. DSPM for AI helps teams spot sensitive data risks in AI prompts and responses, identify risky AI behavior, and apply consistent governance using familiar tools like DSPM, Insider Risk Management, Audit, and eDiscovery—so organizations can move faster with AI, without compromising security or compliance. bric_Copilot Figure: Purview DSPM for AI report showing Data Agent interaction in Fabric Learn more about DSPM for Fabric Copilots. Branched workspace with Git integration (Preview) Branched workspace is a new developer experience designed to simplify how teams work with feature workspaces during a branch‑out flow. With clearer visual cues and richer context, developers can easily understand workspace relationships and work more confidently when branching and iterating on features. This feature will be released by the end of March 2026. Figure: Fabric workspace tree showing the new relation between workspace and branched workspace Follow our new Git developer experiences in Microsoft Fabric (Preview) announcement. Selective branching with Git integration (Preview) Fabric Git Integration Branch-out with selective branching introduces a more focused branch‑out experience in Fabric. Developers can select only the items they need for a feature, reducing clutter in the target workspace, improving reliability, and accelerating time‑to‑code. By working with a smaller, purpose‑built workspace, developers can iterate faster and with greater confidence. Figure: Branch-out selective branching dialog Follow our new Git developer experiences in Microsoft Fabric (Preview) blog announcement. Compare code changes with Git integration (Preview) The new compare code changes experience helps developers confidently sync their Fabric workspace with a connected Git branch by clearly showing what changed before taking action. It provides a familiar code‑compare experience that highlights the exact differences since the last sync—down to the item and file level—whether the change originated in the workspace or in the repository. This makes it easier to review updates, understand their impact, and resolve conflicts by comparing workspace and Git versions side by side before committing, updating, or undoing changes. Figure: Git Integration code compares changes dialog Follow our new Git developer experiences in Microsoft Fabric (Preview) blog announcement. Connection reference item type in Variable Library (Preview) The new connection reference item type in Variable Library introduces a new way to manage external data connections in Microsoft Fabric. This new variable type lets you reference existing connections—such as Azure SQL or Snowflake—by storing a connection ID in the Variable Library, instead of embedding static connection strings in code. Figure: Variable Library “connection reference” item type option Connection reference variables work seamlessly with CI/CD and Git, enable safer environment‑specific configuration across dev, test, and prod, and ensure only authorized connections can be selected through the UI. This makes it easier to build, deploy, and manage Fabric solutions with cleaner configuration, stronger governance, and improved portability across CI/CD stages. Bulk import and export items definition APIs (Preview) These APIs enable you to programmatically export, import, and synchronize Fabric item definitions across workspaces at scale—all through the Fabric REST API. Every Fabric item—whether it’s a Notebook, Report, Semantic Model, Data Pipeline, or KQL Dashboard—has an underlying item definition: a portable schema containing the item’s full configuration and content (encoded in Base64). The Import & Export Batch APIs let you: Export item definitions individually or in bulk from any workspace Import (create) items from definitions into a target workspace Update existing item definitions in-place for continuous deployment List & paginate through all items in a workspace for batch operations Key scenarios Workspace migration: Moving items across workspaces, tenants, or regions is one of the most common requests from Fabric customers. The batch APIs let you export all items from a source workspace into a portable JSON manifest, then import them into any target workspace. This is invaluable for replicating environments across different tenants and cloning a production workspace for testing purposes. CI/CD and DevOps integration: To support enterprise DevOps practices in Microsoft Fabric, organizations can integrate the new Bulk Export and Import APIs into their CI/CD pipelines. Fabric item definitions can be treated as code—exported and versioned in Git using Fabric Git Integration or the bulk-export API, validated through pull request workflows, and promoted through a well-defined release process. When deploying across workspaces, the bulk-import API enables consistent, automated promotion into test and production environments using the underlying Fabric dependency logic that creates new items in the correct order, retains the original relations, and updates existing ones in place. Metadata backup and recovery: Schedule periodic batch exports to capture the full state of your workspace as versioned JSON manifests. Store them in Azure Blob Storage, a Git repository, or any durable storage. If something goes wrong, re-import the manifest to restore your workspace to a known-good state. Metadata scanning and lineage analysis: Tools that analyze report definitions to discover data lineage—such as which semantic model columns are used in each report—can extract hundreds of report definitions in bulk instead of one at a time, reducing scan time.Export (read) operationsMethod Endpoint Description POST /workspaces/{workspaceId}/items/bulkExportDefinitions?beta=true Export an item’s full definition as Base64-encoded parts. May return 202 for LRO. Import (write) operationsMethod Endpoint Description POST /workspaces/{workspaceId}/items/bulkImportDefinitions?beta=true Create/Update an existing item’s definition in-place. Ideal for CI/CD sync. Resources Full announcement: Bulk import/export items definition APIs API documentation: Fabric Items API Reference Comprehensive guide: Item Management Overview Item definition structure and formats: Item Definition Reference How to handle async operations with polling: Long-Running Operations Guide App registration and authentication setup: Microsoft Entra ID Documentation CI/CD tutorial using Bulk API: CI/CD tutorial using the Bulk Export and Import APIs Fabric CLI v1.5—Power BI Scenarios, CI/CD Deployments, and DX Improvements The Fabric CLI v1.5 is the most scenario-driven update yet. Power BI developers can now trigger semantic model refreshes, rebind reports, and script end-to-end deployment workflows—all from the terminal, without portal context-switching. The release also adds a new deploy command for CI/CD, interactive REPL mode, JMESPath filtering, notebook export in multiple formats, Python 3.13 support, and expanded coverage for Fabric items. Many of these improvements are community-contributed, making the CLI a comprehensive open-source automation surface for Fabric. CI/CD deployments from the CLI—deploy workspaces in One Command A new deploy command integrates the Fabric CI/CD Python library directly into the Fabric CLI, enabling full workspace deployments—including item rebinding and configuration—from a single command. Teams can run deployments from their terminal, GitHub Actions, or Azure DevOps pipelines. Combined with Service Principal authentication and federated credentials for GitHub OIDC, this enables zero-touch, Git-based promotion workflows that fit modern DevOps practices—no custom scripts or additional tools required. For usage examples, refer to the CI/CD examples and setup guide. Fabric CLI as an Execution Layer for AI Agents Fabric CLI is designed to work well with AI agents. A structured agent instructions file and a dedicated Fabric CLI Skill provide AI assistants like GitHub Copilot and Claude with the context they need to generate correct CLI commands from natural language. Improved error messages with actionable guidance help agents self-correct, and the interactive REPL mode enables persistent terminal sessions for multi-step agent workflows. Using a CLI as the execution layer for AI agents is an emerging industry pattern—instead of agents calling raw REST APIs (which require extensive token-heavy context about endpoints, auth, and payloads), agents issue concise CLI commands that encapsulate that complexity, making AI-driven Fabric automation more practical and reliable. Learn more with Fabric CLI agent docs and AI assets on GitHub. Fabric Remote MCP Server: AI agents operate directly in your Fabric environment Fabric Remote MCP is a cloud-hosted MCP server that allows AI agents to perform real operations in your Fabric environment—create workspaces, manage permissions, work with item definitions, and more. No local installation is required. Agents authenticate via Entra ID and operate within your existing RBAC boundaries, with every tool invocation recorded in audit logs. The preview launches with capabilities spanning workspace management, item CRUD and definitions, and permission management. It works with any MCP-compatible client, including GitHub Copilot, Cursor, and Claude Desktop. Learn more in this blog post: Introducing Fabric MCP (Preview). Fabric MCP AI code assistants (Generally Available) The Fabric Local MCP is an open-source MCP server that runs on your machine. This solution integrates AI coding assistants with the comprehensive Fabric API, offering OpenAPI specifications, best-practice guidelines, item definition schemas, and example payloads to enable agents to produce precise, production-ready code while minimizing errors. OneLake tools enable live file operations including upload, download, table inspection, and item creation. This update introduces integrated authentication, automatic retry, production SLAs, and telemetry. Install via npx Microsoft/fabric-mcp in any MCP-compatible client—it works with VS Code, Claude Desktop, Cursor, and more. Fabric Local MCP on GitHub.Fabric Fabric Extensibility Extensibility (Generally Available) After six months in preview mode, gathering feedback, resolving bugs, and strengthening the platform, we’ve reached the next milestone. Partners and customers can build, validate, and publish custom Fabric workloads to production with full Microsoft support. Key highlights: All core capabilities are stable and supported: OneLake storage, native item lifecycle, Entra token acquisition, iFrame relaxation, Workload Hub publishing. The Starter Kit ships with production-ready UI components (ItemEditor, WizardControl, OneLakeView, and more) that reduce time to first workload. GitHub Copilot integration and a new DevContainer/GitHub Codespaces setup reduce setup effort—no local machine required. The first Fabric Extensibility Community Contest drew strong community participation, with real workloads already appearing in the Workload Hub. Learn more about Fabric Extensibility (Generally Available). CI/CD & remote support (Preview) Three new features further enhance the professional development experience for Fabric workloads. CI/CD Support Workload items are first-class citizens in Fabric's CI/CD platform. Items participate in Git integration and Deployment Pipelines with no custom tooling. Variable Library support means items automatically pick up workspace-specific configuration (e.g., the right Lakehouse reference) when promoted across dev, test, and production—no hard-coded IDs, no manual reconfiguration. Figure: CICD enablement for Hello World Sample Variable Library Support Items can be read from Fabric's Variable Library, allowing workspace-specific configuration (e.g., the right Lakehouse reference) to resolve automatically when an item is promoted across dev, test, and production stages—no hard-coded IDs, no manual reconfiguration, and no custom deployment hook logic required. An opt-in webhook that fires whenever a workload item is created, updated, or deleted—regardless of whether it happened through the UI, the REST API, or a CI/CD pipeline. It’s designed for licensing checks, infrastructure provisioning, and external system synchronization. There’s no impact on workloads that don't register an endpoint. Figure: Variable Picker within Fabric Cloud Shell Item Remote lifecycle notification API Workloads are no longer just passive objects sitting in a workspace. The Remote Lifecycle Notification API is an opt-in capability—there is no requirement to use it. If your workload does not need backend notifications, you simply don't register an endpoint, and everything works exactly as before. Fabric Scheduler / Remote Jobs This feature allows workload items to expose named job types that users can schedule directly from Fabric. When a scheduled job fires, Fabric calls a registered endpoint on your workload backend—passing along the item context and a delegated user token. For all these features, you’ll find samples in the Toolkit Starter Kit. Learn more about Fabric Extensibility CI/CD and remote capabilities in this blog post. What's new in workload management As the Fabric extensibility ecosystem grows, with partners publishing workloads and organizations building custom solutions, managing workloads at scale demands more than a single settings page. IT admins need centralized governance and a clear overview of what's being used across the organization, and workspace teams need self-service agility. Three key workload management features for Microsoft Fabric Extensibility will launch by April 1, 2026: Workload admin portal (Generally available) Add workload to workspace (Generally available) Workload management admin APIs (Preview) These will enhance governance through portal, API, and self-service capabilities. Admin portal: centralized admin workload overview (Generally Available) The Fabric Admin Portal now includes a dedicated Manage Workloads tab, a single pane of glass for workload governance across your organization. Centralized workload visibility: view all workloads available for assignment in your tenant in a single centralized view, including status information and workload details. Tenant assignment controls: manage workload assignment at the tenant and workspace level. Add workload to workspace (Generally available) The workspace-level workload assignment was previously introduced in Preview. It allows workspace admins to add workloads directly to one or more workspaces. How it works (Workspace admins): Navigate to the Workload Hub from the left menu in Microsoft Fabric or from your workspace settings. Browse or search for the workload you want to add. Select "Add Workload" and select "To Workspace" from the dropdown. Select your workspaces: search, check the workspaces you want, and use "View more/less" to manage the list. Select "Add": the workload is immediately available in your selected workspaces. Workload Management Admin APIs: Overview and Control at Scale (Preview) Capabilities For Fabric admins who need a programmatic view of their workload landscape, the new Workload Management Admin APIs provide governance and oversight across the tenant through a REST interface. List all workloads: view all workloads available to be added in the tenant, and view which workloads were added. List all workload assignments in the tenant. Drill down into a specific workload and view where it was added (tenant, workspace, capacity). Manage workload assignments (add or remove) to capacities, workspaces, and tenant. Self-service workload publishing (Generally Available) A frequent question from ISV partners using the Microsoft Fabric Extensibility Toolkit is: "How do I get started publishing?" Key features Self-service workload publishing is expected to be generally available by the end of March 2026. ISV partners will be able to publish workloads directly to selected customer tenants for private preview without requiring a manual submission request. This can accelerate time to market and support faster iteration with customers. Self-Service Workload Publishing gives ISV partners full control over their private preview journey: Publish to up to 20 customer tenants: share your workload with selected customers for testing and validation, no Microsoft certification required. Workload name reservation: reserve your globally unique workload name (e.g., Contoso.DataQuality) to protect your brand identity before formal publication. Automated validation: your workload package is automatically validated against manifest schema, naming conventions, and security requirements at upload time. Seamless path to general availability: once validated with customers, use the same workload package to pursue formal certification and publish to the global Workload Hub. OneLake Third-party support for OneLake security This month, we announced third‑party support for OneLake security, taking an important step toward interoperable data security. As customers increasingly build lake‑first architectures on open formats like Delta and Iceberg, they expect the freedom to use multiple analytics engines without copying data or redefining security. OneLake security addresses this need by enabling security to be defined once and enforced consistently wherever data is accessed. At the core of this capability is the authorized engine model. Security policies—including role‑based permissions, row‑level security (RLS), and column‑level security (CLS)—are centrally defined and managed in OneLake, while enforcement happens at query time inside the engine reading the data. Authorized third‑party engines securely retrieve the relevant metadata and effective security definitions through OneLake APIs and apply them during query execution. This ensures users see only the rows and columns they are permitted to access, while OneLake remains the single source of truth for access control. To support adoption, we’ve published implementation guidance and setup documentation for both engine builders and users. The APIs are designed to be engine-agnostic and easy to integrate by providing pre-computed effective access definitions. Looking ahead, we’ll continue evolving OneLake security APIs, including adding support for bitmap-based RLS enforcement. With this release, data vendors can integrate directly with OneLake security, customers can maintain a single security model, and users gain the flexibility to query OneLake data using the engines of their choice. OneLake file explorer (Generally Available) You can easily access and organize all your OneLake data from Windows using the OneLake file explorer. The file explorer lets you browse every workspace and data asset, and upload, download, or edit these files using the same familiar experience as OneDrive. By bringing data lakes into the Windows file system, the file explorer makes enterprise data more accessible for business users. Data Engineering Fabric Runtime 2.0 (Preview) Fabric Runtime 2.0 (Preview) is a next-generation runtime that is purpose-built for large-scale data computations in Microsoft Fabric and introduces key features and components that enable scalable analytics and advanced workloads. Apache Spark: 4.0 Components include Operating System: Azure Linux 3.0 (Mariner 3.0) Java: 21 Scala: 2.13 Python: 3.12 Delta Lake: 4.0 This screenshot demonstrates how you can switch to Runtime 2.0 at the Workspace settings and the Environment levels. Figure: Change runtime at the workspace settings level Explore the full documentation and start using Runtime 2.0 in Fabric. Custom Live Pools for Fabric Data Engineering Modern data engineering workloads are rarely one‑size‑fits‑all. Teams often need predictable performance, isolated resources, or customized configurations for critical production pipelines and high‑value interactive development. At the same time, Spark session startup times can degrade in real-world enterprise environments, especially when users have custom library dependencies. Workspaces or tenants are secured with Private Links or Managed Private Endpoints. In these scenarios, Spark clusters must be created on demand within strict network boundaries, and libraries need to be resolved and installed dynamically, adding noticeable startup latency. Custom Live Pools address this challenge by introducing dedicated, long‑lived Spark pools that stay warm inside your network boundary and come preconfigured with the required dependencies. With Custom Live Pools, Fabric Data Engineering now enables you to: Create dedicated Spark pools and schedule them tailored to your workload needs. Reduce session startup overhead by keeping pools warm with libraries preinstalled. Run reliably within Managed VNets and Private Link–enabled environments. ompute_configuration_panel_within_a_data_analytics_platfo Figure: Animated GIF demonstrating the setup of custom live pools in an Environment Because these pools are already provisioned within the workspace’s network boundary and fully initialized with dependencies, users can start working immediately, without paying the repeated cost of cluster spin‑up and library installation. Custom Live Pools are ideal for: Production pipelines that require consistent SLAs. High‑value interactive notebooks used by data developers. Teams operating in secure or regulated environments. How to set up a Custom Live Pool: Navigate to your Compute tab in your Environment. Select Spark pool and enable the option for Live Pool. Specify the Schedule, Time period of inactivity, and Retrigger frequency. Job concurrency and queue monitoring experience for Fabric Data Engineering As organizations scale their Fabric usage, understanding what’s running, what’s queued, and why becomes essential. The new job concurrency and queue monitoring experience delivers deep visibility into Spark workload execution across your environment. View active, queued, and completed jobs in a single place. Understand why jobs are queued and how concurrency limits are applied. Identify bottlenecks caused by capacity or concurrency constraints. Make informed decisions to tune workload scheduling and resource allocation. Figure: GIF demonstrating the new job concurrency and queue monitoring view in the Data Engineering/Science Spark settings page of Workspace settings Accessing workspace monitoring To view concurrency and queue signals for your specific workspace: Navigate to Workspace settings. Select Data Engineering/Science > Spark settings. Select Jobs to view the live view of your workspace level Spark queue and concurrency. Resource Profiles for Fabric Data Engineering Modern data engineering teams shouldn’t need to be Spark experts to get great performance. With Resource Profiles in Fabric Data Engineering, users simply describe what they’re trying to do, and Fabric automatically recommends the optimal compute configuration. Figure: GIF demonstrating the new resource profiles experience in workspace settings Simple inputs, smart recommendations Instead of tuning dozens of Spark settings, users provide a few high‑level workload details through an intuitive UI: Primary use case, such as a specific medallion layer (Bronze, Silver, or Gold) or task‑based optimization (read‑heavy or write‑heavy workloads). Typical data volume. Data characteristics, such as whether input data contains many small files. Maximum capacity units (CU) for the Spark pool. Once these inputs are provided, users select Get recommendation, and Fabric automatically generates an optimized configuration tailored to that workload. Based on the inputs shown above, Fabric recommends: The appropriate Resource profile. Node family and size. Autoscale and dynamic executor settings. Optimized Spark driver and executor cores and memory. A compatible runtime version. All recommendations are derived from proven best practices and internal performance tuning, removing guesswork and trial‑and‑error. Where to configure Users can enable and manage Resource Profiles from workspace settings: Go to Workspace settings > Data Engineering and Data Science > Resource optimization. Select or edit the optimized profile for the workspace. Rerun the Optimize for your use case flow as workloads evolve. Apply consistent configurations across all Spark workloads in the workspace. Once configured, all notebooks and pipeline‑triggered Spark jobs inherit these optimized settings automatically, without requiring per‑notebook configuration. Why this matters This experience enables: Performance by default: optimized compute without manual tuning. Consistency: the same performance characteristics across users and jobs. Better price‑performance: right‑sized resources aligned to workload intent. Lower operational overhead: fewer tuning cycles and support escalations. As workloads change over time, teams can simply revisit the optimization flow, update a few inputs, and let Fabric adapt the configuration—without rewriting code or Spark settings. Figure: Introduction to Resource Profiles Experience Figure: Recommendations generated based on user inputs To learn more about the Resource Profiles experience in Microsoft Fabric Data Engineering, refer to the Microsoft Learn documentation. Installing libraries with Quick mode in Spark Environment (Preview) Managing libraries shouldn’t slow down your development workflow. In Microsoft Fabric Environments, we’re introducing a more efficient way to iterate on libraries while keeping production workloads stable and reliable. Fabric Environments now support two complementary library installation modes that you can use side by side: Quick mode: a fast, on-demand installation path designed for development and experimentation, where libraries are installed when a notebook runs. This avoids heavy processing during the environment publishing and significantly reduces publish time and notebook startup latency when you’re iterating on lightweight or frequently changing dependencies. Full mode: a snapshot‑based installation path optimized for production workloads and pipelines, where libraries are fully resolved, validated against the Spark runtime, and published as a stable snapshot to ensure consistency and reproducibility. Figure: Add libraries in Quick mode and Full mode This new feature lets you move faster during development without compromising production stability. You can keep your core, production‑ready libraries in the snapshot‑based mode, while using the on‑demand path to quickly test new packages or iterate on custom libraries, all within the same Environment. Dynamic session sharing limit up to 50 for high concurrency Fabric High Concurrency Spark sessions enable both interactive exploration and large‑scale, pipeline‑driven notebook execution, supporting parallel, scheduled, and event‑driven workloads at enterprise scale. Customers often achieve higher density by packing notebooks into a shared High Concurrency (HC) session using session tags, effectively fitting up to five notebooks per session to control startup overhead and cost. While effective, this approach relies on static limits and manual tuning. With this update, Fabric Data Engineering allows the maximum number of notebooks attached to a High Concurrency session to be increased up to 50, enabling dynamic session sharing at much higher scale. Where to set the configuration You can set the configuration in the Environment item that your notebooks or pipeline‑triggered notebooks use: Go to Workspace → Environments Select the Environment attached to your notebook or pipeline Open Spark Properties Add the High Concurrency configuration Set spark.highConcurrency.max to a value between 2 - 50 Note: This update does not change the default limit of five. This enables: Interactive notebooks, used for exploratory analysis and collaboration. Notebook jobs triggered by pipelines, running in parallel within shared HC sessions. Dynamic adjustment of session sharing limits based on workload intensity, cost, and price‑performance goals. By increasing the session sharing limit, customers can: Improve session acquisition times during peak load. Increase notebook density without fragmenting sessions. Tune concurrency to match workload demand rather than fixed defaults. Achieve better price‑performance efficiency while preserving isolation and fairness across jobs. To learn more about increasing your session sharing limit in High Concurrency mode, please check out Microsoft Learn documentation. Data export settings for notebooks With data export settings for notebooks, Microsoft Fabric empowers administrators with explicit, tenant-level control over how data leaves notebooks. This feature helps ensure that interactive analytics do not inadvertently become channels for data exfiltration. Administrators can restrict the downloading of notebooks, preventing files that may contain sensitive data, credentials, or proprietary logic from leaving the environment. Additionally, they can disable downloads of rich output content, such as table results generated from DataFrames, within the notebook experience. By managing these controls, Fabric admins can effectively prevent unintended data exfiltration from interactive notebook workflows and consistently enforce security and compliance policies across all workspaces and teams. Figure: New data export tenant setting for notebooks Figure: New data export tenant setting enabled What users experience when downloads are blocked When an administrator blocks data export: The Download option is removed from the notebook UI. Users can no longer download notebook files or rich output content generated from DataFrames in the notebook experience. Interactive exploration continues in‑place, but data cannot be extracted outside Fabric through the notebook UI. Figure: Notebook with download controls disabled due to tenant-level enforcement This ensures that notebooks remain a secure analysis surface, rather than a data export mechanism, without disrupting day‑to‑day exploration inside the platform. Why this matters Notebooks often contain more than just code: Embedded datasets Derived analytical results Business logic Confidential insights By controlling export behavior at the platform level, Fabric helps organizations: Reduce risk of accidental data leakage. Meet regulatory and audit requirements. Standardize governance across teams and regions. Data Export Settings for Notebooks reinforce Fabric’s commitment to secure‑by‑default analytics, enabling powerful interactive experiences without compromising enterprise security posture. Session starts insights into Fabric Data Engineering Fast session startup is critical for interactive analytics, and Fabric’s Starter Pools are designed to deliver Spark sessions in ~5 seconds by default. However, when that target isn’t met, users have historically had little visibility into why. Session Start Insights closes that gap by making session acquisition transparent, debuggable, and actionable. Why sessions don’t always start in five seconds In practice, session startup delays are almost always driven by user‑side configurations, not platform regressions. Common causes include: Custom compute configurations that prevent reuse of pre‑warmed Starter Pools. Pre‑installed libraries or environment dependencies that require cluster customization. Managed VNets or private networking that force isolated cluster provisioning. Unexpected high regional demand triggering fallback to on‑demand clusters. What Session Start Insights delivers Previously, users could see that a session was “starting,” but not what was happening under the hood. With this feature, Fabric surfaces clear, explicit reasons for session startup behavior directly in the product experience: Whether the session was served from a Starter Pool or required an on‑demand cluster. The exact reason a fast‑path session could not be used (for example, libraries, networking, or custom configs). Where time was spent during session acquisition. Using the session detail view to diagnose delays Navigate to the notebook’s session status or monitoring pane. Open Session Details for the active or recent session. Figure: Notebook with Session Details option Review the delay reason and session source (Starter Pool vs. on‑demand) Figure: Notebook with Session Details pane showing session start details This makes it immediately clear whether the delay was: Expected due to configuration choices Related to libraries or networking Learn more about session start insights in the Microsoft Learn documentation. Z-order and liquid clustering support in the Native Execution Engine With the Native Execution Engine, Fabric Data Engineering continues to raise the bar on price‑performance leadership for large‑scale analytics. Beyond execution‑time optimizations, the engine now includes native support for Z‑Order and Liquid Clustering, allowing advanced data layout techniques to fully benefit from vectorized, C++‑based execution paths. This ensures that storage‑level optimizations and execution‑level acceleration work together, delivering compounding performance gains for real‑world analytical workloads. Why this matters Modern analytical queries frequently: Filter on multiple high‑cardinality columns. Scan large Delta tables repeatedly. Rely on selective predicates to narrow down results. Without intelligent data layout, even a highly optimized execution engine can spend unnecessary time scanning data. By combining the Native Execution Engine with Z‑Order and Liquid Clustering, Fabric ensures that: Related data is co located on disk, enabling aggressive file and row‑group skipping. Queries scan fewer files and fewer bytes. CPU‑efficient native operators are paired with I/O‑efficient data access. On a one‑billion‑row dataset, internal benchmarks comparing fallback execution versus Native Execution Engine with clustering showed: 20–32 seconds absolute runtime reduction per query. Roughly 20%–27% improvement across multiple clustered column combinations. Performance gains observed consistently across different predicate shapes and data distributions. This brings a compounding performance effect: faster scans, fewer CPU cycles, and lower cost per query, without requiring users to rewrite Spark code or change query semantics. This helps deliver strong price-performance for analytics workloads. How users enable and use this 1. Enable the Native Execution Engine Users must first ensure that the Native Execution Engine is enabled for their Spark workloads (at the workspace, environment, or session level). Once enabled, supported Delta operations automatically run through native execution paths. 2. Use Z‑Order or Liquid Clustering on Delta tables Users can apply clustering using standard Delta Lake commands: Define Liquid Clustering at table creation or apply it to existing unpartitioned tables Use OPTIMIZE … ZORDER BY for multi‑column access patterns To learn more about the Z-Order and Liquid Clustering support or Native engine, refer to the Microsoft Learn documentation. Copilot for data engineering and data science Microsoft Fabric notebooks now include a context-aware Copilot experience designed to support you across the full notebook lifecycle. By automatically understanding your workspace environment—including attached Lakehouses, notebook structure, and runtime behavior—Copilot provides assistance that stays aligned with how your notebook is built and executed. It’s easy to get started with no session startup required. Choose the Copilot icon on the toolbar to open the chat panel. Copilot can help accelerate notebook development by generating and refining code, explaining unfamiliar logic, and assisting with larger notebook workflows. For more complex tasks, Copilot can first propose a plan and then help implement it across the notebook, allowing you to move from idea to working solution more quickly. Copilot also improves the troubleshooting experience when notebook executions fail. Instead of navigating long stack traces or ambiguous error messages, you can use Copilot to analyze failures, identify likely root causes, and review suggested fixes directly within the notebook. Figure: Fix with Copilot provides error summary and suggested fixes Throughout this process, built-in guardrails ensure you remain in control. Copilot suggestions are transparent, and proposed code changes can be reviewed before being applied. Together, these capabilities help teams reduce development friction, resolve issues faster, and build more reliable data workflows. Try the new Copilot experience today. To learn more, visit the Copilot for Data Engineering and Data Science documentation. Fabric notebook custom agent inside VS Code The Fabric notebook custom agent is a Fabric-native AI development agent embedded in the Fabric Data Engineering VS Code extension. It helps data engineers build, debug, and publish Microsoft Fabric notebooks and Spark workloads. Unlike generic coding assistants, this agent operates with full awareness of the Microsoft Fabric workspace, runtime, environments, and Lakehouse resources. It ensures every action—code generation, execution, artifact management, and publishing—is context-aware, validated, and safe for enterprise environments. Prior to the introduction of the Fabric Notebook custom agent within the VS Code extension, there were notable limitations in how language models understood and interacted with the Microsoft Fabric environment. For instance, when users provided a prompt such as "read the parquet file from the current default Lakehouse and save it to a delta table," the language model was unable to interpret what was meant by "default Lakehouse." As a result, it would generate standard Spark code without leveraging the built-in spark variable available within the notebook, which is essential for initializing and managing Spark sessions in the Fabric environment. With this new agent, the following code will be generated and ready to run. # Read parquet file from default lakehouse df = spark.read.parquet("Files/green_tripdata_2022-08.parquet") # Write to delta table in dbo schema df.write.mode("overwrite").format("delta").saveAsTable("dbo.raw_green_tripdata_202208") This custom agent should be automatically activated once the Notebook is open. Figure: Fabric notebook custom agent For more detail, refer to the Author notebook inside VS Code documentation. Tenant switching inside Fabric Data Engineering VS Code extension ISV and partners often collaborate with multiple end customers, each typically operating within their own dedicated Microsoft Fabric tenant. To address this need for flexibility, the Fabric Data Engineering VS Code extension now enables tenant switching. With this enhancement, ISVs and partners can easily transition between different customer projects within the same VS Code window, eliminating the need for repeated sign-in processes. This streamlined experience simplifies managing multiple projects and improves overall productivity for professionals working across diverse customer environments. To switch to a different tenant, select the currently signed-in Fabric user in the status bar and pick the target tenant from the list. Figure: Switch Fabric tenant inside VS Code Enable new kernels inside Fabric Data Engineering VS Code extension Users can now run Fabric notebooks within VS Code using a variety of new kernels. Previously, running notebooks required users to specify the language of each cell using cell magic commands and rely on PySpark as the execution environment. With this enhancement, three additional kernels have been introduced, allowing users to select their preferred programming language directly at the kernel level. This eliminates the need for cell magic commands and streamlines the process, enabling notebooks to be executed in Python, Scala, or Spark SQL natively within VS Code. Choose Microsoft Fabric Runtime from the top-level kernel list. The available languages then appear in the second panel. Figure: Microsoft Fabric Runtime entry Figure: Supported Fabric notebook languages in VS Code For more detail, please refer to the documentation Author notebook inside VS Code. Support for multiple schedules in Fabric materialized lake views MLVs now support multiple named schedules per lakehouse. Previously, all MLVs shared a single schedule, and teams needing different refresh timings resorted to notebook-triggered refreshes. This workaround bypasses dependency management, centralized error reporting, and retry logic; failures can persist for weeks undetected. Each named schedule now targets a specific subset of views. A finance pipeline can refresh hourly while an analytics pipeline runs every six hours, with no scripting required. When a schedule fires, Fabric refreshes upstream dependencies in order, runs independent views in parallel, surfaces errors centrally, and skips overlapping runs. Figure: Schedules panel for materialized lake views, showing configured schedules and available actions For more information, refer to the Schedule a materialized lake view run documentation. PySpark support for Fabric materialized lake views (Preview) MLVs now support PySpark authoring (Preview), letting data engineers create, refresh, and replace MLVs from Fabric notebooks using the DataFrameWriter API. Previously, teams wrote definitions in Spark SQL, which made custom cleansing logic, UDFs for business rules, and procedural transformations harder to express. With PySpark authoring, MLVs gain access to the entire Python ecosystem. A gold-layer MLV can score transactions against a fraud detection model, standardize addresses using a geocoding library, or validate records against external regulatory rules. All existing MLV capabilities, including data quality constraints, table properties, and scheduled refreshes, work identically with PySpark-authored definitions. Full refresh only today; optimal refresh is coming soon. For more information, refer to the PySpark reference for materialized lake views (Preview) documentation. Move data from source to Lakehouse in a few moves using Copy job Getting data into your Lakehouse should be straightforward. For many customers, the first interaction with Microsoft Fabric begins right after creating a Lakehouse and selecting Get data. With this update, Copy job appears at the top of the Get data experience in Lakehouse, making it a more discoverable way to bring data into Fabric. Whether you’re onboarding your first dataset or scaling ingestion across multiple sources, Copy job can help you move data with minimal setup so you can focus on insights instead of configuration. Fabric notebooks now support lakehouse auto‑binding when used with Git, making notebooks far more portable across environments such as dev, test, and prod. Instead of hard‑binding a notebook to a specific lakehouse, Fabric automatically resolves the correct lakehouse as the notebook moves across Git‑connected workspaces, reducing manual rebinding and environment‑specific fixes. This feature is opt‑in and must be enabled from the notebook settings page. Once enabled, it applies to all lakehouses referenced in the notebook, including the default and any additional lakehouses. The configuration is stored in a system‑managed notebook-settings .JSON file in the Git repo, which should not be edited manually. Overall, lakehouse auto‑binding helps teams focus on versioning notebook logic while keeping data and environment configuration cleanly separated. Try it out in just a few steps: Create or open a Lakehouse. Select the Get data dropdown in the ribbon. Figure: Start ingesting data into a Lakehouse directly from the Get data dropdown using Copy job Select New Copy Job You’ll be redirected to the Copy Job experience, where you can choose the source data you want to ingest from. In just a few clicks, your data is copied into the Lakehouse and ready for exploration, analysis, and downstream analytics. Learn more: What is Copy Job in Data Factory – Microsoft Fabric Notebook supports Lakehouses auto-binding in Git Fabric notebooks now support lakehouse auto-binding when used with Git flow, making notebooks more portable across environments such as dev, test, and prod. Instead of hard-binding a notebook to a specific lakehouse in the original workspace, auto-binding lets the notebook automatically resolve the linked lakehouse as it moves across Git-connected workspaces. This reduces manual rebinding and environment-specific fixes. This feature is opt‑in and must be enabled from the notebook settings page. Once enabled, it applies to all lakehouses referenced in the notebook, including the default and any additional lakehouses. Figure: Entry of auto-binding setting in notebook The configuration is stored in a system‑managed ‘notebook-settings .json’ file in the Git repo. Overall, lakehouse auto‑binding helps teams focus on versioning notebook logic while keeping data and environment configuration cleanly managed. Notebook Resources Folder Support in Git Notebook projects often depend on more than just notebook code—such as reusable Python modules, configuration files, or small supporting assets. Fabric notebooks now support committing the built‑in Resources folder to Git, enabling true end‑to‑end source control for notebook‑based projects. These resources are versioned alongside the notebook and automatically restored during Git sync. To support real‑world workflows, this feature includes fine‑grained controls. Teams can define Git exclusion rules or use standard .gitignore files inside the Resources built in folder to avoid tracking large files, temporary assets, generated outputs, or test data. Figure: Define resources in git settings in notebook The feature is disabled by default to ensure safe adoption and does not introduce noticeable performance impact during commit or sync. The support for Environment resources folder, deployment pipelines, and public APIs is coming soon. Learn more: Notebook source control and deployment - Microsoft Fabric Fabric notebook public APIs (Generally Available) Fabric Notebook Public APIs enable notebooks to be managed and executed programmatically as first‑class assets. The APIs provide full CRUD support—enabling teams to create, update, list, and delete notebooks at scale—making them ideal for CI/CD and automated environment management. In addition, notebooks can be executed on demand via the Job Scheduler API. You can parameterize notebook runs, customize session configuration, specify environments and lakehouses, monitor execution status, and cancel runs if needed. Secure service principal authentication is also supported. A key enhancement is the ability for notebook runs to return exit values, enabling conditional branching and richer orchestration in pipelines. Together, these APIs unlock seamless integration with Fabric pipelines, external schedulers, and enterprise automation platforms. Learn more: Items - REST API (Core) and Job Scheduler - REST API (Core). Improved Copilot completion for Fabric notebooks We’re introducing upgraded Copilot completion in Fabric notebooks to deliver a faster, more accurate, and more intuitive coding experience. With this update, auto-completion is closer to what developers expect from VS Code‑style inline suggestions, helping you stay in flow while writing notebook code.You can enable the feature from the Copilot completion button in the notebook status bar. It supports both Python and PySpark notebooks. Figure: How to enable copilot completion A More Natural, Inline Coding Experience The upgraded auto‑completion is designed to work inline as you type, offering context‑aware code suggestions that better match your intent. Whether you’re writing Python logic, data transformations, or helper functions, Copilot now provides suggestions that feel more predictable, relevant, and easy to accept—reducing friction compared to earlier experiences. Faster and More Responsive Performance has been a key focus of this upgrade. Auto‑completion now responds more quickly, reducing latency between keystrokes and suggestions. This makes Copilot feel less intrusive and more like a natural extension of the editor, especially during rapid iteration or exploratory development. Higher‑Quality Suggestions That Fit Notebook Workflows Beyond speed, the quality of suggestions has improved. Copilot is better at understanding notebook context, including surrounding cells and in‑progress code, resulting in completions that require less manual editing. The goal is simple: help you write correct, readable code with fewer interruptions and less back‑and‑forth. Designed for Everyday Notebook Development This upgraded auto‑completion brings Fabric notebooks closer to the editing experience developers are already familiar with, while remaining optimized for data engineering and analytics workflows. Learn more by exploring Develop, execute, and manage notebooks - Microsoft Fabric. Create files in the notebook resources folder Fabric notebooks now let you create and manage files directly in the built‑in Resources folder, making it easier to develop and maintain notebook dependencies. You can create and edit Python modules, configuration files, and other lightweight assets alongside your notebook code and use them directly within the notebook. Figure: Entry of creating new file in notebook resources folder To learn more, refer to How to use notebooks - Microsoft Fabric. Data Science and AI Fabric data agents (Generally Available) Data sources: Build and consume data agents on a broad set of data sources, including Lakehouse, Warehouse, semantic models, Eventhouse, SQL databases, and mirrored databases. Configurations: Configure data agents using agent-level instructions, data source–specific instructions, and example queries to tailor behavior to your scenarios. Publish and share: Publishing and sharing data agents within Microsoft Fabric is generally available, making it easier to operationalize and collaborate on data agents. Figure: End-to-end data management workflow from creation through consumption This release also includes diagnostic, Git integration, and deployment pipelines as part of Microsoft Fabric’s Application Lifecycle Management (ALM) capabilities, enabling troubleshooting and lifecycle management of your agents! Advanced security and governance in data agents (Preview) Data agents in Microsoft Fabric (Preview) include capabilities that elevate security and governance standards. Through integration with Microsoft Purview, organizations gain access to comprehensive auditing, eDiscovery, data lifecycle management, communications compliance, and classification. These tools capture prompt and response telemetry along with user context, supporting enterprise protection and regulatory compliance. Additionally, we are introducing outbound access protection support for the Data Agent artifact to help mitigate sensitive data exfiltration risks and adhere to strict security policies at the individual workspace level. With these updates, organizations can monitor, control, and safeguard all data agent interactions more effectively. Data source enhancements for data agents (Preview) The latest preview brings significant enhancements to data agent source capabilities in Microsoft Fabric. Users can now connect Graph as a data source, enabling them to model and analyze complex relationships within their data for richer, AI-driven insights. Additionally, support for KQL user-defined functions (UDFs) and SQL functions is available, allowing for more sophisticated and efficient querying in KQL- and SQL-enabled sources. These enhancements make data agents more flexible and powerful, supporting faster analytics and expanded scenario coverage. Insider Risk Management PAYG Usage Report (Generally Available) Multimodal support for AI functions in Fabric enables notebook users to apply AI capabilities directly to their unstructured data—including PDFs, images, and text files. With just a few lines of code, users can perform tasks such as summarization, classification, sentiment analysis, and more, all within their existing workflows. This capability is designed to work across both pandas and Spark, making it easy to bring AI-driven insights to a wide range of data science and analytics scenarios in Fabric. Figure: Load files into a table or classify insurance claim images with multimodal AI functions Check out the multimodal AI functions documentation to learn more. AutoML in Fabric (Generally Available) AutoML delivers a fully production-ready, low-code machine learning experience. In addition to the core AutoML capabilities powered by FLAML—automated model selection, feature engineering, and hyperparameter optimization—the end-to-end UI experience, making it easy to configure experiments, monitor training progress, compare models, and deploy the best performer directly from the interface. With integrated experiment tracking, reproducibility, and seamless deployment workflows, teams can confidently move from raw data to high-quality predictive models faster—while maintaining transparency, governance, and control within Fabric. Figure: AutoML includes the fully integrated UI experience for configuring experiments, tracking model performance, and deploying models end to end Check out the AutoML in Fabric documentation to learn more. Data Warehouse Fabric Data Warehouse recovery (Preview) In fast moving production environments, an item gets dropped accidentally due to an incorrect script. Suddenly, critical reports are broken, and teams are asking the same question: “How fast can we recover?” With dropped warehouse recovery in Microsoft Fabric, a deleted warehouse no longer means starting over. You can now restore a dropped warehouse together with everything that goes with it—data, schemas, snapshots, permissions, and saved queries—in minutes, without rebuilds, re‑ingestion, or complex restore workflows. Figure: Warehouse recovery in action—from drop to restore in minutes There’s no need to recreate environments, rerun pipelines, or scramble through backups. Recovery is simple, predictable, and designed to bring your warehouse back exactly as it was before the drop. This capability is built for the realities of modern analytics: rapid iteration, frequent deployments, and shared production environments. Instead of turning accidental deletes into prolonged outages, Fabric makes recovery a routine, low-stress operation. No panic. No need to rebuild. Just built in resilience—designed for real-world production analytics. To learn more, refer to the manage workspaces documentation. Alerts and actions Microsoft Fabric Data Warehouse provides operational intelligence closer to the data by integrating SQL queries with Fabric Activator rules. Traditionally, identifying an issue in query results is only the first step. Teams then need separate systems or manually follow up to notify the right people and act. With this integration, Data Warehouse makes it possible to define rules directly from SQL query outputs, so changes in data can trigger alerts and downstream actions automatically. This unlocks a simpler way to monitor business-critical conditions using familiar SQL workflows. Teams can create queries that detect scenarios such as SLA risks, failed processes, unusual trends, or threshold breaches, then attach rules that evaluate results continuously and respond in real time. The result is a more proactive analytics experience, where insights move beyond the warehouse and are acted on immediately. Figure: Create rules on SQL query results to detect data issues, monitor KPIs, and automatically trigger alerts or Fabric workflows. Analyze unstructured text using T-SQL AI functions (Preview) Microsoft Fabric Data Warehouse extends modern analytics beyond structured and semi‑structured data by introducing built‑in AI functions for working directly with the unstructured text. Traditionally, processing free‑form content such as notes, logs, or comments requires external services or complex pipelines. With these new capabilities, Fabric Data Warehouse enables text extraction, classification, sentiment analysis, and transformation directly in T‑SQL language, allowing data engineers and analysts to keep AI‑driven text processing inside the warehouse. The new AI functions simplify common text analytics scenarios using familiar SQL patterns. You can extract structured insights from unstructured text, analyze sentiment in feedback or messages, and classify content such as application logs or incident reports using contextual understanding rather than fragile rules or expressions. Fabric Data Warehouse also supports text transformation scenarios, including summarization, grammar correction, and translation, making it easier to standardize and enrich text data as part of existing data preparation workflows. The following visual is example of processing unstructured text in the Comments table: Figure: Analyzing comment text with AI functions This query enriches each user's comment by determining its sentiment; labeling the type of feedback or intent; and extracting key discussion signals such as the main topic, user intent, and any requested action using built-in AI functions. For advanced scenarios, Fabric Data Warehouse enables custom prompt-based processing through a generic ai_generate_response(instructions, text) function. This function allows teams to define precise transformation or extraction rules as prompts; apply domain specific logic; and reuse AI behavior consistently across queries and pipelines. Together, these capabilities significantly broaden the scope of data warehousing scenarios supported in Fabric, unlocking new ways to analyze and operationalize unstructured text using T-SQL. Refer to AI functions in Fabric Data Warehouse to learn more about analyzing text with built-in AI functions. ANY_VALUE aggregate Fabric Data Warehouse provides the ANY_VALUE() aggregate, which lets you return an arbitrary value from each group in T-SQL query. This is especially useful when you need to group results by a key (such as GeographyID) but you still want to project descriptive attributes (such as city name, and country) that are functionally the same for every row in that group. An example of such a query is illustrated in the following picture, demonstrating how ANY_VALUE() aggregate can be used to get the values from the group that are not changing. Figure: Using ANY_VALUE() to project descriptive columns while aggregating trips by GeographyID In this pattern, city, state, and country don’t add meaning to the aggregation because they’re constant for a given GeographyID. Adding these columns in the GROUP BY clause or applying more complex or costly aggregates like MIN or MAX is unnecessary overhead and makes queries harder to read and maintain. ANY_VALUE keeps the grouping logic minimal and the intent clear: aggregate by the key and simply carry through the descriptive columns. Refer to the ANY_VALUE function in Fabric Data Warehouse documentation to find additional scenarios where it helps simplify grouping and aggregation logic. Fabric warehouse custom SQL pools (Preview) Custom SQL pools for Fabric Data Warehouse gives administrators finer-grained control over how SQL compute resources are allocated across workloads. Custom SQL pools build on the warehouse’s autonomous workload management by letting you define your own isolation boundaries, explicitly assign resources, and route queries based on application context. With Custom SQL Pools, you can create multiple isolated SQL pools within a single workspace and allocate a percentage of available compute to each. Queries are routed to the appropriate pool ensuring that critical workloads get the resources they need without being impacted by other activities in the warehouse. Figure: Custom SQL Pool Configuration Key benefits include: Predictable performance for critical workloads—Reserve compute for business‑critical reporting or dashboards, so they aren’t disrupted by ad‑hoc queries or background processing. Flexible workload isolation without added complexity—Allocate resources where they matter most without needing to split workloads across multiple workspaces or scale capacity just to protect one workload. Custom SQL Pools are especially useful when multiple applications share a single Fabric warehouse or SQL analytics endpoint and have different performance or priority requirements. As your capacity scales up or down, your pool allocations automatically scale with it, preserving the relative resource distribution you’ve defined. Learn more about custom SQL pools in Fabric Data Warehouse: Custom SQL Pools - Microsoft Fabric | Microsoft Learn. SQL Audit Logs (Generally Available) SQL Audit Logs for Fabric Data Warehouse enable organizations to capture and analyze database activity for security monitoring, compliance, and forensic analysis. Figure: Configuring SQL Audit Logs With this release, we are expanding support and improving accessibility: Support for SQL Analytics Endpoint auditing. Direct access to audit files stored in OneLake, through OneLake Explorer. Ability to download or copy audit files through OneLake Explorer. Ability to open the .xel audit files directly in SQL Server Management Studio (SSMS) for deeper investigation. These capabilities make it easier for security and compliance teams to perform detailed investigations, long-term retention, and external analysis of workflows. For configuration steps and usage details, see the documentation: SQL Audit Logs in Fabric Data Warehouse - Microsoft Fabric | Microsoft Learn COPY INTO and OPENROWSET support for OneLake sources (Generally Available) Previously, this capability supported Lakehouse sources only. With this release, we are expanding support to all OneLake items (except Warehouses). This enables much more flexible ingestion scenarios, including: Using partner workloads such as COPY Jobs. Using staging areas across different Fabric items. Loading data stored anywhere in OneLake-backed items. Customers can now leverage OneLake as a unified staging layer for ingestion workflows while maintaining a consistent SQL experience. Figure: Executing COPY INTO from OneLake sources For full usage examples and configuration guidance, see the documentation: Ingest Data into Your Warehouse Using the COPY Statement - Microsoft Fabric | Microsoft Learn COPY INTO (Transact-SQL) - Azure Synapse Analytics and Microsoft Fabric | Microsoft Learn Outbound Access Protection (OAP) support for Warehouse (Generally Available) Outbound Access Protection for Fabric Data Warehouse provides stronger data exfiltration protection for enterprise environments. Warehouse now supports connector rules that allow organizations to control which external sources the warehouse can access. Customers can define rules to allow access to: Specific Azure Data Lake Storage Gen2 accounts Other Fabric workspaces Approved external connectors This expands the model introduced during Preview, where access was limited to OneLake and local workspace sources only. With connector rules, organizations can enforce controlled and auditable outbound connectivity, helping meet strict governance and compliance requirements. Figure: Supporting OAP Data Connection Polices For details on configuring connector rules, see the documentation: Workspace outbound access protection for data warehouse workloads - Microsoft Fabric | Microsoft Learn Full query text available in Query Insights Query Insights now show the full SQL query text, removing the previous 8,000‑character truncation. The complete query text is available in: The command column of queryinsights.exec_requests_history. The Query Details pane in the Query activity tab. This makes it significantly easier to understand what ran, especially for large, auto‑generated queries from BI tools, ORMs, or complex workloads. You can now retrieve the full query text directly using: SELECT distributed_statement_id, submit_time, total_elapsed_time_ms, command FROM queryinsights.exec_requests_history ORDER BY submit_time DESC; This query takes the most frequently executed query (from Frequently Run Queries) and pulls every historical execution with the full, untruncated SQL text, making it easy to understand exactly what is running and how often: WITH TopQuery AS ( SELECT TOP 1 query_hash FROM queryinsights.frequently_run_queries ORDER BY number_of_runs DESC ) SELECT erh.query_hash, erh.distributed_statement_id, erh.submit_time, erh.total_elapsed_time_ms, erh.status, erh.allocated_cpu_time_ms, erh.data_scanned_remote_storage_mb, erh.command AS full_query_text FROM queryinsights.exec_requests_history AS erh JOIN TopQuery AS tq ON erh.query_hash = tq.query_hash ORDER BY erh.submit_time DESC; Because the full statement is preserved, users can: Immediately understand what logic was executed, not just which query ran. Compare query text across executions to detect subtle changes or regressions. Correlate performance issues with specific joins, filters, or aggregations. This enhancement removes a major gap between observability and action, making Query Insights a more complete tool for day-to-day production troubleshooting. Live connectivity in Migration Assistant for Fabric Data Warehouse (Preview) The live connectivity in Migration Assistant for Fabric Data Warehouse lets you migrate object metadata by connecting directly to your source system into a new Fabric warehouse. This helps you accelerate migration and reduce upfront prep by eliminating the need to generate and upload a DACPAC for the metadata step. The object metadata of schemas, tables, views, functions, and stored procedures gets migrated to warehouse. Figure: Migration using direct connection to the source system Learn more about Migrate with a Direct Connection. Simplify data access with data sources (Generally Available) Fabric Data Warehouse lets you define external data sources that act as named references to locations in your lake (for example, a root folder of a Fabric Lakehouse or an Azure Storage account). External data sources (Preview) were introduced in preview in October 2025, now they are generally available and fully integrated into the Fabric experience, with the full IntelliSense and Copilot support in the SQL query editor. The following visual shows how to create a reusable reference to the Fabric Lakehouse root folder that represents a landing zone where you can store the files before ingesting them in warehouse: Figure: Creating external data source in Fabric Data Warehouse. In the following visual, you can see how you can query files using short, relative paths that are resolved against the data source root: Figure: Accessing files in the referenced data source using the relative path Using a data source keeps queries clean and portable. You can write easy to remember relative paths (like /Files/bronze/logs/*.jsonl) instead of embedding long, environment-specific URLs everywhere in the code. This makes scripts simpler to maintain and easier to share across workspaces and environments. Find more examples about the reading files form the lake in OPENROWSET(BULK) (Transact-SQL) documentation page. Real-Time Intelligence Business Events in Microsoft Fabric (Preview) With Business Events, organizations can move from observing what happened to acting on what matters, in real time. It enables organizations to respond faster, operate more intelligently, and scale real-time decision making across analytics, automation, and AI. You can generate business events from user data functions (UDFs) and notebooks. Once generated, a single business event can power multiple downstream actions, such as: Trigger alerts and automations with Activator, responding immediately via email or Teams. Execute custom logic using user data functions, reacting programmatically to business events. Run analytics and workflows in notebooks, using events to drive downstream analysis. Provide real‑time context with AI and ML, enriching models with governed business signals. Integrate with Spark jobs, dataflows, and Power Automate, enabling distributed processing and business process automation. With Business Events in Real-Time Hub, you can explore, define, and act on critical business signals for the whole organization in a unified experience. Figure: Business Events creation experience For more information about this feature, please refer to the documentation: Business Events in Microsoft Fabric. Building event-driven, real-time applications on database changes with Fabric Eventstreams Deltaflow (Preview) Building intelligent systems that react quickly to operational database changes is simpler with this update. With the release of DeltaFlow, Fabric Eventstreams can seamlessly capture inserts, updates, and deletes from operational databases, transform them from their raw Debezium format, and make them available to downstream event-driven applications using Activator and for real-time analytics in Eventhouse. There’s no need for custom Debezium/JSON processing code or to explicitly manage destination tables through source table schema changes. Easily connect to, ingest from, and transform raw CDC feeds into analytics-ready form. to_connect_to_a_CD Figure: Enabling DeltaFlow when connecting to an Azure SQL database Detect, fetch and register source database & table schemas in the Eventstream schema registry as they evolve. Figure: Automatic registration and use of source table schemas Automatically manage tables in analytics store as they continuously evolve with source schema changes without breaking pipelines. Figure: Automatically created Eventhouse tables with analytics-ready shapes For more information about these features, please refer to the document Building real-time, event-driven applications with Database CDC feeds and Fabric Eventstreams DeltaFlow (Preview). Real-time stream processing with Fabric Eventstreams and Spark notebooks (Preview) This update brings together Fabric Eventstreams and Spark Structured Streaming, making it easier for Spark developers and data engineers to work with real-time data in Microsoft Fabric. These enhancements enable you to access streaming data in Eventstreams directly from Spark notebooks, supporting low-latency processing and end-to-end real-time AI pipelines. Easily discover Eventstreams and real-time sources available through the Real-Time Hub, right from within Fabric notebooks. Figure: Real-Time Hub view inside Fabric notebook—discover Eventstreams in seconds Connect to and process streaming data within minutes using auto-generated PySpark code snippets. Figure: Auto-generated PySpark snippet in a Fabric notebook for an Eventstream Load and use existing notebooks from the Fabric Eventstreams portal. Figure: Load a Spark notebook as an Eventstream destination—reuse and collaborate Securely connect to any Eventstream from a Fabric Spark job without connection strings and secrets, using the enhanced Spark adapter for Eventstreams. Securely connect to any Eventstream from a Fabric Spark notebook using the enhanced Spark adapter—without connection strings or secrets, and with built-in auto-retry support. For more information about these features, please refer to the blog post Bringing Together the world of Real-time Intelligence and Spark Structured Streaming (Preview). Anomaly Detector full-item experience Introducing a refreshed Anomaly Detector full‑item experience that makes it easier to create, run, and explore anomaly detection workflows from end to end. Instead of working through disconnected steps or modal flows, you now get a single, full‑page canvas that brings configuration, analysis, and results together in one place. The updated layout follows Fabric’s shared item experience, so navigation and interactions feel consistent with the rest of the platform. With this new experience, you can more quickly define detection scenarios, run analyses, and immediately dig into detected anomalies without losing context. Clearer sectioning and action cues guide you through the process—from selecting signals and parameters to reviewing anomaly trends and individual data points. Results stay visible alongside configuration, making it easy to iterate, compare outcomes, and refine your setup in real time. Figure: An updated Anomaly Detection UI that simplifies setup, visualizes trends and anomalies in real time, and highlights high‑confidence events for faster investigation The full item Anomaly Detector experience also sets the stage for deeper investigation and richer insights. By consolidating analysis and results into a unified view, you can spend less time navigating and more time understanding what’s changing in your data. Whether you’re monitoring operational metrics or exploring unexpected behavior in real-time signals, these improvements help you move from detection to insight faster and with greater confidence. Operations agent playbook improvements and messages This month sees several improvements for operations agents’ ability to monitor your data and take actions. Based on usage and feedback we’ve heard so far, operations agents are better at mapping between your instructions and the fields in the Eventhouse you connect them to. You’ll also see they can build different types of rules to monitor the specific conditions in your data, including comparing string values in the data and counting data points over time. Finally, you’ll also see better messages if the agent can’t generate a playbook based on the data, goals, and instructions you’ve configured. In those cases, the LLM will try and describe the issue for example not being able to ground a field it inferred from your instructions to a field in the Eventhouse KQL database, or the parameters for a condition or action not being clear in the instructions. For example: This makes it easier for you to debug and unblock the agent configuration. Finally, we’ve updated our best practices and sample for how to give guidance and steering to the operations agent for it to follow your instructions. Learn more about this in the Operations Agent Best Practices and Limitations documentation. Live update for Real-Time Dashboards Real-Time Dashboards now support Live update, a feature that automatically refreshes dashboard visuals when new data is ingested into your underlying data sources. Instead of relying on fixed-interval refresh - which polls your data source on a set schedule regardless of whether new data exists - Live update uses a lightweight background query to detect when data arrives and triggers a refresh only when needed. This event-driven approach offers several benefits. Your dashboards stay current without the compute overhead of constant polling, making it particularly valuable for high-frequency data monitoring scenarios where you need to see data the moment it arrives. For organizations running multiple dashboards or monitoring large data volumes, Live update reduces compute load by eliminating unnecessary refresh cycles during quiet periods. Dashboard viewers also gain flexibility with the ability to pause live updates temporarily. If you're investigating a specific data point and don't want the visuals to change, you can pause updates to analyze the current state without interruption, then resume when you're ready to return to real-time monitoring. Dashboard editors can enable Live update through the dashboard settings, with configuration options including Live update (recommended), manual update only, or a fallback refresh interval for visuals that don't support ingestion detection. Learn more about configuring Live update for your dashboards, with the What is Real-Time Dashboard? documentation. Eventstream SQL Operator (Generally Available) During preview, the Eventstream SQL Operator introduced SQL-based stream processing in Fabric, enabling customers to transform live event data using familiar SQL with rich authoring, preview, and debugging capabilities. Write to multiple destinations from a single SQL operator Consolidate your real-time processing logic into one streamlined SQL block and route results to multiple destinations in a single step. This simplifies pipeline design, making it more efficient and lowers operational overhead. The updated authoring experience makes it easy to add multiple destinations directly within the SQL editor and preview results for each output independently. During testing, dedicated output previews let you validate transformations before you publish. Figure: Route data to multiple destinations from one SQL operator. Event ordering and late event arrival handling Configure event ordering policies directly within the SQL operator to handle late-arriving and out-of-order events. Define thresholds for how long to wait for delayed data and ensure accurate, event-time–correct processing—even in the presence of network delays or asynchronous producers. These policies help build more resilient real‑time pipelines that reflect how data behaves in the real world—not just in perfect conditions. Learn more about Fabric Eventstream SQL Operator. Together, these enhancements make Eventstream SQL Operator more powerful, more intuitive, and ready for production‑grade real‑time workloads. Anomaly Detection as a source in Eventstream Anomaly Detection can be added as a source in Fabric Eventstream, allowing you to publish anomaly events directly into your Eventstream for processing and action. You can add Anomaly Detection as a source from either Eventstream or Real-Time Hub. You can enrich your anomaly events by adding business context and additional information. Further, you may route real-time events to downstream workloads for automated alerting and dashboard visualization. Where to Add This Source You can add Anomaly Detection as a source in two ways: From Eventstream – Create a new Eventstream, select Anomaly detection events as a source. Figure: Adding Anomaly Detection source within Eventstream From Real-Time Hub – Navigate to Real-Time Hub, find the Fabric events and select Anomaly detection events. Figure: Adding Anomaly Detection events in Real-Time Hub Once added, anomaly events flow seamlessly into your Eventstream, ready for transformation and routing to downstream workloads. Get Started Anomalies are now another streaming event—ready to be transformed, enriched, and acted upon. Try out Anomaly Detection as a source in Eventstream today and unlock the power of real-time anomaly pipelines. Learn more about this in the Operations Agent Best Practices and Limitations documentation. Data series colors for real-time dashboard visuals When monitoring operational data, color choices matter. A status indicator showing “Critical” in red and “Healthy” in green communicates meaning instantly—viewers can interpret the visual without reading legends or labels. With data series colors, you can make these intentional choices rather than accepting system defaults. To configure data series colors, switch to Editing mode, select the Edit icon on your tile, and expand the Data series colors section in the Visual tab of the formatting pane. From there, you can select a color for each data series in your visual. Figure: Visual formatting: data series colors setting Figure: Color palette for setting visual elements colors Learn more about customizing your Real-Time Dashboard visuals, refer to the Customize Real-Time Dashboard visuals documentation. Use Copilot to create visuals in real-time dashboards (Preview) Dashboard editors can now use Copilot to create and edit visuals in Real-Time Dashboards using natural language. When you're in Edit mode, open the Copilot pane while creating a new tile or editing an existing one. Describe the insight you need - for example, "Show me the top 10 repositories by push events this week" - and Copilot generates the KQL query, returns the data, and suggests a visual that fits your results. Figure: Real-time dashboard visual in edit mode after Copilot answer has been applied You can accept Copilot's suggestion, refine your question with follow-ups like "Group by event type" or "Filter to the dotnet organization," or edit the query directly. Once you're satisfied, add the visual to your dashboard and use the no-code formatting options to customize its appearance. To learn more, refer to the Copilot-assisted real-time data exploration documentation. Instantly run and preview functions in Microsoft Fabric Eventhouse: no code required (Preview) Previously, working with an Eventhouse function involved manually writing KQL queries. You needed to enter the function name, provide parameters in the right format, and execute the query just to see what results you would get. If you wanted to view the function's body or metadata, you had to run a separate command. That's no longer the case. With the new Preview Functions capability in Microsoft Fabric Eventhouse, you can open the function definition, run the function, and instantly preview its results, with no manual KQL, no parameter guesswork, and no extra commands. Why this matters Eventhouse functions are powerful, but environments evolve. Databases grow, teams change, and you often inherit functions you did not write. Instead of guessing what a function does or manually building a query just to test it, you can: View the function definition instantly. Run the function and preview results with a single click. Test parameterized functions interactively. Browse your function list with search and sorting. This removes friction from everyday workflows. Whether you are exploring unfamiliar logic, validating outputs before building reports, or troubleshooting unexpected results, you get clarity in seconds instead of minutes. How to view or preview a function: In DB Explorer, expand Functions and select a function. A read-only version of the function opens. Select Preview results to instantly run the function and see the output. If the function has parameters, enter your values and preview the results based on your input. The preview shows up to 100 records, providing a quick snapshot of the function’s output. Figure: DB Explorer with the Functions folder expanded and a function selected. The function opens in read-only mode, and the Preview results option is available to run the function and display the output, with fields provided to enter parameters You can view a complete list of all stored functions, including their folder, description, and optional sorting. Built-in search makes it easy to find specific functions, making navigation and discovery simple even in large databases. Figure: Functions list with all available functions in the database, with options to sort, search, and open a menu with additional actions The new Run & Preview Functions feature in Microsoft Fabric Eventhouse lets you instantly inspect function definitions and preview results without writing KQL or handling parameters manually. Quickly explore, test, and manage all your stored functions in one place, saving time and reducing friction. Learn more with the Stored functions list documentation. Workspace monitoring dashboard templates in Microsoft Fabric Eventhouse (Preview) Fabric workspace monitoring provides rich telemetry across your workspace assets, including Eventhouses, Power BI Semantic Models, Data Engineering (GraphQL), and Mirrored Databases. The workspace monitoring data is stored in an Eventhouse, part of Fabric Real-Time Intelligence. To help you turn this data into actionable insights, we have created ready-to-use real-time dashboard templates with out-of-the-box visualizations. Currently, two templates are available: one for Eventhouse items and one for semantic models. From any Workspace Monitoring Eventhouse, users can create these dashboards directly. To get started, go to your Workspace Monitoring Eventhouse, open the upper ribbon, and select Fabric Monitoring. Figure: The ribbon in the Workspace Monitoring Eventhouse allows you to create out-of-the-box dashboards for monitoring From there, choose to create: Eventhouse Monitoring Dashboard—to monitor Eventhouse items in your workspace. You can track: Ingestion results and logs. Commands and queries monitoring. Metrics related to Eventhouse performance. Semantic Model Monitoring Dashboard—to monitor Semantic Models in your workspace. You can use semantic model logs to: Identify periods of high or unusual Analysis Services engine activity by capacity, workspace, report, or user. Analyze query performance and trends, including external DirectQuery operations. Track semantic model refresh durations, overlaps, and processing steps. Monitor custom operations sent using the Premium XMLA endpoint. Once created, the dashboards are ready to use immediately, giving you instant visibility in your workspace. You can also customize them to fit your specific needs. These templates make it fast and easy to track, analyze, and act on workspace activity, all in one place. Learn more in the documentation: Visualize your workspace monitoring . Databases Database Hub in Fabric The Database Hub in Fabric is a new unified database management experience that brings databases across edge, on‑premises, cloud, and Fabric into a single, coherent view. It provides teams with one place to explore, observe, govern, and optimize their entire database estate. Built for scale, the Database Hub uses agent-assisted intelligence to continuously reason over estate-wide signals, surface what changed, explain why it matters, and guide teams toward the right next actions. With built-in observability, delegated governance, and Copilot-powered insights, database agents help teams move from insight to action faster, while humans remain firmly in control of goals, boundaries, and trust. The result is a simpler, more confident way to manage databases at scale today, and a foundation for increasingly autonomous, intelligent database operations over time. Sign up for early access. SQL database in Fabric Since reaching general availability in November 2025, SQL database in Fabric has seen rapid customer adoption as organizations modernize SQL workloads with less operational overhead and tighter integration with analytics and AI. Guided by customer feedback, the platform emphasizes simplicity, autonomy, security, and AI optimization. We are introducing a set of improvements and new features that make it easier to migrate, manage, and optimize SQL workloads in Fabric: Simplified migration with new assistant: The Migration Assistant in public preview helps SQL developers move SQL Server and Azure SQL workloads into Fabric by importing schemas, assessing compatibility, and guiding migration with minimal manual effort. Configurable autonomous management: While maintaining a SaaS-first approach, new options allow database-level control over vCore scaling, expanded compatibility levels, enhanced T-SQL features, and settings that ease application transitions without code changes. Support for all collations: All Azure SQL database collations are supported when creating a new SQL database in Fabric for enhanced global data compatibility and app development flexibility. Collations control text sorting and comparison in SQL databases, impacting filtering, searches, and multilingual content management. Users can specify collations seamlessly during database creation via the REST API across deployment methods. Check out the How to set a different collation for SQL database in Fabric demo and explore the sample code in Git repo. Enhanced data mirroring and security: Auditing and Customer Managed Keys are generally available. CMK for Fabric SQL lets you encrypt databases with your own Azure Key Vault keys to gain full control over key ownership, access, rotation, and compliance. Users can selectively manage which tables are mirrored to OneLake for immediate analytics access. AI and monitoring integration: SQL database in Fabric supports vector search with DiskANN and integrates with Azure AI Foundry for advanced semantic search and AI scenarios, alongside workspace performance dashboards for unified monitoring and optimization. Enhanced data recovery: In Fabric, when a database is deleted, it goes into a soft-deleted state into the Fabric Workspace’s Recycle Bin tab. Depending on the retention configured, the deleted database can be recovered from the Recycle bin while in retention. In addition to this Recycle bin experience, the Fabric SQL database also has the backup retention period configurable from 1-35 days. When the database is hard deleted from the Recycle bin, the backups are still available for the configured backup retention period. This new improvement allows you to restore the backup into a new database to any point in time within the restorable period. Cosmos DB mirroring with Private Link and VNET Cosmos DB mirroring with Private Link and VNET enables customers to mirror data from privately secured Azure Cosmos DB accounts into OneLake. This allows organizations to maintain consistent network security and compliance while supporting near‑real‑time analytics and AI workloads in Microsoft Fabric—strengthening Fabric’s enterprise readiness by design. Figure: Mirroring data from Azure Cosmos DB accounts secured with Private Endpoints or VNETs into OneLake To learn more, refer to the Cosmos DB Fabric Mirroring for Private Networks documentation. Data Factory—Copy Job Richer Change Data Capture (CDC) with Oracle, Fabric DW, and SCD Type 2 Copy job continues to improve the no‑code CDC experience with richer, enterprise‑ready replication patterns. This release introduces a set of enhancements in Copy job in Microsoft Fabric Data Factory that make CDC replication more powerful and easier to use without writing code: Oracle CDC source—Capture changes directly from Oracle databases. Fabric Data Warehouse sink—Replicate CDC data into Fabric Data Warehouse. SCD Type 2—Preserve full history with valid dating, and handle deletes as soft deletes. With built‑in SCD Type 2 and soft delete handling, Copy job automatically preserves every version of a record as it changes over time, instead of overwriting history. This makes it easy to answer point‑in‑time questions, support regulatory audits, and run accurate historical analytics—capabilities that traditionally require complex MERGE logic or custom code. Figure: Enabling SCD Type 2 in Copy job with One Click Learn more in the Change data capture (CDC) in Copy Job documentation. Every row is traceable with built-in audit columns Audit columns are additional metadata columns that Copy job can automatically append to every row it writes to the destination. These columns don't come from your source data—they're generated by the platform to describe the data movement itself. When you enable audit columns in Copy job, each row in your destination table can be enriched with information such as: Audit Column What It Captures Data extraction time The timestamp when the row was extracted from the source by a Copy job run File path The source file path the row was read from (applicable for file-based sources) Workspace ID The Fabric workspace ID where the Copy job resides Copy job ID The unique identifier of the Copy job item Copy job run ID The unique identifier of the specific Copy job execution Copy job name The name of the Copy job that moved the row Lower bound The lower bound value of the incremental window for the current run Upper bound The upper bound value of the incremental window for the current run Custom A user-defined static value—add any additional context your team needs. For example, you can add your source server name here Table: Audit column list With audit columns enabled, you can answer the following questions for any row in your destination table: When was this data extracted? Exact timestamp from when the row was read from the source. Where did it come from? Which file path, which data store. Which job moved it? Which Copy job from which Workspace, which specific run, by name and ID. What was the incremental scope? Lower and upper bounds tell you exactly what slice of data this run covered. No custom code. No expression authoring. Add as many audit columns as you want, and every row in every table your Copy job writes will include this metadata automatically. Figure: Setup audit column in Copy job. Figure: Output on destination data Learn more in What is Copy job in Data Factory - Microsoft Fabric. Workspace Monitoring for Faster, Scalable Troubleshooting As Copy jobs scale from a handful to hundreds, visibility becomes critical. Fabric Workspace Monitoring brings centralized, log‑level observability to Copy job executions, streaming detailed run data into a query able Monitoring Eventhouse inside your workspace. Teams can analyze failures, throughput, duration, and data volumes across all Copy jobs in one place—without inspecting jobs individually. With historical logs, cross‑item correlation, and integration with Data Activator for alerts, Workspace Monitoring helps DataOps teams detect issues earlier and troubleshoot faster at scale. Figure: PBI Report against Fabric Workspace Monitoring metric from Copy job Learn more in Workspace Monitoring for Copy Job in Microsoft Fabric - Microsoft Fabric Boost performance automatically with AutoPartitioning Moving large tables efficiently often requires careful partition tuning—but Copy job now does this automatically. With auto‑partitioning, Copy job detects large datasets and applies an optimal parallel read strategy without any manual configuration. This delivers dramatically higher throughput out of the box, whether you’re copying millions or hundreds of millions of rows. The system adapts dynamically based on data size and source characteristics, ensuring consistent performance across tables while eliminating per‑table tuning effort. Figure: Enabling auto partitioning in Copy job Learn more in What is Copy job in Data Factory More flexible incremental copy with new watermark column types Incremental copy is a core pattern for keeping analytics data up to date—but in real world systems, changes aren’t always tracked with a clean datetime column. To address this, Copy job now supports additional watermark column types, making incremental copy more flexible and applicable across a broader range of source systems Copy job now supports ROWVERSION, Date, and String (interpreted as datetime) watermark columns. This allows you to choose the column that best represents change in your source system, while Copy job continues to automatically manage state, checkpoints, and incremental windows. ROWVERSION enables precise and reliable change tracking in SQL‑based systems, capturing every insert and update without relying on application‑managed timestamps. Date watermark support works seamlessly with common columns like LastUpdatedDate or ModifiedAt, with built‑in delayed extraction to prevent data loss or overlap between runs. String (interpreted as datetime) support removes the need for custom queries or schema changes when timestamps are stored as strings, improving compatibility with real‑world schemas. These enhancements make incremental copy easier to configure, more resilient in production, and better suited for diverse enterprise data models—without adding complexity for users. Learn more in What is Copy job in Data Factory - Microsoft Fabric. Data Factory—Dataflow Gen2 Preview-only steps (Generally Available) This capability improves authoring performance without changing runtime behavior. Preview-only steps let you run specific transformations during data preview only, automatically excluding them from dataflow execution and refresh. That means your production logic stays exactly the same—while the authoring experience becomes faster, smoother, and more responsive. This capability addresses a common challenge when building Dataflow Gen2 items: iterating on logic can be slow when previews must evaluate full datasets. Preview‑only steps make it possible to temporarily reduce data volume or complexity during development, enabling faster validation of transformations without introducing conditional logic or modifying the final query definition. Common uses include filtering or isolating subsets of data to accelerate previews, testing transformation logic without waiting for full evaluation, and exploring new data sources while keeping production execution intact. Because preview‑only steps are ignored during refresh and run operations, they provide a safe way to optimize the authoring workflow without risking unintended changes in published outputs. Figure: The "Enable only in previews" option within the applied steps section Preview‑only steps are also integrated into specific authoring dialogs, including file system views and the Combine files experience. In these contexts, Dataflow Gen2 can automatically introduce preview‑only logic to limit sample data used during preview evaluation, further reducing load time while preserving the behavior of the final dataflow. With general availability, preview‑only steps become a standard part of the Dataflow Gen2 authoring model—helping teams iterate faster, validate transformations more efficiently, and maintain a clear separation between development‑time experimentation and production execution. Learn more: Preview only step in Dataflow Gen2 (Preview) - Microsoft Fabric. Fabric Variable Library integration (Generally Available) Throughout the last couple of months, we’ve removed some of the limitations such as: Variable limit: the previous limit was 50 variables. You can now reference as many variables as you need in your Dataflow. Power Query editor support and using a default value: you can now see how the variable gets evaluated within the actual Power Query editor. Not only that but within the Dataflow Gen2 experience, after enabling the input widgets through the Options menu, you will also have a new way that simplifies how you can reference variables with a complete no code experience fully integrated into the experiences and dialogs that you love from Dataflow Gen2: Figure: The Filter rows dialog within Dataflow Gen2 showing the input widget and the option to Select a workspace variable After selecting this option in any of the dialogs, the experience of selecting a Variable from a library will appear. Figure: The select variable dialog invoked from within a Dataflow Gen2 Be sure to test this improved experience and share your feedback. Learn more: Use Fabric variable libraries in Dataflow Gen2 (Preview) - Microsoft Fabric New data destinations Dataflow Gen2 continues to expand where curated data can land, supporting both lake‑first architectures and hybrid data estates. With new and updated destinations, teams can publish transformed outputs in the formats and platforms that best fit their downstream consumers—whether that’s open lake storage, lakehouse files, enterprise warehouses, or business‑friendly file formats. Azure Data Lake Storage Gen2 (Generally Available) Dataflow Gen2 now supports Azure Data Lake Storage Gen2 (ADLS Gen2) allowing teams to land curated outputs directly into their data lake using open formats and folder structures aligned to organizational standards. This enables lake‑first ingestion patterns for organizations that treat ADLS as their system of record, while still authoring transformations using low‑code Dataflow Gen2 experiences. Common scenarios include reusing curated outputs across Fabric (Spark and SQL) as well as external systems that are read directly from Azure Data Lake Storage Gen2. Figure: ADLS gen2 destination option Lakehouse files (Generally Available) Dataflow Gen2 can write outputs directly into the Files area of a Fabric lakehouse. This is useful when downstream consumers expect file‑based outputs rather than tables, or when teams need to align with existing folder and file conventions inside the lakehouse. This enables patterns where transformed extracts are consumed by Spark notebooks, pipelines, or external tools, while also supporting hybrid designs where some Dataflow outputs are tables and others are files within the same Fabric workspace. Figure: Lakehouse files option Snowflake databases (Preview) This enables transformed outputs to be published directly into Snowflake databases as part of Fabric‑based, low‑code transformation workflows. This supports hybrid data estates where Fabric is used for transformation while Snowflake remains the target platform for analytics or data sharing. This preview helps standardize transformations across platforms and enables analysts to departments to publish governed outputs into Snowflake without duplicating transformation logic. Figure: Snowflake destination option Excel files (Preview) Dataflow Gen2 is introducing the ability to write outputs as Excel files (Preview) for supported filesystem destinations such as SharePoint and ADLS Gen2. This makes it easier to support business processes that still rely on Excel, while keeping transformation logic centralized and governed in Fabric. Typical scenarios include publishing refreshed Excel extracts for operational reporting or legacy workflows, and standardizing Excel output formatting from a single Dataflow definition. Schema support in Fabric data destinations (Generally Available) As Dataflow Gen2 adoption grows, many teams run into organizational challenges when publishing tables into shared destinations. Without schema control, teams often resort to creating separate databases, warehouses, or lakehouses just to keep tables logically grouped—adding complexity and making collaboration harder. With this release, Dataflow Gen2 data destinations now support writing into specific schemas (where applicable). This capability is now generally available for destinations such as Fabric SQL databases, Lakehouses, and Warehouses, giving teams more control over how Dataflow outputs are structured and governed. What’s improved Better organization without extra destinations: Teams can organize tables by domain—such as finance, sales, or HR—using schemas instead of creating separate destinations for each area. This keeps environments cleaner while still enforcing logical separation. Smoother collaboration in shared environments: Multiple teams can publish tables into the same warehouse or SQL database while maintaining clear ownership and structure through schemas. This reduces naming conflicts and supports shared analytics models without friction. Figure: The connection settings for the Warehouse connector using the advanced options to set the Navigate using full hierarchy to true By aligning Dataflow Gen2 outputs with enterprise schema conventions, this enhancement makes it easier to support multiteam data platforms, improve governance, and scale Dataflow Gen2 usage across the organization without restructuring existing destinations. Learn more: Dataflow Gen2 data destinations and managed settings - Microsoft Fabric AI-Powered Prompt Transform (Generally Available) Fabric AI Prompt is integrating generative AI features into the low-code data transformation process. Authors can enrich and transform data using natural language prompts without building or managing machine learning models, while staying within the Dataflow Gen2 execution model. The AI Prompt capability is accessed from the Add column experience, where authors define a prompt and select columns to provide contextual input. This allows AI-driven enrichment to be expressed inline alongside existing Power Query transformations, keeping logic centralized and auditable. Figure: The AI Prompt dialog in Dataflow Gen2 Moving forward, all operations associated to the usage of AI Prompt within Dataflow Gen2 will be accounted towards an explicit AI meter with the operation name of “AI Functions”. Learn more: Fabric AI Prompt in Dataflow Gen2 (Preview) - Microsoft Fabric Publish experience UX + performance improvements (parallelized query validations) Publishing complex Dataflow Gen2 items can be time‑consuming, especially when dataflows contain many queries or multiple destinations. In these cases, validations are required before a dataflow can be published, and waiting for those checks to complete often slows down iteration and troubleshooting. With this release, we’ve improved the Dataflow Gen2 publish experience through a refreshed user interface and performance enhancements that parallelize query validations. By running validations concurrently, publish operations complete faster and surface issues earlier in the process. What’s improved Less time waiting: Dataflows with multiple queries and destinations publish faster, reducing idle time during validation and helping teams move through development and testing more efficiently. Clearer guidance during publication: Validation results are available sooner, making it easier to identify and resolve issues without repeated publish attempts or back‑and‑forth edits. Together, these improvements shorten the publish cycle, reduce friction when working with larger dataflows, and help teams iterate on Dataflow Gen2 solutions with more predictable and responsive feedback. Learn more: Dataflow Gen2 with CI/CD and Git integration. Save As Improvements: Scheduled Refresh Policies and Public APIs Save As continues to improve the migration experience to Dataflow Gen2 (CICD), especially for teams moving large numbers of dataflows across workspaces or tenants. One common challenge during migration is preserving refresh behavior—copied dataflows often require manual reconfiguration before they are production ready. With this release, Save As now supports Scheduled Refresh Policies for Dataflows Gen1, ensuring that refresh configurations are retained when copying a dataflow. This reduces post migration cleanup and helps teams move faster with fewer manual steps. In addition, we’re introducing a new public Save As API for Dataflows Gen1 designed for automation and bulk operations. This enables organizations to programmatically copy dataflows at scale, making it easier to support structured migration plans and repeatable rollout processes. What’s improved Streamlined refresh configurations: Dataflows created using Save As can now inherit scheduled refresh policies from the source dataflow, helping ensure consistent refresh behavior without re‑authoring schedules after migration. Automation at scale: The new Save As public API enables automated and bulk copy scenarios, allowing teams to migrate many Gen1 dataflows to Gen2 programmatically. This is particularly useful for multi‑workspace and multi‑tenant deployments where manual migration isn’t practical. Together, these enhancements reduce migration friction, minimize manual edits, and help teams adopt Dataflow Gen2 more efficiently—whether migrating a handful of dataflows or rolling out Gen2 at enterprise scale. Figure: Dialogs for the refresh and scheduling mechanism when using the Save as experience for Dataflow Gen2 Learn more: Save As Dataflow Gen2 documentation and public Save As API reference. SharePoint site picker in Modern Get Data and Data destinations (Preview) SharePoint Site Picker replaces manual URL entry with a browsable dropdown, so you can select the right SharePoint site directly instead of finding and pasting URLs Why this matters Eliminates manual URL copy-and-paste and context switching. Reduces connection errors caused by wrong URL formats. Surfaces Recent sites and Favorite sites instantly on dropdown open and enable you to search to find sites. Where the experience is available SharePoint site picker is available for SharePoint sources of Get Data in Dataflow Gen2, Pipelines, Copy Job, and Lakehouse shortcuts, and as a destination in Dataflow Gen2. How to use SharePoint Site Picker Simplified SharePoint Site Selection: Instead of copying URLs manually, use the Site URL dropdown to choose from Recent sites and Favorite sites. Figure: SharePoint Site Picker dropdown Quick Search Capability: Find related sites faster by typing in the dropdown search box. Once you select a site, you can load data into the Power Query editor for transformation. Figure: SharePoint Site Picker searched for results Learn more about SharePoint folder connector, SharePoint list connector, and SharePoint online list. Diagnostics download (Preview) Dataflow Gen2 diagnostics download provides a simple way to collect logs and diagnostic artifacts for both cloud-based and VNET gateway dataflows. Instead of rerunning refreshes or guessing at failures, you can download the information needed to investigate issues directly. This helps teams fail faster and fix issues sooner. Downloadable diagnostics make it easier to identify refresh failures, performance bottlenecks, and connectivity problems, including complex networking scenarios that rely on VNET data gateways. Figure: The recent runs dialog showing the new button at the bottom left of the dialog to Download detailed logs With clearer signals available upfront, support investigations are shorter and operational friction is reduced. Learn more: An overview of refresh history and monitoring for dataflows. Advanced Edit for destinations (Preview) The new Advanced Edit experience for Data Destinations enables editing of the underlying M logic that configures destination settings. This unlocks deeper customization, including the ability to leverage parameters to drive destination behavior—an important step for teams standardizing deployments across environments. Parameter-driven destinations: switch target schema/table, file paths, or naming conventions without rewriting queries. Unblock advanced scenarios that require destination settings not yet available in the simplified UI. Figure: The new Advanced editor for data destinations Learn more: Advanced edit for data destination queries in Dataflow Gen2 - Microsoft Fabric. Data destination validations during publish (Preview) Dataflow Gen2 now validates data destinations during publication, helping catch common issues earlier in the development cycle. These validations surface problems such as missing permissions, invalid destination settings, or naming conflicts before the first refresh runs. By shifting these checks to publish time, authors get clear, actionable errors sooner—when changes are easiest to fix. This shortens the feedback loop and reduces time spent troubleshooting runtime refresh failures after deployment. For creators, earlier validation means fewer broken dataflows entering production. This reduces operational noise, minimizes rework, and helps ensure that published dataflows are refresh ready and more stable by default. Learn more: Dataflow Gen2 data destinations validation rules. Evaluate query API (Preview) The Execute Query API (Preview) enables on-demand execution of Power Query logic in Dataflow Gen2 scenarios—without requiring a full scheduled refresh cycle. It’s designed for cases where you need to trigger transformations programmatically (or in response to events) and retrieve results quickly for downstream processing. Event-driven pipelines: run a transformation when new data arrives and push outputs to a destination or consumer immediately. Streaming and near-real-time scenarios: execute queries more frequently than a typical scheduled refresh to support operational dashboards and alerting workflows. Automation at scale: integrate with orchestration tools and scripts to run specific queries as part of broader ETL/ELT jobs. Faster debugging: re-run targeted queries to validate fixes without republishing the entire dataflow. Learn more: Execute Query API (Streaming) documentation (Preview). Data Factory Data Factory MCP (Preview) Dataflow Gen2 offers a suite of pipeline functions, including dataflow creation, M (Power Query) scripting, connection management, query execution, and refresh coordination. These tools are directly accessible to AI assistants. Access is available through platforms such as VS Code, Claude, ChatGPT, Gemini, or via the command line. Why it matters AI assistants create, test, and deploy dataflows through natural language—no browser tabs or manual configuration required. Iterative M development via execute_query lets the AI test transforms against live data before committing to a full refresh. MCP Apps provide guided UI forms (connection setup, gateway selection) inside the chat panel. Open source (GitHub), ships as a NuGet package, runs locally—credentials never leave your machine. Learn more: Data Factory GitHub repo. IBM Netezza ODBC Driver (Generally Available) As we move away from using the embedded Simba driver, customers now have a more dependable and supported option by using their own Netezza driver. This update ensures continued connectivity, long-term support, and a more future-ready experience for organizations using the Netezza connector. Customers do not need to install the new connector; you may reuse your existing connector but will need to install the new IBM Netezza ODBC driver. Figure: IBM Netezza Connector Selection in Fabric UI Reference the IBM Netezza ODBC documentation for more information Google BigQuery connector (Generally Available) This update reflects a shift to the newer GBQ connector as the supported, long-term path forward, providing customers with improved reliability, and alignment with our evolving security standards. With this update, customers can use a connector designed for durability, compliance, and future enhancements. Figure: Google BigQuery Connector in Fabric UI Additional details are available in the Google BigQuery connector documentation. QuickBooks Online connector retirement The QuickBooks Online connector is being retired and will no longer be supported as of March 2026. As part of our ongoing platform evolution, this change streamlines our connector portfolio and ensures our continued commitment to only the highest level of secure data connectivity. After retirement, customers will no longer be able to create new connections, and existing connections may no longer function. Lakehouse Maintenance activity in Fabric Pipelines (Preview) Keeping your Lakehouse healthy shouldn’t require a long checklist or manual scripts. The new Lakehouse Maintenance activity (Preview) makes it easy to automate common upkeep tasks directly inside Fabric Data Factory pipelines. Figure: The Lakehouse maintenance activity in Fabric pipelines With this activity, you can schedule and run actions like vacuuming old files, optimizing table layouts, and managing storage—all in a repeatable, governed workflow. It’s a simple way to keep performance high and storage costs in check, especially for teams managing large or fast‑growing datasets. Figure: The Lakehouse maintenance activity settings Whether you run maintenance nightly or as part of a broader DataOps process, this activity helps support reliable Lakehouse operations. Check out our Lakehouse Maintenance documentation. Refresh SQL endpoint activity in Fabric pipelines (Preview) The process of keeping your SQL analytics layer current is now simpler. The new Refresh SQL endpoint activity (Preview) lets you refresh your Lakehouse SQL endpoint on-demand or as part of your pipeline orchestration. Figure: The Refresh SQL endpoint activity You can trigger targeted refreshes after data ingestion, run coordinated refreshes alongside your transformations, or ensure downstream consumers always see the latest state. It’s built for operational consistency – especially for BI, reporting, and real‑time analytics scenarios that rely on predictable SQL performance. Figure: The Refresh SQL Endpoint activity settings. This activity gives you more control, less manual overhead, and a smoother end‑to‑end refresh experience. Check out the RSQL documentation for more details. Generate Pipeline expressions with Copilot (Generally Available) Writing expressions doesn't have to be time-consuming; simply describe your needs in natural language, and Copilot will generate pipeline expressions for you. Figure: Generate Pipeline expressions with Copilot Whether it’s building dynamic folder paths, conditional logic, string parsing, or parameterized values, Copilot now handles the expression authoring for you. This feature removes friction for both new users and power users – saving time, reducing errors, and making expression logic easier to understand. Workspace monitoring for Fabric Data Factory’s pipelines and Copy job (Preview) Operational observability continues to evolve in Fabric. We’re taking the first major step toward workspace‑level observability in Microsoft Fabric Data Factory. Until now, understanding how pipelines and copy jobs behave at scale often meant inspecting individual runs via Monitoring Hub. With the introduction of workspace monitoring (Preview), Data Factory begins a shift to a workspace‑wide view of operational health. The newest workspace monitoring updates bring clearer visibility and faster troubleshooting across your pipeline ecosystem. What’s available A workspace-wide view of item-level runs Rich filtering, sorting, and drilldown Insight into failure patterns, duration trends, and operational health Faster navigation—no need to click into each pipeline This gives DataOps teams a unified lens to understand performance and diagnose issues quickly. Figure: A view of your pipelines and Copy jobs within the workspace monitoring solution Coming Soon Activity-level L2 monitoring for pipelines. Copy job L2-level monitoring (Preview) for deeper insights and debugging. These improvements continue building toward a more comprehensive, intuitive monitoring experience for production workloads. Check out our docs on Enable Workspace Monitoring in Microsoft Fabric and Workspace Monitoring for Copy Job in Microsoft Fabric for more information on how to use this experience. Interval-based schedules The latest enhancement to Fabric Data Factory pipelines is the availability of interval-based schedules! This powerful new feature allows you to automate data workflows at regular non-overlapping intervals, like the popular tumbling window trigger in Azure Data Factory. Figure: Interval-based schedule configuration in Fabric Data Factory With interval-based scheduling, you can easily configure recurring pipeline runs that ensure timely data processing and seamless integration across your architecture. New Airflow APIs New Airflow Operators Apache Airflow jobs in Fabric Data Factory facilitate the execution of a wide range of Fabric artifacts through native operator integration. Users can run artifacts such as Notebooks, Spark job definitions, Pipelines, Semantic Models, and user data functions directly from their DAGs. Apache Airflow jobs now provide support for executing Copy jobs and dbt jobs! Figure: Airflow operators for Fabric items, including Copy job and dbt job execution To learn more, refer to Run a Fabric item using Apache Airflow DAG. PowerShell model for gateways (Generally Available) The PowerShell model for gateways now delivers fully supported, production-ready automation for gateway lifecycle, update, restore, and configuration management. This release introduces new commands for version discovery and upgrade control, along with reliability and usability improvements that make large-scale, script-driven gateway operations easier and more robust. Figure: Using the Gateway PowerShell module to manage gateway operations from the command line Learn more through the gateway PowerShell documentation and cmdlet reference on Microsoft Learn. Certificate and proxy support for VNet data gateway (Generally Available) Certificate and proxy support for VNet data gateway enables secure, compliant connectivity in enterprise environments. Organizations can use enterprise-issued certificates for gateway authentication and configure proxy routing when direct internet access is restricted. Together, these capabilities strengthen security, support corporate network policies, and expand deployment flexibility in controlled and regulated infrastructures. nshot_of_certificate_and_proxy_settings_for_a_Virtual_Network_Data_Gateway Figure: Configure certificate authentication and proxy for a Virtual Network Data Gateway Learn more through Manage virtual network (VNet) data gateways. Virtual network data gateway supports up to nine instances This update enables greater scalability and higher throughput for enterprise workloads. With expanded instance capacity, organizations can handle increased data movement and processing demands, improve parallel job performance, and enhance reliability for mission-critical tasks. This update provides more flexibility to scale gateway infrastructure in line with growing business needs. Figure: Virtual Network Data Gateway now supports scaling up to nine instances per cluster Learn more What is a virtual network (VNet) data gateway. SSIS Pipeline Activity (Preview) SQL Server Integration Services (SSIS) has been a cornerstone of enterprise data integration for decades, powering mission-critical ETL workloads across thousands of organizations worldwide. Invoke SSIS Package activity in Data Factory in Microsoft Fabric (Preview), provides the power of your existing SSIS investments directly into Fabric's unified SaaS analytics platform. Figure: Add an Invoke SSIS Package activity Many enterprises have significant investments in SSIS packages that orchestrate complex ETL workflows across on-premises databases, file systems, and cloud services. Until now, running these packages required either an on-premises SQL Server, or the Azure-SSIS Integration Runtime in Azure Data Factory. Both options meant managing additional infrastructure and staying outside the Fabric ecosystem. Figure: Invoke SSIS package activity configuration But the Invoke SSIS Package pipeline activity in Microsoft Fabric Data Factory changes this. It allows you to execute your existing SSIS packages directly from a Fabric pipeline, enabling true lift-and-shift of legacy ETL workloads into Fabric—no package rewrite required. There is no need for integration of runtime management or stopping and starting IRs; simply incorporate them into your pipeline. Seamlessly upgrade Azure Data Factory and Synapse pipelines to Microsoft Fabric (Preview) Microsoft Fabric Data Factory now offers a guided (Preview) migration experience to help you move existing Azure Data Factory (ADF) and Azure Synapse Analytics pipelines into Fabric—starting with an assessment-first approach so you can migrate intentionally and validate before switching production workloads. Review readiness and plan next steps: The assessment categorizes pipelines and activities so you can decide what to migrate now vs. what to fix or defer. You can also export results to CSV for offline review and remediation planning. Figure: Review pipeline and activity readiness results in Azure Data Factory (ADF only) Mount your factory to Fabric For Azure Data Factory migrations, you’ll mount your ADF into a Fabric workspace and then continue the remaining steps inside Fabric. Figure: Continue the migration flow in Fabric after mounting Migrate selected pipelines in Fabric In Fabric Data Factory, open the mounted factory (ADF) or chosen workspace (Synapse), then migrate the pipelines you want to migrate. Map linked services to Fabric connections and complete migration During migration, you’ll map ADF/Synapse linked services to Fabric connections. For guidance on creating and managing connections in Fabric, refer to Data source management. Figure: Map Linked Services to Fabric Connections Validate and promote After migration, validate connections and credentials, run end-to-end tests, and then re-enable triggers as needed. Pipelines migrate safely, with triggers disabled by default so you stay in control of execution. Learn more: Upgrade your Azure Data Factory pipelines to Fabric. Data Factory—Mirroring Mirroring for SAP (Generally Available) Built on top of SAP Datasphere’s Premium Outbound Integration, mirroring for SAP seamlessly integrates Fabric’s advanced mirroring engine with SAP Datasphere’s replication flows, unlocking connectivity through SAP’s native data extraction technologies. This means direct access to the full suite of SAP applications—whether it’s SAP S/4HANA (on-premises or cloud), SAP ECC, SAP BW, SAP BW/4HANA, or cloud solutions like SAP SuccessFactors, SAP Ariba, and SAP Concur. Mirroring capabilities allow you to: Eliminate data silos by bringing SAP data alongside other enterprise sources in OneLake. Maintain end-to-end data lineage and governance for compliance and auditability. Accelerate time-to-insight with near real-time data replication no custom ETL required. Figure: Mirrored database for SAP Learn more in Microsoft Fabric Mirrored Databases From SAP. Mirroring for Oracle databases (Generally Available) Mirroring for Oracle is now available in Microsoft Fabric, bringing a production‑ready, enterprise‑grade way to continuously replicate Oracle data into OneLake with no custom ETL pipelines. This milestone reflects strong validation from customers already running Mirroring for Oracle in production and marks a major step forward in Fabric’s zero‑ETL data integration strategy. With near real‑time data replication, customers can keep analytics, BI, and AI workloads continuously in sync with their operational Oracle systems. This release delivers improved stability, scale, and operational readiness, informed directly by customer feedback from public preview deployments. Mirroring for Oracle integrates natively with Fabric experiences like Power BI, Notebooks, and Lakehouses, enabling faster insights without disrupting existing Oracle workloads. As a fully supported capability, Mirroring for Oracle is now ready for broad enterprise adoption with long‑term investment from the Fabric platform team. Figure: Mirroring for Oracle creation steps Learn more at Mirroring for Oracle in Microsoft Fabric. Mirroring for Azure Database for MySQL (Preview) Mirrored databases now support Azure Database for MySQL. This capability enables you to directly replicate data from Azure Database for MySQL Flexible Server into Fabric in near real time, ensuring that information remains current, readily query-able, and seamlessly integrated throughout the analytics stack without the need for traditional ETL processes. Mirrored MySQL data is managed alongside other data sources, facilitating cross-source querying, unified reporting, and comprehensive analytics. Figure: Screenshot of configuring a mirrored database for Azure Database for MySQL Learn more in Microsoft Fabric Mirrored Databases for MySQL. Mirroring for SharePoint List (Preview) Mirroring for SharePoint Lists enables continuous replication of SharePoint Lists and Document Libraries into OneLake without building custom ETL pipelines. This capability keeps SharePoint data automatically synchronized in near real time, ensuring analytics in Fabric stay aligned while SharePoint remains the system of record. When mirrored, both list tables and document library metadata land in OneLake in an analytics‑ready format, with document libraries replicated via shortcuts and converted into Delta Lake tables. Figure: Mirroring setup for a SharePoint list Fabric automatically creates a mirrored database and a read‑only SQL analytics endpoint, providing a rich analytical surface over the replicated data. As changes are made in SharePoint—such as new columns or updated rows—those updates flow continuously into Fabric, keeping schemas and data in sync. This public preview unlocks a simple, unified way to analyze SharePoint operational data across Fabric workloads including SQL, Power BI, notebooks, and data engineering experiences. Extended Capabilities in Mirroring: Change Delta Feed and Snowflake Mirroring Support for Views (Preview) Optional enhancements that build on core mirroring to support more advanced, real‑world analytics scenarios. These capabilities are designed for customers who need more than basic replication—enabling faster freshness, incremental processing, and business‑ready data without building or maintaining complex ETL pipelines. Including Change Data Feed (CDF), which captures inserts, updates, and deletes at a granular level and applies them incrementally into OneLake, allowing mirrored data to stay continuously fresh without full reloads. Extended Capabilities also include Mirroring Views for Snowflake (with support for other sources coming soon), which replicate logical views from the source system into OneLake so that source‑defined business logic—such as joins, filters, and transformations—can be preserved directly in Fabric. Together, CDF and Views enable incremental pipelines, near real‑time analytics, and shaped datasets that are immediately ready for consumption across Fabric workloads. Extended Capabilities are enabled during mirror setup and operate on top of core mirroring, allowing customers to selectively opt into advanced functionality as their analytics and AI needs grow. Billing will be available as part of these extended capabilities starting April 1, 2026. More details about these capabilities and billing can be found on our documentation: Extended Capabilities in Mirroring – Overview. Mirrored database now supports up to 1000 tables To meet growing business demands and improve scalability, mirrored databases now support up to 1000 tables, raised from the previous limit of 500. This enhancement significantly expands the scale of datasets that can be mirrored from the source database, enabling customers to bring more comprehensive data into Fabric without fragmentation, drive deeper analysis and scale the data solution to meet evolving requirements. Learn more in Mirroring in Microsoft Fabric. That’s a wrap! Publishing this update on the first day of FabCon feels especially meaningful. The features in this release reflect not just ongoing platform investment, but the ideas, feedback, and candid conversations we continue to have with the Fabric community—in sessions, online, and across preview programs. Thank you for showing up, sharing your experiences, and helping shape where Fabric goes next. We encourage you to explore these updates, ask questions, and tell us what’s working—whether that happens here at FabCon, in community forums, or through ongoing feedback channels. We’re grateful to be building Fabric alongside such an engaged community, and we’re excited to keep learning from you throughout FabCon and beyond.
Microsoft Fabric July 2023 Update
Welcome to the July 2023 update. We have features in Core, Synapse, Data Factory, Data Activator, Community, and Power BI. Contents Core Help Pane Monitoring Hub improvements Admin Microsoft Public Preview Default Sharing On by Default Effective July 5 th , 2023 OneLake OneLake file explorer update with support for switching organizational accounts Item sharing Synapse Data Warehouse Automatic updates for statistics UPDATE FROM and DELETE FROM T-SQL Optimization in data movements during query execution Dataflows Gen2 from the Warehouse Zero Copy Table Clones Data warehouse sharing Dbt Support for Fabric Data Warehouse Data Engineering Load to Table Feature Updates Lakehouse sharing and permission management Notebook Resources Notebook: Show cell output after session reconnection Notebook: Sharing and permission management Notebook: New status bar Data Science 1 Microsoft Fabric and SynapseML Offer New Features for Large Language Models Real-time Analytics KQL Database Create a Continuous Data Connection from Event Hubs to KQL Database Query KQL Database from MATLAB Data Factory Dataflows Gen2 Power Query Editor Enhancements to output destination experience Improvements to existing experiences Connectors Google Analytics (Connector Update) Oracle Database (Connector Update) Azure Databricks, Databricks (Connector Update) Denodo (Connector Update) EQuIS (Connector Update) Snowflake (Connector Update) Anaplan (Connector Update) Gateway and Connections SSO option in cloud connections Flag in cloud connections to allow them to be with gateway Data pipelines Activities Teams Activity Productivity Artifact parameterization support Performance improvement for moving parquet files to Lakehouse tables “Save as” support now available from your workspace Support column mapping for Lakehouse destination in Copy activity Data Activator Sample data Adding multiple streams or properties Designing triggers and alerts Trigger lifecycle Testing actions Teams app Community Implementing a Lakehouse with Microsoft Fabric Courses Filter by product for User Groups & Events Data Activator Community now live Power BI Reporting Smoothed and Leader lines – Updates On-Object Interaction (Preview) – Updates Customize the Pane Switcher CTRL click to open multiple panes in the Pane Switcher Treemap sub-selections now supported: Noteworthy bug fixes Modeling Edit your data model in the Power BI Service– Updates (Preview) Relationship validation Service Dataset details page revamp Mobile Datasets are coming to Power BI Mobile apps! Developers Boost Your Custom Visuals Performance: Check out our Latest Article! Visualizations New visuals in AppSource Elevate Your Map Charts With Drill Down Map PRO We are ready to present our new visual: an "all-in-one" Multi Target KPI card Other WebView2 GA Core Help Pane Help pane is feature-aware and displays articles about the actions and features available on the current Fabric screen. It is also a search pane that quickly finds answers to questions in the Fabric documentation and community. Open the Help pane From the upper-right corner of the Fabric screen, select the ? Icon. Help pane is feature-aware The feature-aware state is the default view of the help pane when you open it without entering any search terms. It shows a list of recommended topics, resources that are relevant to your current context and location in Fabric, and a list of links for other resources. It has three sections: Feature-aware documents: This section groups the documents by the features that are available on the current screen. As you explore Fabric, the feature-aware documents update based on what you've selected and where you are in Fabric. This is a great way to learn how to use Fabric. Give yourself a guided tour by making selections in Fabric and reading the feature-aware documents. Forum topics: This section shows topics from the Community forums that are related to the features on the current screen. Select a topic to open it in a new separate browser tab. Other resources: This section has links for feedback and Support. Help pane is a search engine Enter a keyword to find relevant information and resources from Microsoft articles and Community forum topics. Use the dropdown to filter the results. Monitoring Hub improvements With the new “column options” feature in Monitoring Hub, you can now customize the columns in the job list to check the information you are interested in. The columns selections would be memorized automatically. Feel free to go to other places, we will keep the columns and the filters for you once you are back at the monitoring hub. Admin Microsoft Public Preview Default Sharing On by Default Effective July 5 th , 2023 On July 5th, 2023, Microsoft began a staged rollout of an update to the Fabric Public preview setting. After the update, Fabric Public preview will be ON by Default unless customers explicitly opt out. With Fabric Public preview set to ON, users in your organization will be able to create Fabric items in workspaces attached to a Power BI Premium or Fabric capacity. When the Microsoft Fabric Preview was launched on May 23, 2023, Fabric was disabled by default, using a new tenant setting named "Users can create Fabric items (public preview)". Starting July 5th, 2023, Microsoft started activating Fabric preview by default for tenants who have not explicitly opted out. Any capacity level overrides of this setting will remain unaffected. If you are comfortable with your organization using the Fabric preview features, no further action is required. However, if you want to restrict usage of the Fabric preview in your P-SKU and F-SKU capacities, you have two options: Use the security group configuration of this setting at the tenant and capacity levels to limit Fabric to a smaller set of users. Turn off the setting at both the tenant and capacity levels to prevent all users from accessing the Fabric preview features. If you make these updates before the automatic update rolls out your tenant, this tenant setting update will not impact your organization. You may also make these changes at any time after the update. For more information about controlling Fabric use, visit Microsoft Learn. OneLake OneLake file explorer update with support for switching organizational accounts With the latest OneLake file explorer v1.0.9.0, it’s simple to choose and switch between different Microsoft Azure Active Directory (AAD) accounts. This was a highly requested feature from users who work with multiple AAD tenants, across organizations or even within a single organization. To switch accounts, right click the OneLake icon in the Windows notification area, select “Account” and then “Sign Out”. Signing out will exit OneLake file explorer and pause the sync. To sign in with another account, start OneLake file explorer again by searching for "OneLake" using Windows search (Windows + S) and select the OneLake application. Item sharing Starting July 2023, you can share Fabric items like data warehouses, lakehouses, Spark job definitions, Kusto databases, and KQL query-sets with users or groups. This enables collaboration with users that are not in workspace roles. Admins, Members, or users that have been granted reshare permission on a specific item, can share it with additional users outside of the workspace. You can share an item by clicking on Share in the item list or within the item. When an item is shared, the recipients can discover the item in Data Hub and may also receive a link to the item via email (if the option is selected while sharing). While sharing, a user can choose the level of access that the recipient can have. For example, when sharing a lakehouse, you can choose to grant the recipient with Read All SQL endpoint data permission, in addition to the read permission. This enables the recipient to read the default dataset associated with the lakehouse and access lakehouse data through the SQL endpoint. Here’s an example of sharing a lakehouse: Depending on the item being shared, you may find a different set of permissions that you can grant to recipients when you share. Read permission is always granted during sharing, so the recipient can always discover the shared item in Data Hub and open it. Here’s an example of sharing a warehouse: Sharing a Kusto database You can also grant or revoke permissions on specific items to users by selecting manage permission from the context menu. For example, you can grant the reshare permission on a Lakehouse to a user in the Workspace Contributor role. This would allow the contributor to share the lakehouse with users outside of the workspace. Note that you cannot modify the permissions inherited from a workspace role. Select manage permission from the context menu 2. Select Add user on Direct access tab and enter the names of users or groups that you want to provide access to. 3. Select grant To learn more about sharing Fabric items, read: Share items in Fabric. For more information on sharing of specific items, please read Share your Warehouse and manage permissions and How lakehouse sharing works? Synapse Data Warehouse Automatic updates for statistics SQL statistics are now automatically updated by the query engine! In the Fabric Data Warehouse and Lakehouse SQL Endpoint, statistics are a critical tool for helping your query run quickly and efficiently. When a query is executed, the engine will try to collect existing statistics for certain columns in the query and use that information to assist in choosing an optimal execution plan. Today, column statistics are automatically generated when the query engine requires statistics on columns that don’t yet have any. When data in your table’s columns changes significantly, it’s important that those statistics objects also update to accurately reflect the new data. Previously, this meant users would need to manually update on a regular basis. Now with automatic statistic updates, any statistics required during a user query are automatically assessed and refreshed if determined as outdated, allowing your query to use the most precise plan for execution and ensuring your workload is positioned for the best performance possible – all with zero user intervention. For more information on automatic statistic updates, Statistics - Microsoft Fabric | Microsoft Learn UPDATE FROM and DELETE FROM T-SQL You can now update and delete data in your target table from an existing source table using the FROM argument in your UPDATE and DELETE scripts in Fabric SQL respectively! These T-SQL commands allow you to perform MERGE T-SQL like operations. For more information, see UPDATE (Transact-SQL) - SQL Server | Microsoft Learn and DELETE (Transact-SQL) - SQL Server | Microsoft Learn Optimization in data movements during query execution The Fabric SQL engine creates a query plan that comprises of execution steps and data move operations. Data move operations ensure that the data required for a query step is ready in the location of its execution. With this new optimization in Fabric SQL, the engine makes data movement in intermediate steps of query execution more balanced and improves overall query performance. No user intervention or code change is required to use this optimization; it works out of the box! Dataflows Gen2 from the Warehouse You can now create a new Dataflow Gen2 directly from your warehouse to ingest data. Simply click Get Data --> new Dataflow Gen2. The newly created Dataflow Gen2 artifact will already prepopulate the destination to the warehouse. For more information about creating, ingesting, and transforming data with Dataflows Gen2, see Create your first Microsoft Fabric dataflow - Microsoft Fabric | Microsoft Learn. Zero Copy Table Clones We are pleased to announce that zero copy Table Clones are now available in Public Preview! Zero copy clones are a near-instantaneous metadata only operation that enables you to easily create a copy of your Warehouse table(s) with no additional cost and minimal overhead. Table clones contain a reference to the source table that the clone was created from. Underlying parquet files are not duplicated when a clone is created – under the hood, a fork is created, and the clone behaves as an independent table which can be modified as needed. Any changes made to the source table after it was cloned are not reflected in the clone; similarly, any changes made to the cloned table are not reflected in the source. They are independent of one another. As of today, customers can create a Table Clone within the same schema or to a different schema within the same Warehouse. Customers looking to create a Zero Copy Table clone would typically do so for a variety of test, development and production use cases and experimentation. For example, a customer may want to stage a production release in a table clone prior to going live with the changes. Once they are happy with the changes, they can be merged to the production table. For more information on Table Clones, see Clone table in Microsoft Fabric Data warehouse sharing We are excited to announce that Data warehouse sharing is now available in Public Preview! Data sharing is essential to fostering a data-driven culture within an organization. Sharing a Warehouse allows you to easily provide read access to enable downstream users to consume this data, without making copies of data. With this new capability, an Admin or Member within a Fabric workspace can share a Warehouse with another recipient (AAD user or AAD groups) within your organization. The following are the permissions that are provided: [Default] Connect permissions to the warehouse – This option is provided by default and it provides permissions to connect to the warehouse (equivalent of Connect permissions in SQL) but not query any table or view. You can grant granular object access using GRANT in T-SQL. [Default] Build reports on the default dataset - This option is provided by default and provides “build” permissions on the default dataset that is connected to your Warehouse. This option can be useful for your Power BI developers who want to create reports on this default dataset. [Optional] Read all data using SQL – This option provides “readData” (equivalent of db_datareader) permissions which allows for read access to all tables and views within the Data warehouse. This option can be useful for users who want to read using SQL. [Optional] Read all data using Apache Spark – This option provides “readAll” permissions which allows read access to the Warehouse’s underlying files in One Lake that you can read through Spark. This option can be useful for your data scientists who want to read using Spark. For more information, see the [detailed sharing blog]. Dbt Support for Fabric Data Warehouse We are excited to announce the preview of dbt plugin adapter for Synapse Data Warehouse in Microsoft Fabric (preview). This data platform-specific adapter plugin allows you to connect and transform data in Synapse Data Warehouse in Microsoft Fabric. For more information, see the Introducing the dbt adapter for Synapse Data Warehouse in Microsoft Fabric, Microsoft Fabric Synapse Data Warehouse dbt adapter setup and Microsoft Fabric Synapse Data Warehouse dbt adapter configuration. Data Engineering Load to Table Feature Updates Previously the "Load to Table" feature allowed users to load a single file to a new table, which was very well met with data engineers due to the added productivity value of quickly using simple right click actions to enable table loading on Files and as well as because of the no-code experience, which lowers the entry bar for all personas. This new release brings improvements to this experience with several new functionalities: Folder-level load: Users can now load all files under a folder and its subfolders at once by selecting "Load to Delta Table" after clicking on a folder. This feature automatically traverses all files and loads them to a Delta Table. Load to existing table: User can now choose to load their files and folders to a new or an existing table of their choice. If they decide to choose to load to an existing table, they have the option to either append or overwrite their data in the table. Source file option: User can specify if their source file includes the column names as the first row of data and the separator used in it. For more detailed information on this feature, visit the documentation here. Lakehouse sharing and permission management We are announcing a capability for users with Admin and Member roles to share an individual Lakehouse with users without providing them access to the workspace. That will grant access to a specific lakehouse without exposing other items in the workspace. The users will get access to shared items through Data Hub or the link included in the sharing notification email. With access to a lakehouse, users can access the SQL endpoint and default dataset. That provides features like querying data using T-SQL and building Power BI reports on top of the lakehouse data. Permission management also allows users in Viewer role to get additional permissions to access Lakehouse data using Spark. These features will push data democratization in Fabric even further, enabling more collaborative work. Notebook Resources We are excited to announce the "Notebook resources" on Fabric notebook. This feature offers notebook users a writeable file system space where you can store small-sized files, such as code modules, datasets, and images. You can access them with code in the notebook as if you were working with your local file system. The Notebook Resource explorer provides a Unix-like file system to help you manage your folders and files. You can use common operations such as create/delete, upload/download, rename, duplicate, and search through the UI, and rich built-in snippets are provided through “Drag & Drop”. For more details please see How to use notebooks - Microsoft Fabric | Microsoft Learn. Notebook: Show cell output after session reconnection Fabric notebook now supports displaying the running cell output after reconnecting to the original session. This feature allows you to easily recover your ongoing work after accidentally closing the browser or leaving the live session. You don't need to take any additional action to enable it, as this feature is available by default on the Notebook. Notebook: Sharing and permission management Starting now, sharing a single notebook with your colleagues is easier than ever before, without having to grant workspace permissions. With the Notebook sharing feature, you can collaborate with team members and share your work conveniently. Additionally, we now support managing permissions for each Notebook instance. You can easily check and update the permissions of notebooks after they have been shared, ensuring proper access to the notebook. Notebook: New status bar Notebook recently upgraded the design of status bar. In this upgrade, we have styling refinement as well as functional enhancements. Now you can easily discover the “Save options” status, you can easily navigate to the failed cell, and you can find more useful information in the floating info card - the diagnostic information is especially helpful when you encounter issue and need service support! In the next release we’ll add another series of quick access entries on the status bar, so stay tuned for the upcoming new features! Data Science Microsoft Fabric and SynapseML Offer New Features for Large Language Models Microsoft Fabric and SynapseML has featured a variety of new features that have simplified the use of large language models. In our latest release, we have integrated the Azure OpenAI service with the distributed machine learning library SynapseML, which makes it easy to use the Spark distributed computing framework to process millions of prompts with the OpenAI service on Microsoft Fabric. The new OpenAI APIs that have been introduced into SynapseML include “OpenAICompletion”, “OpenAIEmbeddings”, “OpenAIChatCompletion”, and “OpenAIPrompt”. The following helps to better understand these new OpenAI APIs. Leveraging the new OpenAI APIs in SynapseML and Microsoft Fabric, we have also demonstrated how to perform Q&A on PDF Documents. You can read more about it here. Please note that native access to the Azure OpenAI service will be coming to Microsoft Fabric later this year. Real-time Analytics KQL Database Create a Continuous Data Connection from Event Hubs to KQL Database Azure Event Hubs is a big data streaming platform and event ingestion service that can process and direct millions of events per second. Now you can easily stream your Azure Events Hubs data directly into your Fabric KQL Database. There are 2 main steps required to stream the Event Hubs data to the KQL Database: Create a Microsoft Fabric platform-based data connection to a specific event hub instance. This data connection can be used across all Microsoft Fabric workspaces and is managed centrally. Connect this Microsoft Fabric-based data connection to a KQL database. This process creates a database-specific Event Hubs data connection. The connection streams data into the table you specified during setup, and the data will then be available to query using a KQL queryset. A pre-requisite for creating a cloud connection in Microsoft Fabric is to set a share access policy (SAS) on the event hub and collect information to be used later in the setting up the cloud connection. This step is performed in the Azure portal. Go to your event hubs instance and under Settings select the Shared access policies. Add a new SAS Policy or select an existing one: To Create the Cloud Connection from Azure Event Hubs and Fabric, go to the menu bar of your Fabric workspace and go to Manage Connection and gateways. In the New Connection form, you will need to enter details taken from the defined SAS policy. The final step is to connect a table in your KQL database to the event hub cloud connection defined above. This can be done by selecting the Event Hubs option in the Get Data menu bar: For more details, go to the docs: Get data from Azure Event Hubs. Query KQL Database from MATLAB In manufacturing and energy verticals, MATLAB is still heavily used. MATLAB is a programming and numeric computing platform used to analyze data, develop algorithms, and create models. We are happy to announce that Fabric now supports the querying of KQL database data directly from MATLAB. Now engineers that are familiar with MATLAB do not need to learn KQL to query the high performant RTA KQL database, but rather reuse their MATLAB skills to query the data. This functionality is supported by a very lightweight MATLAB Connector that securely connects MATLAB to a Fabric KQL Database. For more details, go to the docs page: Query data using MATLAB. Data Factory Dataflows Gen2 Power Query Editor Enhancements to output destination experience Auto-fix column names during mapping of columns dialog Auto-fix data types during mapping of columns dialog Improvements to existing experiences Rename a dataflow inside of the Power Query Editor A similar experience to other artifacts inside of Microsoft Fabric, you can now change the name of a Dataflow Gen2 inside of the Power Query Editor. Connectors Google Analytics (Connector Update) The Google Analytics connector has been updated to support Google Analytics Data API (Google Analytics 4). To use this new functionality, use “Implementation 2.0” when connecting. Existing connections will not be affected. Oracle Database (Connector Update) The Oracle connector has been updated to enable Azure AD-based Single Sign-On functionality through the on-premises data gateway. This will require the July release of the on-premises data gateway. Azure Databricks, Databricks (Connector Update) The Azure Databricks and Databricks connectors have been updated. Please find notes from the Databricks team below. Add a new DSRHandler to databricks-multicloud Fix UC_NOT_ENABLED and Catalog 'spark' not found error in legacy code path using Databricks.Contents Denodo (Connector Update) The Denodo connector has been updated. Please find notes from the Denodo team below. This new version adds graphical support for the specification of native SQL queries at data source creation time EQuIS (Connector Update) The EQuIS connector has been updated. Please find notes from the EQuIS team below. Remove "Beta" attribute Retrieve report content as .csv to remove the row limitation of .xlsx files Optimize handling of facility groups in navigation tree Show report and/or location folders in navigation tree even if one or the other is empty Snowflake (Connector Update) The Snowflake connector has been updated to include various performance improvements, such as usage of SQLBindCol. Users should experience better performance when running queries. Anaplan (Connector Update) The Anaplan connector has been updated. Please find notes from the Anaplan team below. This version of Power BI connector for Anaplan includes backend changes for compatibility with ongoing Anaplan infrastructure updates. There is no change to user facing connector features. Gateway and Connections SSO option in cloud connections Added support for Single Sign-On (SSO) via Azure AD in cloud connections. Currently this feature is only applied for DirectQuery mode in datasets. However, we plan to progressively extend SSO capabilities to other Fabric workloads in the future. Flag in cloud connections to allow them to be with gateway We have enhanced our security measures to allow users to disable cloud connections to be used with gateway connections. This precaution prevents the decryption and logging of cloud connection credentials within on-premises systems. Data pipelines Activities Teams Activity We’re excited to announce that the Teams activity is now available to use in your Data Factory data pipelines. In your pipeline, you can use the Team activity to customize a message to send to a Teams channel or a Teams group chat. For example, you can use the Teams activity to send a notification if a pipeline has failed, helping you to better monitor your data integration pipelines. Productivity Artifact parameterization support We’re excited to share that you can now parameterize Fabric artifacts in your data pipeline. This will allow you to use expressions, functions, parameters, and variables to dynamically refer to your Lakehouse or Data Warehouse, allowing more flexibility when designing your data pipelines. Performance improvement for moving parquet files to Lakehouse tables We’ve made performance improvements when copying parquet files to your Lakehouse! “Save as” support now available from your workspace “Save as” is now supported by data pipelines from your Data Factory workspace. When you are in your workspace, you can use the selection menu to save a new copy of your pipeline. This will allow you to build upon or edit existing pipelines without having to completely rebuild a data pipeline. Support column mapping for Lakehouse destination in Copy activity Data pipelines now supports column mapping when a Lakehouse is selected as a data destination. In the Mapping tab, you can now add, edit, or delete column mappings from your data source to your data destination. Data Activator Sample data The simulator that generates sample data now includes multiple event streams that you can use to try out building objects and alerts. If you want to try the simulator, check out our tutorial: End-to-end tutorial using simulated data. Adding multiple streams or properties The Data screen now gives you options to create objects directly from columns in your data streams, and gives you shortcuts to quickly create properties directly rather than having to create them and manually select the column. We’ve also planned a new UX to let you create multiple properties or objects across multiple streams in one go – stay tuned for that in the coming months! Designing triggers and alerts We’ve combined the first two steps of triggers (Reference to a property and selecting the value from an event) into one step. This means the first thing you’ll see is a chart that plots the values you select helping you understand the data more quickly. It’s currently called “Property field reference”, which will be updated to be more user-friendly soon! We’ve added many more functions to the library you can use to build triggers: Detect functions. Changes, Changes from/to Sends an event each time the field changes from/to a value, or changes at all. Is less than/greater than, Is false/true, Is equal/not equal Sends an event whenever the field meets that condition. Becomes less than/greater than, becomes true/false, Exits/Enters range Sends an event the first time the field meets that condition (if subsequent events also meet that condition, no further events are emitted) Summarize functions. Maximum, minimum, average, count over time. Calculates a aggregation of the values of all events in a time window. Filter functions Filter Filters individual events where the value meets a specified condition. Other events are dropped. This is useful for filtering out error values such as a sensor that returns -99 for errors. Property filter Filters out instances that match the criteria. This is useful if you only want a trigger to fire for certain instances (this will be more usable when the trigger changes described below are done). ‘Detect’ functions such as ‘Crosses above’ or ‘Changes to’ also have options to only trigger when the criteria is met a certain number of times, for example 3 times in an hour. This can help reduce noise in your triggers. Coming soon, we have a major change to the way triggers are defined. Early preview feedback showed that the detailed step-by-step setup for triggers was too complicated, so we’ve geared them around selecting 3 things: what you what to monitor, what condition you want to detect, and what action to take. This should make it easier to configure your triggers and make it clearer what’s needed to get alerts on your data. Trigger lifecycle Based on feedback to make the management of triggers simpler, we removed the Draft concept. You don’t need to Publish a trigger before you can start it. Triggers are either just Started or Stopped! Any time you make changes, you select Update to change the running trigger to use the new values. Testing actions The ‘Test action’ button confused early users so we’ve updated the messaging to clarify that it will send a test to the current user. It’s also only enabled if there is some data that met the criteria, which is used as sample data in the test message. Teams app Data Activator can now send alerts to you from our new Teams app. To install it, search the Teams store! A Teams administrator can set it up for everyone in your organization if you need to deliver messages broadly, rather than each user installing it themselves. Community Implementing a Lakehouse with Microsoft Fabric Courses This course is designed to build your foundational skills in data engineering on Microsoft Fabric, focusing on the Lakehouse concept. This course will explore the powerful capabilities of Apache Spark for distributed data processing and the essential techniques for efficient data management, versioning, and reliability by working with Delta Lake tables. This course will also explore data ingestion and orchestration using Dataflows Gen2 and Data Factory pipelines. This course includes a combination of lectures and hands-on exercises that will prepare you to work with lakehouses in Microsoft Fabric. Check out the course: Course DP-601T00: Implementing a Lakehouse with Microsoft Fabric Filter by product for User Groups & Events You can now filter by product for both Fabric User Groups and Events in the Fabric Community site. With the broad range of experiences within Fabric, product filtering allows you to easily find User Groups or events that match your interests! You can also quickly refilter your results by removing products either in the Product drop down or with the tiles below the search bar. Data Activator Community now live Data Activator now has its own community forum, alongside the other product forums across Fabric: https://community.fabric.microsoft.com/t5/Data-Activator-preview-Community/ct-p/dataactivator (or use the short URL https://aka.ms/dataActivatorCommunity!). If you’ve got any questions or feedback about Data Activator you can post there and the product team will be in touch. Power BI Reporting Smoothed and Leader lines – Updates We are excited to announce the launch of one of our most highly acclaimed features! Report creators can now create smoother line and area charts, providing a more polished look to their visualizations. To access this setting, go to Lines > Shape > Line Type. We've recently added leader lines for both line and area charts. This new feature creates a visual connection between each data point and its corresponding label. To access this feature, simply navigate to the Data labels > Options > Leader lines. These features are just the beginning of the many improvements we have in store for graphs, charts, plots, and markers in the coming months. Get ready for even more exciting updates! On-Object Interaction (Preview) – Updates The new on-object interaction feature released to preview back in March. This month we bring more improvements and bug fixes. Customize the Pane Switcher We’ve now added a new “+” button on the pane switcher to quickly add new panes directly from the pane switcher without having to go to the View ribbon. This menu also gives a brief description of what panes are available and what their functions are. Even better, the panes added to the switcher are saved across reports. Configure once and you’re done! You can also access the 2 preference settings released last month for “always show the pane switcher” and re-attaching the build menu as a pane by using the gear icon. CTRL click to open multiple panes in the Pane Switcher In addition to the right click option “open in new pane”, it is now even easier to open multiple panes from the pane switcher by simply holding down the CTRL key and clicking the pane you wish to open. Treemap sub-selections now supported: Noteworthy bug fixes Overlap of the on-object buttons on the formula bar has finally been resolved! We appreciate your patience as this bug was a bit trickier to fix the right way. Visual tooltip showing automatically when opening the build menu, blocking the formatting on-object button is now fixed. Selected visual type is now reflected in the ribbon visual gallery accordingly. Mini-toolbar’s fill color icon now reflects conditional formatting gradient as well. Thanks for continuing to try out the new preview and provide feedback. We’re working hard to react to your suggestions and add the necessary changes to make on-object work for you. Please continue to provide your comments directly in this blog post or in our community forum via the “Share feedback” button next to the preview switch. Modeling Edit your data model in the Power BI Service– Updates (Preview) The new data model editing in the Service feature released to preview in April. We've been busy reacting to your feedback and enhancing the experience. Below are the improvements we are adding this month: Relationship validation We are adding relationship validation in the Service, making it easier to create and edit relationships in the web! Like Power BI Desktop, as you define the properties of your relationship, the system will automatically validate it and offer appropriate choices for cardinality and cross filter selections. Please continue to submit your feedback directly in the comments of this blog post or in our feedback forum. Service Dataset details page revamp We are happy to announce the revamp of our dataset details page! Now, when you click on a dataset item in the OneLake data hub and workspace view, you will be directed to the redesigned page that not only enhances the look and feel but also introduces new capabilities for an improved user experience. Here's what you can expect to find on the dataset details page: Actions: You will find various actions that can be performed on the dataset, such as creating a report and refreshing the dataset. With this release, we have added the option to view the refresh history under the refresh menu. Dataset Metadata: Gain insights into the dataset through its description and last refresh time. Related Items: Explore existing related items associated with the dataset. Dataset Schema: Get a comprehensive view of the dataset's tables and columns. Clicking on a table provides a table preview, with export capabilities available using paginated reports behind the scenes. Additionally, we have made significant improvements to the related items list. It now showcases all the downstream and upstream dependencies for the dataset. This enhancement allows you to easily identify the sources of the dataset, composite model relations, reports, and dashboards associated with it. We believe that these updates will greatly enhance your experience with the dataset details page, providing you with a more intuitive and comprehensive understanding of your data. We look forward to your feedback as you explore these new features!" Mobile Datasets are coming to Power BI Mobile apps! In the next Power BI Mobile app release, we are adding a long-waited feature that will help dataset owners and report creators to manage their dataset directly from their phone. That means that you will be able to see in your mobile device datasets. Go to a workspace, make sure to select the “dataset” pill at the top and get the list of datasets, that you have access to in that workspace. When tapping on a dataset, you will get the dataset metadata pane, which includes the name, owner, sensitivity label and also the latest refresh status. From this pane you can also trigger a dataset refresh – all directly from your mobile app! Dataset owners will also get push notifications when schedule refresh fails. They will be able to view the failure details and be able to re-try the refresh while they are on-the-go. Developers Boost Your Custom Visuals Performance: Check out our Latest Article! We have recently published an article that focuses on techniques to improve the performance of custom visuals. In this article, we discuss the performance improvements we have made in visual rendering and load times. We identified and addressed certain bottlenecks in the code, and these improvements are available for any visual that has been updated to API version 4.2 and onwards. Along with these fixes, we also provide code practices and techniques that can greatly enhance the performance of rendering custom visuals. I encourage you to check out the article here. We believe that these techniques can make a significant impact on the performance of your custom visuals. Visualizations New visuals in AppSource Spider Chart for Power BI by VisioChart HTML Content (lite) Stacked Lipstick Bar Chart (Standard) Stacked Lipstick Column Chart (Standard) 100% Stacked Column Chart with Values instead of % (Standard) Dual Axis Scatter Chart (Standard) Category Comparison Bar Chart Stacked Column with Percentage and Total in Label (Standard) Likert Scale Chart for Power BI by ChartExpo Elevate Your Map Charts With Drill Down Map PRO Drill Down Map PRO by ZoomCharts is a custom map visual for Power BI that lets you show your data on an interactive map and give it location-based context. Features include: Built-in shape layers – use preset shapes for easy filtering of countries. Custom shape layer support – provide custom shapes through KML and GeoJSON files. Lasso tool – draw and save your own filter shapes on top of the map. Node clustering capabilities – clusters can be turned into donut or pie charts for category display. Map base layer lets you choose from 4 options – Azure maps, Custom (OpenStreetMaps, Google, CartoDB etc.), Image (e.g., floor plans), None (visualize shapes without a background). Aura, image, and custom label support. Popular use cases: Production – monitoring production data by location. Sales and marketing – mapping sales results by region. Public sector – visualizing environmental and sociodemographic data. Get Drill Down Map PRO now! Learn More about Drill Down Map PRO. ZoomCharts Drill Down PRO Visuals are known for their interactive drilldowns, smooth animations, rich customization options. All Drill Down PRO Visuals support: touch input devices, interactions, custom and native tooltips, filtering, bookmarks, and context menu. We are ready to present our new visual: an "all-in-one" Multi Target KPI card The Multi Target KPI card works with a single query and includes three additional indications, multiple categories, a pixel perfect alignment setting, and built-in conditional formatting. You can change the settings of layout type and color conditional formatting for additional measures in our visual, and it is fairly simple for non-designers to use! Just select the desired measure and category, if necessary. Add up to three additional indicators to provide the context you need for your metric. It will help you improve reporting performance and save you time when designing and developing supplemental measures. Start a new level of business dashboarding! Link to AppSource: https://appsource.microsoft.com/en-us/product/power-bi-visuals/ndividualentrepreneuralekseikolokolov1654522457293.80416480?tab=Overview Link to our website: (delete if not needed) https://data-2-speak.com/cardkpi Other WebView2 GA WebView2 is now generally available. Thanks to everyone who reported issues during the preview phase! Your input helped us improve reliability to higher than it was before we introduced WebView2. Please continue to report any issues using the “There was a problem with WebView2” dialog.Microsoft Fabric August 2023 update
Welcome to the August 2023 update. We have lots of features this month including the new layout switcher for Power BI, SSD caching in Synapse Data Warehouse, in-line Python support for KQL in Synapse Real-time Analytics, lookup activity for Data Factory Dataflows, and much more. Continue reading for more details on our new features! Contents Core Multitasking navigation improvement Monitoring Hub support for personalized column options Power BI Reporting New layout switcher New bubble range scaling setting On-Object Interaction (Preview) – Updates Resizing and positioning to the on-object menus On-object format subselections now supported in spotlight and focus mode Modeling Updates to ORDERBY function Data Connectivity New modern data connectivity and discovery experience in Dataflows Lakehouses (Connector Update) Service XMLA Write support for Direct Lake datasets Automatic replica synchronization for Dataset Scale-Out (preview) Mobile Choose your start-up content Developers Published AppSource visuals now generate leads in Partner center and CRM Publish a Power BI Project (PBIP) directly from Power BI Desktop Visualizations New visuals in AppSource Editor's pick of the quarter Drill Down Combo Bar PRO by ZoomCharts Sunburst Chart by Powerviz xViz Performance Flow by Lumel Synapse Data Warehouse SSD Caching Data Engineering High Concurrency Mode for Notebooks Data Science Model & Experiment Usability Improvements New Data Science Sample are Coming to Microsoft Fabric Real-time Analytics KQL Database support for inline Python KQL Database provisioning in a few seconds Data Factory Dataflows Gen2 Power Query editor Edit connection in Manage Connections Enable Staging labels Publish dataflow without staged queries Limit to number of queries in a Dataflow when publishing Refresh history Direct navigation to last failed refresh Statistics for tables Connectors Lakehouse connector update Other improvements Data pipelines Connectors FTP Connector Productivity Lookup activity for Lakehouse, Data Warehouse, and KQL Database Get Metadata activity for Lakehouse and Data Warehouse Pipeline run status now available Secure input and output options for Set Variable activity Data Activator Trigger and property design template Quickly assign columns as properties in an object Data Activator now supports Power BI visuals with a time axis Trigger Power Automate flows from Data Activator Core Multitasking navigation improvement In our latest update, we've introduced an enhancement that preserves all Fabric items opened in a single browser tab on the left navigation bar, even in the event of a page refresh. This ensures you can refresh the page without the concern of losing context. Monitoring Hub support for personalized column options We have updated Monitoring Hub to allow users to personalize activity-specific columns. You now have the flexibility to display columns that are relevant to the activities you're focused on. Power BI Reporting New layout switcher We’ve added new buttons that make it easy for you to quickly switch between web and mobile layouts while you’re developing your reports. You'll find the new switcher buttons down at the bottom of the screen, right next to the page navigator. New bubble range scaling setting We are introducing the new bubble range scaling setting for scatter chart and map visuals! This setting gives report creators more control over how the bubble (marker) sizes respond to the data, making it more accurate or distinctive based on preference. With the magnitude option, the bubble areas closely follow the data proportions. With the data-range option, the bubble size limits are mapped to data minimum and maximum. The auto option, which is the default setting for new reports, selects the appropriate option based on data characteristics. For more information, visit our docs. This setting can be adjusted in formatting panel, Markers > Shape > Range scaling for scatter charts or Bubbles > Size > Range scaling for maps. For reports authored in earlier Power BI versions, these settings default to (Deprecated) for scatter charts (which differs in handling negative values), and Data range for map charts. Azure Maps charts will also include this feature in a coming product update. In the figure above, the size of each country represents Urban Population, which is also shown on the y-axis. On-Object Interaction (Preview) – Updates The new on-object interaction feature released to preview back in March. We’ve been busy adding even more improvements to the preview, here’s what’s part of the Aug release: Resizing and positioning to the on-object menus We’ve now added the ability to resize the on-object menus horizontally, this is especially helpful when you’re working with long field names. We’ve also improved the positioning of the on-object menus to make better use of the canvas space. Previously, when a visual was near the bottom of the canvas, the on-object menu was super small and required scrolling to be able to see and use the field wells. Now, the on-object menu moves up and stretches into the canvas to bring the field wells into view without needing to scroll. Before After On-object format subselections now supported in spotlight and focus mode When spotlighting a visual or expanding the visual in focus mode, you can now use on-object formatting to subselect and format styles. In focus mode, it can be hard to tell when you’ve entered format mode with just the subtle border. To address this, we’ve added a button to the header to better indicate when you’re in format mode and how to exit format mode while staying in focus mode. Thanks for continuing to try out the new preview! We’re working hard to react to your suggestions and add the necessary changes to make on-object work for you. Please continue to provide your comments directly in this blog post or in our community forum via the “Share feedback” button next to the preview switch. Modeling Updates to ORDERBY function If there are blanks in the data, you can specify where to order them by adding ‘BLANKS LAST’ or ‘BLANKS FIRST’. For example, this is a perfectly valid expression: BiggestSpender = SELECTCOLUMNS ( INDEX ( 1, ALLSELECTED ( 'DimCustomer' ), ORDERBY ( SUM ( 'FactInternetSales'[SalesAmount] ), DESC BLANKS LAST) ), 'DimCustomer'[FullName] ) Specifying how blanks are handled is optional and can be combined with specifying the order direction (DESC/ASC). Valid values include: BLANKS DEFAULT. This is the default value. The behavior for numerical values is blank values are ordered between zero and negative values. The behavior for strings is blank values are ordered before all strings, including empty strings. BLANKS FIRST. Blanks are always ordered on the beginning, regardless of ascending or descending sorting order. BLANKS LAST. Blanks are always ordered on the end, regardless of ascending or descending sorting order. Read more in our documentation. Data Connectivity New modern data connectivity and discovery experience in Dataflows We are excited to introduce the new data connectivity and discovery experience in Dataflow, Dataflow Gen2, and Datamart. Today, users spend a lot of time finding the right data, the right connection info and credentials. With the new Get Data experience, we make it easy to browse different Fabric artifacts through the OneLake data hub. This improved experience aims to expedite this process and get you closer to the data that you’re looking for in the quickest way possible. Learn more in our blog post Announcing a new modern data connectivity and discovery experience in Dataflows Lakehouses (Connector Update) This update includes significant performance improvements to the Lakehouses connector. Be sure to update to the August version of Power BI Desktop and Gateway to experience these improvements! Service XMLA Write support for Direct Lake datasets We are excited to announce that Direct Lake datasets now support XMLA-Write operations. Now you can use your favorite BI Pro tools and scripts to create and manage Direct Lake datasets. Whether you prefer SQL Server Management Studio (SSMS), Tabular Editor, DAX Studio, or something else, you can connect to your Direct Lake datasets using XMLA endpoints and perform operations such as deploying, customizing, merging, scripting, debugging, and testing. You can use tools like Azure DevOps or GitHub to implement source control, versioning, and continuous integration for your data models. You can automate and streamline your development and deployment processes. You can also use PowerShell or REST APIs to automate tasks such as refreshing or applying changes to your Direct Lake datasets. XMLA Write is incredibly powerful and the key to data modelling efficiency and productivity. For more information about XMLA Write support in general, check out the article Dataset connectivity with the XMLA endpoint in the product documentation. Automatic replica synchronization for Dataset Scale-Out (preview) We are excited to announce that we have finalized Dataset Scale-Out configuration APIs and completed the replica synchronization feature. Specifically, you no longer need to enable Scale-Out at the workspace level by using a burdensome XMLA request. The XMLA command is deprecated and will no longer work. You can now enable Scale-Out on a dataset-by-dataset basis using the Power BI REST API for datasets. You also no longer need to synchronize read replicas manually if you want to take advantage of automatic replica synchronization. Automatic replica synchronization is enabled by default. However, it is also possible to disable automatic synchronization to synchronize the read/write and read replicas of a dataset manually for controlled refresh isolation. Learn more in the automatic replica synchronization announcement blog Mobile Choose your start-up content Power BI mobile app users can now choose which item they want to have open automatically whenever they launch the Power BI Mobile app. This feature saves time for users who mostly view a specific item on their mobile app, and don't want to waste time navigating from the app’s home page every time they open the app. To configure a launch item for yourself, open the item you want to see when you launch the app. This can be a specific report page, dashboard, scorecard, report in an app or entire app. When the item is open, open the More options (…) menu from the header and select Set as launch item. This will mark the item as the launch item. Only one item at a time can be marked as the launch item. Administrators can also use mobile device management (MDM) tools to remotely configure a launch item for a group of users (front-line workers, for example) to simplify their experience with the app. Developers Published AppSource visuals now generate leads in Partner center and CRM Until recently, your customer leads may have come solely from customers downloading them from AppSource.com. However, now you have access to even more leads through Power BI - both the desktop and web embed AppSource. To access these new leads, simply navigate to the Referrals workspace in Partner Center. Here, you can see all the leads you receive from Power BI, as well as those from AppSource.com. Plus, if you've connected your CRM, you'll see them there too. By utilizing these new leads from Power BI, you can potentially reach more customers and increase your business's growth. So be sure to check your Referrals workspace and CRM regularly to stay on top of your leads! Publish a Power BI Project (PBIP) directly from Power BI Desktop You are now able to directly publish your Power BI Project (PBIP) files directly from Power BI Desktop, eliminating the need to save it as PBIX in order to activate the Publish feature. From an opened Power BI Project, choose: File > Publish > Publish to Power BI or select Publish on the Home ribbon. Select the destination workspace: And that's it, your PBIP dataset and report (or just report if it's a Live Connect) will be published to the selected workspace: To learn more about Power BI Projects, visit: https://aka.ms/pbidesktopdevmode Visualizations New visuals in AppSource The following are new visuals this update: Performance Flow - xViz Time-lines by BI-Champ Composed Line Area Bar Chart by Devlup Funnels Galigeo For Power BI Radial chart by Devlup Funnels Editor's pick of the quarter accoPLANNING Enterprise - Planning Power BI Writeback Sunburst Chart by Powerviz Spider Chart for Power BI by VisioChart Advanced Trellis / Small Multiples - xViz Drill Down Combo PRO Zebra BI Cards Shielded HTML Viewer Drill Down Combo Bar PRO by ZoomCharts Drill Down Combo Bar PRO by ZoomCharts offers a wide selection of customization options, letting creators build everything from regular bar charts to box and whisker plots. This visual also offers powerful cross-chart filtering capabilities combined with intuitive on-chart interactions. MAIN FEATURES: Multiple chart types – choose between column, line, and area charts Full customization – customize X and Y axes, legend, outline and fill settings Stacking and clustering – choose normal, 100% proportional, or zero-based stacking Static and dynamic thresholds – set up to 4 thresholds Multi-touch device friendly – get the same experience on mouse and touch input devices POPULAR USE CASES: Sales and marketing – sales strategies, sales results, and campaign-by-campaign marketing metrics Human resources – hiring, overtimes and efficiency ratios by department Accounting and finance – financial performance by region, office, or business line Manufacturing – production efficiencies and quality metrics by product line ZoomCharts Drill Down Visuals are known for their interactive drilldowns, smooth animations, and rich customization options. They are mobile friendly and support: interactions, selections, custom and native tooltips, filtering, bookmarks, and context menu. Try Drill Down Combo Bar PRO now by downloading the visual from AppSource. Learn More about Drill Down Combo Bar PRO by ZoomCharts. Sunburst Chart by Powerviz We are thrilled to present the new Sunburst Chart by Powerviz, a powerful visualization designed to display hierarchical data in a user-friendly and intuitive format. With its concentric circle design, you can easily display part-to-whole relationships and gain valuable insights from your data. Key Features: Rich Customization: Control display style, labels, Center circle, fill patterns and dynamic images. Color Options: Choose from 30+ color palettes, including color-blind safe options. Ranking: Easily filter the Top/Bottom N by each level and show the remaining categories as "Others". Conditional Formatting: Easily identify outliers based on value or category rules. Interactive Features: Enjoy full interactivity with zoom, drill down, and cross-filtering for detailed exploration. Many other features included annotations, grid view, show condition, and accessibility support. Business Use Cases: Sales and Marketing: Market share analysis and customer segmentation. Finance: Department budgets and expenditures distribution. Operations: Supply chain management, identify inefficiencies in manufacturing process. Education: Course structure, curriculum creation. Human Resources: Organization structure, employee demographics. Check out our video Introducing Sunburst Visual by Powerviz – A Powerful Power BI Custom Visual Get Sunburst Visual for FREE from AppSource Check out the visual features in sample file. Step by Step instructions and documentation can be found here. To learn more, visit Powerviz website xViz Performance Flow by Lumel xViz Performance Flow by Lumel is an integrated business flow monitoring visual with an interactive KPI tree visualization for organizational performance management use cases. It offers insights into People, Places, Processes and Entities with Performance Indicators, Trendlines, and Advanced Alerting on Goals, Metrics and their Variances. Performance Flow is suitable for a wide range of use cases: Organization Performance - Unveil HR/Employee Performance insights through an interactive pictorial Org Tree Chart with performance attributes and trends. Process Flow - Visualize business flows in various stages using Swim lanes with connector lines, icons, KPI metrics, and trends in one single integrated view. Financial Performance – Cost Center analysis with Scorecards or KPI Trees. Sales Performance - Dive deep into Sales Performance at Regional, Departmental, and Salesperson Levels with Decomposition Trees. The visual is packed with analytical and interactive features like: Interactive hierarchy navigation including quick search, zooming, subtree analysis, etc. Streamline Processes using Swim Lanes like in Visio Uncover hidden/Dotted connections using Links Data-driven conditional formatting rules Custom Tabs for end-users Try xViz Performance Flow visual from Microsoft AppSource today. To know more about the visual refer to our website Performance Flow and our video xViz Performance Flow - Introduction Synapse Data Warehouse SSD Caching T-SQL queries targeting massive amounts of data not fitting in-memory cache suffer from cache misses and higher latency due to repetitive reads from remote storage. SSD caching stores frequently accessed data on local disks in highly optimized format, significantly reducing IO latency and accelerating query processing. For more information check https://learn.microsoft.com/en-us/fabric/data-warehouse/caching. Data Engineering High Concurrency Mode for Notebooks We are thrilled to announce session sharing in Fabric through High Concurrency mode for Data Engineering and Data Science workloads. You can run notebooks simultaneously on the same cluster without compromising performance or security when paying for a single session. Session sharing is strictly within a single user boundary offering enhanced security and isolation also allowing you to do more while paying less. To learn more about High Concurrency mode in Fabric for Data Engineering and Data Science check out the documentation : High Concurrency mode in Fabric Spark Data Science Model & Experiment Usability Improvements We’ve made several usability enhancements to our Model & Experiment tracking features. You can now stay informed with real-time notifications for model and experiment updates. Plus, users can now enjoy a more seamless browsing and comparison experience with improved Run List and Model List views. New Data Science Samples are Coming to Microsoft Fabric We're thrilled to announce that Microsoft Fabric is introducing two exciting new Data Science Samples that showcase the power of Microsoft Fabric capabilities. The first sample focuses on bank customer churn problem and aims to build a machine learning model to predict whether bank customers would churn or not. The second sample is about machine failure and revolves around the use of machine learning to have a more systematic approach to fault diagnosis to proactively identify issues and take actions before a machine's actual failure. Both these samples provide a comprehensive display of end-to-end data science workflow, demonstrating Microsoft Fabric versatility in addressing diverse real-world challenges with AI-driven solutions. We can't wait to see the incredible possibilities these samples will unlock for our customers! To check these new Data Science samples along with others, please visit Microsoft Fabric and then select Synapse Data Science. Click on Use a Sample to access all Data Science samples. Real-time Analytics KQL Database support for inline Python Fabric KQL Database supports running Python code embedded in Kusto Query Language (KQL) using the python() plugin . The plugin runtime is hosted in a sandbox, an isolated and secured environment hosted on KQL Database compute nodes. This sandbox contains the language engine as well as common mathematical and scientific packages. The plugin extends KQL native functionalities with a huge archive of OSS packages, enabling Fabric users to run advanced algorithms, such as machine learning, artificial intelligence, statistical tests, time series analysis and many more as part of the KQL query. The Python plugin runs a user-defined function (UDF) using a Python script. The Python script gets tabular data as its input, and produces tabular output. The plugin is disabled by default. Before you start, enable the Python plugin in your KQL database. To enable the plugin browse to your KQL Database, select Manage -> Plugins, and enabled the plugin by toggling the button to On. KQL Database provisioning in a few seconds The KQL Database provisioning process has been optimized. Now you can provision a KQL Database within a few seconds. All you need to do to Create a new KQL database, is to give it a name, and after a few seconds you will have a fully functional KQL Database, that you can now start ingesting and querying your data. Data Factory Dataflows Gen2 Power Query editor Edit connection in Manage Connections The Manage Connections feature was recently release with only the capability to view the linked connections to your dataflow and the ability to unlink a connection from your dataflow. We’ve now added the capability to Edit a connection credentials and gateway from within the dialog. We continue to work on improving this experience during this calendar year and we welcome you to give this feature a try and look forward to future improvements on this feature. Enable Staging labels The concept of staging data was introduced in Dataflows Gen2 for Microsoft Fabric and now you have the ability to define what queries within your Dataflow should use the staging mechanisms or not. Learn more about the staging mechanism used in Dataflows Gen2 from the Data Factory Spotlight blog post. Publish dataflow without staged queries With the introduction of the ability to set the behavior of a query to be staged or not, you can now set all your queries to be evaluated without any staging and load the data directly to a destination of your choice. Note that your Dataflow Gen2 must at least have a single query with a data destination defined. The main benefit of this pattern is that your data will not need to be staged first which could potentially save time if the evaluation of your Dataflow might be quick and doesn’t need the overhead of staging mechanisms or it simply goes more in line with how you want your solution to be designed for a number of reasons. Limit to number of queries in a Dataflow when publishing When creating a Dataflow Gen2, we are now modifying the maximum number of entities that can be part of a particular Dataflow. The new maximum number of entities is 50. If you have 51 or more entities in your Dataflow, you will receive a warning letting you know that you need to reduce the number of entities to a maximum of 50 before you can publish your Dataflow. You are still able to save your Dataflow as draft before you make any changes. Refresh history Direct navigation to last failed refresh If you are ever in a situation where your Dataflow refresh failed, you can now click the warning sign right next to the timestamp in the Refreshed column and get taken directly to the Refresh history dialog for that particular refresh attempt. The goal of this new improvement is to reduce the number of clicks to get you to the last failure. Statistics for tables Inside the refresh history dialog you are now able to drill down to a particular table and see the volume processed for it as well as the endpoint where the volume was processed. More information such as Duration, Start Time and End Time are still available to you in this dialog. Connectors Lakehouse connector update This update includes significant performance improvements to the Lakehouses connector. Be sure to update to the August version of Power BI Desktop and Gateway to experience these improvements! Other improvements We are actively listening to your feedback and the feedback of thousands of customers who are trying out Dataflow Gen2 today. Some of the feedback doesn’t directly translate to new features, but rather to fixes or quality improvements to our backend and how reliable our service can be. Our team has been able to triage and work on more than 600 fixes and improvements in the past month. The list below is a small list of some of the categories of these fixes and the impact that it’ll have on how you use Dataflow Gen2 today: Better error messages: We are actively improving the error messages and have improved some of the most common error messages in the past few weeks. There’ll be multiple improvements on the error messages and categorization front in the coming months but we’re happy to be making tactical changes today to improve the experience and make it clear to users on what could be happening to produce the error. Multitasking efforts: There have been some issues in regard to multitasking in Fabric. A few of those have been addressed, but we’re actively working on a much better experience for Dataflow Gen2 with multitasking in mind much similar to how other artifacts leverage multitasking capabilities. Reliability and performance: We’re continuously working towards improving the reliability and performance of Dataflow Gen2 in Microsoft Fabric. This will translate into much faster refreshes times and with more reliability. Data pipelines Connectors FTP Connector We’re excited to announce that the FTP connector is now available to use in your Data Factory data pipelines. In your data pipeline, you can create a new connection to your FTP data source to copy, extract, and transform your data. Productivity Lookup activity for Lakehouse, Data Warehouse, and KQL Database The Lookup activity now connects to Fabric Lakehouse, Data Warehouse, and KQL Database. This now makes it easier for you to read or look up records, table names, and other values from your Fabric artifacts to use in downstream activities in your data pipeline. Get Metadata activity for Lakehouse and Data Warehouse The Get Metadata activity now connects to Fabric Lakehouse and Data Warehouse, making it easy to retrieve metadata from your data in your Fabric artifacts to use downstream in your data pipeline. Pipeline run status now available We’ve recently added Pipeline run status so that developers can easily see the status of the pipeline run. You can now view your Pipeline run status from the Output panel. Secure input and output options for Set Variable activity We’ve added advanced settings for the Set Variable activity called Secure input and Secure output. When you enable secure input or output, you can hide sensitive information from being captured in logs. Data Activator Trigger and property design template As mentioned last month, we have been working on a new experience for designing triggers and it’s now available in our preview! You now see 3 cards in every trigger: Select, Detect, and Act: ‘Select’ is where you choose the value you want to monitor. It can be a direct reference to a column from an event stream, or to an existing property. Once you’ve selected an input, you’ll see a preview and can add grouping/smoothing or filters to get the right value: The Detect card is where you specify the conditions and thresholds that you want to take action on. You select the type of threshold, enter the values and can optionally choose things like firing the trigger every time the condition is met, or only when it’s met a certain number of times over a longer period. The top chart shows, for the instances you have selected in the preview, when this trigger would have fired. The bottom charts shows the overall nubmer of times the trigger would have fired for all instances. This gives you an idea of how many emails or Teams messages would have been generated! Finally the Act card lets you set the action you want Data Activator to take. You can choose the recipient, optional information etc. as you build it out. Quickly assign columns as properties in an object From the data view, you can quickly assign an event stream to a new or existing object and make multiple properties from one UI. You need to choose the key column that identifies the individual object instances that you care about (e.g. a package ID, employee ID, location name etc.). Then, in the ‘Assign properties’ dropdown, select the columns from your event stream that you want to use as properties in the object. You can use the ‘Assign to existing’ option to map a second event stream onto an existing object, combining data from two events. Data Activator now supports Power BI visuals with a time axis Data Activator now supports Power BI visuals with a time axis. In the screenshot below, we create an alert if the “occupancy” measure on the visual goes above 60%. Note that Data Activator has detected the presence of the time axis and is highlighting this in the alert pane. Trigger Power Automate flows from Data Activator When a data activator trigger fires, you can now trigger a Power Automate flow. This means that you can use Data Activator to drive actions in any system that Power Automate can connect to. You can send an alert in a 3rd party alerting system, log a ticket in a ticketing system, or call a REST API to trigger actions in an operational system. The list is (almost) endless! To connect Data Activator to Power Automate, you create a custom action. A custom action is a reusable action template that triggers a flow. Once you have made a custom action, you can use it in any trigger, in any Data Activator workspace or item. Here, we create a custom action to send an SMS message via a 3rd party connector: After creating the custom action you can use it in any trigger, by selecting it in the “Act” card in the trigger designer. The act card prompts you for the input fields that need to go to the flow. Here, we see the action card for our SMS action. It is prompting for the phone number and message: For the trigger creator, the action works just like natively-supported email and Teams actions. This means that you can get a Power Automate expert in your organization to create the custom action, then roll it out to all Data Activator users, even they don’t have any Power Automate experience. That is all for this month! Please continue sending us your feedback! If you haven’t already head over to the Fabric Community to join in the conversation!Microsoft Fabric September 2023 Update
Welcome to the September 2023 update. We have lots of features this month including updates to the monitoring hub, Fabric Metrics app, VS code integration for Data Engineering, Real-time data sharing and many more. Continue reading for more details on our new features! Contents Core Monitoring hub – column options OneLake OneLake file explorer (v.1.0.10) - Menu Option to View Workspaces and Items Online Power BI General Upgrade Power BI Desktop .NET Framework to 4.7.2 or newer Reporting Mobile layout interactive canvas Smart Narrative improvements Modeling Edit your data model in the Power BI Service - Updates Edit linguistic relationships in Q&A setup DAX function changes to MINX and MAXX Edit relationships in the properties pane (GA) Data Connectivity SAP HANA (Connector Update) Emplifi Metrics (New Connector) Service Row-level security test as role improvements Mobile Supporting AAD Shared device mode (preview) Developers Git integration Visualizations New visuals in AppSource Change Chart Beeswarm Chart Explore time-based data down to each millisecond with Drill Down TimeSeries PRO New Updates for accoPLANNING (Release 57) Zebra BI Charts on-visual settings Word Cloud by Powerviz Innofalls Charts: A Versatile and Interactive Visual Hierarchy Chart by MAQ Software Power BI JSON Report Theme Generator by BIBB Synapse Data Warehouse Data Warehouse Utilization Reporting in Fabric Metrics app Column-Level security is now available on Data Warehouse and SQL Endpoint Data Warehouse Row Level Security SQL Projects support for Fabric Data Warehouse Deployment Pipelines Default Dataset Improvements Data Engineering VS Code Integration Introducing Notebook File System support in Synapse VS Code extension for Data Engineering and Data Science workload in Microsoft Fabric Import Notebook in Workspace view Mssparkutils new API for fast data copy Support diff versions when saving notebook Notebook Sharing with Execute-only mode Notebook resources .whl file support Real-time Analytics OneLake shortcut to delta tables from KQL DB Real-Time Data Sharing: Introducing database shortcuts in Real-Time Analytics Sample Gallery – Explore what you can achieve in Real-Time analytics Model and Query data as graphs using Kusto Query Language (KQL) Easily connect to KQL Database from Power BI desktop Eventstream now supports AMQP format connection string for data ingestion Eventstream supports data ingestion from Azure IoT Hub Improved Eventstream Creation Speed Data Factory Data Pipeline Dataflow Gen2 New Service Principal authentication kind Bug fixes and reliability improvements Community New learning path: Implement a Lakehouse with Microsoft Fabric Core Monitoring hub – column options We have released a new feature inside the monitoring hub to better users customize their experience. Column options gives users more room to operate. Users can select and reorder the columns that meet the scene according to their customized needs. Regardless of whether the user switches to any other tool or scene in Fabric, these columns options are persisted along with the filter until the user returns again. You can now easily reorder the columns with drag and drop inside the option menu. You do not need to worry about the ones at the bottom. All the selected column options will be bubbled up to the top of the menu when you open it next time. Try it out yourself! OneLake OneLake file explorer (v.1.0.10) - Menu Option to View Workspaces and Items Online We are excited to announce the new release of OneLake file explorer for Windows! Now you can seamlessly transition between using the OneLake file explorer app and the Fabric web portal. When browsing data in OneLake file explorer, right click on a workspace and select “OneLake->View Workspace Online.” Selecting this will open the workspace browser on the Fabric web portal. Similarly for items within your workspace. In addition, you can now easily find your client-side logs, which you may need to troubleshoot issues. Right-click on the OneLake icon in the Windows notification area, select Diagnostic Operations, then Open logs directory. Get started by downloading the latest OneLake file explorer. Power BI General Upgrade Power BI Desktop .NET Framework to 4.7.2 or newer Power BI Desktop with .NET Framework 4.5 will no longer be supported after 9/30/2023. Please upgrade to .NET Framework 4.7.2 or newer to avoid any support issues. Reporting Mobile layout interactive canvas With this latest update we are happy to announce that we have made the mobile canvas interactive. This new capability provides the ability to test how buttons, slicers, and visuals will behavior on the app before publishing the report. But that's not all. With this canvas interactivity, users can now interact with visuals directly and adjust Table and Matrix column headers to align perfectly with mobile screens. Smart Narrative improvements The Smart Narrative visual can now display either text or numeric values when showing data for a column. Previously this was limited to only numeric values. Learn more about the Smart Narrative’s summarization capabilities at Create smart narrative summaries - Power BI | Microsoft Learn. Modeling Edit your data model in the Power BI Service - Updates The new data model editing in the Service feature was released to preview in April. We’ve been busy reacting to your feedback and enhancing the experience. Below are the improvements coming later this month: Improve layouts limitations Changes you make to layouts will now persist between Desktop and the Service, including: Changes made to data model layouts in Desktop will now be incorporated into the Service upon uploading the .PBIX file. Similarly, changes made to layouts in the data model within the Service will now be incorporated in the Desktop upon downloading the .PBIX file. Please continue to submit your feedback directly in the comments of this blog post or in our feedback forum. Edit linguistic relationships in Q&A setup The Q&A visual is an effective way to help users further understand their data by asking questions and receiving answers in the form of visuals. It offers users a way to explore their data in ways not covered by the rest of the report without requiring deeper knowledge of their data model or report authoring. However, while the Q&A engine is good at answering precise questions about data, it may not be able to associate every word or phrase a user inputs with data in the model. For example, answering “what are our best consoles this year?” may require connecting the term consoles to the name products in the model, and understanding that the term best corresponds to the highest sales values. These terms are contextual, however – users could mean something completely different asking for console and best in other industries, organizations, or even datasets. To help authors ensure that the Q&A visual provides consistent and accurate answers based on the unique language their report consumers actually use, we introduced Q&A setup tools with an emphasis on providing Q&A with synonyms for column and table names in the model. This way, authors can explicitly define console as referring to products, and users will always receive the correct answers when they ask similar questions in the future. However, synonyms (nouns) are only half of the picture. The other half of the terms (adjectives, verbs, prepositions, adverbs) can’t be defined with such straightforward mappings because they must be understood as a part of a phrase – they qualify other terms or relate other terms together. Best in the previous example is one; asking “who sold the most books” requires us to know that stores sell books, connecting stores to books. There are many types of these linguistic relationships, so we built a new tab entirely to help you create and manage linguistic relationships for your data. You can get into the Q&A setup menu using the gear icon on the Q&A visual or the Q&A setup option in the Modeling tab of the ribbon, then selecting the new Relationships tab. There, you’ll be able to define a variety of relationships, including verb, adjective, noun, preposition, and more. Choose a type which fits the term you’re trying to define (for example, “best” is an adjective), then follow the prompts to define what it means in the context of your data. Our investment into Q&A does not stop here. Even in a world where natural language capabilities are increasingly driven by large language models, there is value in the precision, consistency, and customizability of our sophisticated Q&A engine. In the other direction, defining synonyms and relationships can be a lengthy process just asking to be streamlined with the power of AI-generated suggestions. Keep an eye out in the future for the ways we’re bridging the two to bring out the best of both worlds! DAX function changes to MINX and MAXX We have added an optional variant parameter to MINX and MAXX DAX functions. These functions ignore text and Boolean values when there are variants or mixed data types, such as text and numeric. Now with the new optional variant parameter set to TRUE, the functions will consider the text values. Here is an example of a variant measure. In the table below the variant measure is ordered in ascending order: MINX and MAXX without the variant parameter set, or set to FALSE, will ignore text and Boolean data types. MINX and MAXX with the variant parameter set to TRUE will now include text values. Boolean values are still ignored. The default of the MINX and MAXX optional variant parameter is FALSE, so all existing DAX expressions using MINX and MAXX will not be impacted by this change. To allow text values to be included, you can update the DAX expression to include TRUE as the third parameter. Read more about these changes at Microsoft Learn: MINX function (DAX) - DAX | Microsoft Learn MAXX function (DAX) - DAX | Microsoft Learn Edit relationships in the properties pane (GA) We are excited to announce the general availability of the edit relationships in the properties pane! The edit relationships in the properties pane has been available for public preview since the October 2022 release. Now you can click on any relationship line in the modeling to edit your relationship in the properties pane. This is available in the Model view of Power BI Desktop and in web modeling of the Power BI service. Edit relationships without running queries to preview the data and only validating the relationship when you click apply changes. A welcome relief for those using DirectQuery storage mode, as these queries can take time and impact your data sources. You can learn more about how to utilize this feature at Create and manage relationships in Power BI Desktop - Power BI | Microsoft Learn. Data Connectivity SAP HANA (Connector Update) The update enhances the SAP HANA connector with the capability to consume HANA Calculation Views deployed in SAP Datasphere by taking into account SAP Datasphere’s additional security concepts. This enables consumption of Calculation Views in Datasphere and allows customers to connect to HANA Cloud views without the need for additional privileges on the _SYS_BI schema. Emplifi Metrics (New Connector) We are happy to announce the release of the new Emplifi Metrics connector. Please find release notes from the Emplifi team below: “Integrating social media insights alongside the rest of your marketing or business intelligence data gives you a holistic understanding of your entire digital strategy, all in one place. With Emplifi Power BI Connector, you’ll be able to include social media data from the Emplifi Platform in your charts and graphs and combine them with other data you own. The Power BI Connector is a layer between Emplifi Public API and Power BI itself. It helps you work with your data intuitively, directly in the Power BI tool. The majority of data and metrics available in the Emplifi Public API are also available in the Connector. Please visit the official documentation for more information about Emplifi Public API and a list of available metrics. You’ll find it here: https://api.emplifi.io/.” Service Row-level security test as role improvements We have listened to your feedback about the row-level security test as role experience in the Service and have made several improvements this month including: We have added a new dropdown, allowing you to choose and test any report connected to the dataset. You can now easily see important permissions details pertaining to a specific individual while role testing for that person. We have improved our warning and error messages. To learn more about validating row-level security roles in the Service read our documentation. Mobile Supporting AAD Shared device mode (preview) We, in Power BI continue our investments in empowering frontline workers with data and insights in their work. And in this month release, the Power BI mobile apps add support in Azure active directory shared device mode. Shared Device Mode enables you to configure devices to be shared by a group of employees. This is very common for frontline workers, who do not use their personal device for work related tasks but are getting a work-related mobile device from a shared pool of devices for these tasks. The Shared Device Mode enables single sign-on (SSO) and device-wide sign out for Microsoft Power BI and all other apps that support Shared Device Mode. Once a user signs in into a supported app, the user will be logged into any application compatible with Shared Device Mode as well. Once a user signs out of a supported app, all other applications integrated with shared device mode sign out, to to prevent unauthorized or unintended access by the next user. Back to the frontline worker scenario: when an employee picks a device from a pool at the start of their shift, they need only a single sign-in to one supported app (for example Power BI), and they’re automatically signed in to all other apps that support shared device mode on the device, so the device and the apps are ready for work with the user’s account. At the end of their shift, when they sign out of one app, they're signed out globally from all other apps that support shared device mode, having the device is ready for the next employee and can be safely handed off. Developers Git integration Paginated reports now supported with git integration Since Git integration has been launched few months ago, we supported only 2 items in Power BI- Power BI reports and Power BI datasets. We are now excited to add a new item- Paginated reports! After connecting your workspace to Azure DevOps, you can commit your paginated reports and have them versioned in your repository. After that, you can open the .rdl files directly from git in Power BI Report Builder, edit and push the changes into git. The workspace will identify the changes and will prompt developers to update the workspace with the changes to the paginated report. With this feature, we are adding paginated reports developers to enjoy the collaboration, versioning and modern developer workflows offered in Power BI and Fabric. Learn more about using git with paginated reports. Checkout and Switch Branches in a Workspace A workspace can connect to a single branch at a time. Once you’ve setup your separate workspace to work with git, there might be scenarios where you want to change just the connection of a branch. We have now added 2 new features to help you achieve this much faster: Checkout branch- checkout allows you to easily create a new branch, based on the current state of the workspace. After choosing to checkout, you can create a new branch that will be connected to the WS, while the uncommitted changes are retained. This is useful in cases of conflicts, that allows you to commit your changes to a backup branch, and then manage the merge conflict in the git repo. Switch branch- Workspace admins can decide to change just the connection of the branch very easily. It’s helpful in cases you want to start working on a new branch and wipe clean your WS content for that, or when you are moving from your regular project to small bug fixes (assuming they are on the same content), that should happen on a different branch. New E2E CI/CD tutorial now available Many users have been asking, since the release of git integration, how to work with deployment pipelines and git as part of a CI/CD process. These two features can help build a compelling e2e flow, when connected properly. Follow this step-by-step guide to learn how to make the most out of both tools together. Visualizations New visuals in AppSource Change Chart Pro Circle Card by Devlup Funnels Apex Milestone Trend Dot Chart Activity Gauge by Powerviz Bridger Visual by BI Samurai Number lines by BI-Champ Funnel Chart by Powerviz Date Picker by Powerviz Bar Chart with Export Data Feature Wordcloud by Powerviz PackedBubbleChart Box Ploty by Devlup Funnels LeapLytics - Markdown Viewer Aimplan Data Input Table Beeswarm Chart Overview of Beeswarm Chart Beeswarm chart shows the distribution of data along one axis or both axes while also showing individual points. It is like a one-dimensional or two-dimensional scatter plot but with closely-packed, non-overlapping points. How to use this Visual The visual is intuitive and easy to follow. Only 1 field is mandatory i.e. Bee Category. Tip: If you don’t have any category, you can simply add a calculated column with any name and use it in the category field Bee Size field is optional and is represented in the visual by bubble area or size. If this field is missing then all bubbles will be of same size Bee Label field is also optional. It is used to add labels to each individual bubbles. This chart can be used in 3 ways Y – Axis only X – Axis only Scatter Plot (Both Y and X Axes) 1) Y-Axis Only As shown in picture below, if you only use a column/field in “Beeswarm Y-Axis” section while keeping “Beeswarm X-Axis” section empty, your data will be plotted on Y-Axis categorized according to the “Bee Category” section 2) X-Axis Only Similarly, if you only use a column/field in “Beeswarm X-Axis” section while keeping “Beeswarm Y-Axis” section empty, your data will be plotted on X-Axis categorized according to the “Bee Category” section 3) Scatter Plot (Both X and Y Axes) If you use columns/fields in both sections i.e “Beeswarm X-Axis” section and “Beeswarm Y-Axis” section, the visual will work like a scatter plot with data points plotted on X-Axis and Y-axis . This is depicted below You can show images inside bubbles as well using the “Images” field. For example, the report below shows top millionaires of the world with their net worth on Y-Axis and their Age on X-Axis. Its available in demo file. Formatting Options These options are available in the format pane of the visual under “Settings”. 1) Bubble size As the name implies, this setting simple increases / decrease the bubble size 2) Lower Upper and Left Space Beeswarm chart produces non colliding bubbles or in other words circles don’t overlap. This can often cause bubbles do go outside the chart area especially when there is a concentration of data around minimum and maximum data values. This is where these settings are very useful to bring data inside the chart area. These settings increase or decrease the Y-Axis and X-Axis range to achieve this objective. Try adjusting these settings on the demo file’s sheet “ScatterPlot Beeswarm” to see how they work. Purchase of Premium Features To use the visual without Watermark in Power BI Online, please purchase the Visual’s license for a 1 year period. The license is for unlimited number of users and viewers. If you have any questions, please send me email at [email protected] Download Download the demo file here. Download the custom visual from APPSOURCE Explore time-based data down to each millisecond with Drill Down TimeSeries PRO Drill Down TimeSeries PRO lets you create timeseries charts that are easy to explore on any device. Using its intuitive interactions, users can drill down to months, days or hours by clicking on the chart. Combine up to 12 series and choose between multiple charts - line, column, area. Learn more about Drill Down TimeSeries PRO! MAIN FEATURES: On-chart interactions – pan, zoom and drill down Full customization – set intervals, min/max values, colors, fonts, stacking and clustering Static and dynamic thresholds – set up to 4 thresholds to demonstrate KPIs Cross-chart filtering – select data points on multiple charts instead of using slicers Touch device friendly – explore your data anywhere POPULAR USE CASES: Finance – stock exchange indices, capital ratios, transaction volumes Sales – web traffic, audience reach, lead volume IT – network traffic, response times, syslog and error trends Logistics – inventory movement and turnover, loading time, fleet maintenance costs Get Drill Down TimeSeries PRO now! This visual comes with 30 days free access to paid features. ZoomCharts Drill Down Visuals are known for interactive drilldowns, smooth animations and rich customization options. They are mobile friendly and support: interactions, selections, custom and native tooltips, filtering, bookmarks, and context menu. New Updates for accoPLANNING (Release 57) accoPLANNING for Power BI empowers business users with writeback capabilities, advanced planning, forecasting, budgeting, project management and analysis solutions. We have some new and exciting features for you in this release: • Custom calculated rows and columns configuration. It allows you to customize your own client-side calculations in your already established accoPLANNING table. • End user Lock cells. This gives the flexibility for the end user to lock cells and make sure these will not be affected by splashing or use as an approval indicator. • Hide rows and columns. This Allows users to hide rows and columns - adding flexibility in making asymmetric column and row selections in the grid for better reporting or just for ad-hoc purpose to get a better overview. The latest update also includes a range of new features and improvements designed to increase your productivity: • Our navigation has undergone a major overhaul, with the most notable change being the revamped toolbar. • Better API error message handling in the grid. • Support for automatically expanding all rows/columns/both, eliminating the need to manually adjust the size of each cell. With the accoPLANNING visual, you combine the planning and reporting process in Power BI. For more information, visit our website. https://www.accotool.com/ https://appsource.microsoft.com/en-us/product/power-bi-visuals/WA200002600?tab=Overview https://youtu.be/kNIVC6rBJWA Zebra BI Charts on-visual settings Zebra BI visuals for Power BI are taking another step further to make the user experience as seamless and interactive as possible. Zebra BI Charts has received some important updates so that you save time while customizing your visuals. Thanks to the on-visual settings, you don’t need to go to the visualizations pane every time you want to make a change. COMMENTS SETTINGS can now be adjusted directly on the visual by simply clicking on the settings icon next to the comment box. You can adjust the title, which variances you display, the icon, gap between the comments, and padding. CATEGORY SETTINGS let you adjust several settings by just clicking on the category area. Customize the axis font (family and color), trim /rotate long labels, adjust the label density, and set the gap between the columns. LEGEND SETTINGS come in handy when you want to rename the entries and don’t want to search for this option in the formatting pane. Adjust margins, use aliases in tooltips and switch comparisons. STACKED CHARTS SETTINGS on the visual let you adjust the Top N feature, set color of the chart, and display labels as %. Try it on your data for free. Word Cloud by Powerviz We are excited to announce the new Word Cloud by Powerviz, it's an advanced visual which empowers you to create some of the most high-quality and creative word art in the Power BI. Key Features: Word Styling: Make your word clouds pop with personalized text styles. It offers font styling, direction & text editing features. Color Options: Choose from 30+ color palettes, including color-blind safe options. Shapes: Want to make a statement? Create captivating word clouds by choosing shapes from icons and images, or upload your own image. Exclude: No more hassles anymore! Easily remove unwanted words, symbols from the text to create a clean and focused word cloud. Ranking: Filter out Top/Bottom N Words. Conditional Formatting: Easily spot words with dynamic rules. Many other features included lasso/reverse lasso, grid view, show condition, and accessibility support. Business Use Cases: Marketing: Sentiment analysis & SEO keywords. Education: Brainstorming, Improve engagement. Market Research: Opinion poll, open-ended survey. Presentation: Capture attention & easy communication. Try Word Cloud Visual for FREE from AppSource Check out the visual features in demo file Step by Step instructions and documentation To learn more, visit Powerviz website. Introducing Word Cloud by Powerviz – A Powerful Power BI Custom Visual on YouTube Innofalls Charts: A Versatile and Interactive Visual Innofalls Charts is a powerful visual that offers a wide range of interactive features. Our bar and waterfalls charts come with a drill-down history, enabling orientation and facilitating comparisons. All charts can be stacked or grouped into columns. Waterfalls: Our special attention to waterfalls includes running totals and delta waterfalls, enabling effective comparisons of measures or members. The waterfalls are expandable, drillable, stackable, and offer intermediate sums. Drill Down History: Keep track of your drill downs with visual feedback. Change your drill downs retroactively and combine multiple drill downs for in-depth exploration. New Comparison Features: Effortlessly make comparisons while viewing your report and incorporate deviation charts with a single click. Interactivity: Enjoy various interactive features such as drill, expand, undo/redo, dynamic comparisons, and zoom for enhanced data exploration. IBCS Formatting: Our IBCS theme provides scenario formatting, equal scaling, highlights, deviations and more for consistent and standardized reports. Animations: Enhance data comprehension and reduce change blindness with animations. Promptly detect resorted or added data elements. Discover the full potential of Innofalls Charts and get started today. Innofalls Charts is free for Power BI Desktop! Get started today [ https://www.innofalls.com/blog-get-now ] Visit our website [ https://www.innofalls.com/ ] Hierarchy Chart by MAQ Software Struggling to visualize complex hierarchical relationships? Improve understanding and decision-making with the Hierarchy Chart by MAQ software. In today's data-driven environment, visualizing hierarchical relationships is key to business insights and decision-making. From sales structures to budget allocations, our Power BI-verified visual offers a customizable way to represent these structures easily, catering to various business needs. Figure 1: Sample visual showing a color-coded organization chart (with a tooltip). Key business uses: Organization/HR: Organize human resources by department and hierarchy. Operations: Illustrate manufacturing processes, breaking down components or ingredients hierarchically. Finance: Show budget allocations by division or project, with color-coded subcategories to highlight differences. Sales: Visualize sales structure, territory responsibilities, and targets with group field indicators showing performance status. IT: Visualize IT assets by category and availability with detail. Key features: Color-code cards using a legend. Adjust card (i.e. box) size, borders, and corners. Customize the appearance of the links connecting each card. Interact with cards to control levels and cross-filter visuals. Zoom and reposition the visual with ease. Try out the features of the Hierarchy Chart by MAQ Software today on the visual’s sample report. Learn more about our Power BI custom visuals on our website. Power BI JSON Report Theme Generator by BIBB https://powerbithemegenerator.bibb.pro/ BIBB recently released its take on the Power BI Theme Generator, aiming to streamline the theming process for users. This new tool offers a user-friendly interface, allowing for an easy selection of colours and generation of JSON themes. Within BIBB's generator, users can choose colours in various ways, from manual selection to trending combinations and even importing from images or external sources like Coolors. Synapse Data Warehouse Data Warehouse Utilization Reporting in Fabric Metrics app We are excited to announce that Compute utilization reporting for Data warehouse and SQL Endpoint are now available in Public Preview for Microsoft Fabric across all regions! In the capacity-based SaaS world of Microsoft Fabric, customers can purchase a Fabric SKU and get entitled to a set of Capacity Units (CU). Various workloads, including data warehouse, consume capacity units based on usage. Fabric metrics app provides visibility into capacity usage for all Fabric workloads, including data warehouse in one place. It is used by Capacity or SQL warehouse admins to identify CU usage trends across warehouse items within a capacity, monitor overload information, and understand the cost of running operations which leads to informed capacity sizing decisions. For more information, see the detailed blog Data Warehouse Utilization Reporting in Fabric Capacity Metrics App Column-Level security is now available on Data Warehouse and SQL Endpoint We are excited to announce that Column-Level security is now available on Data Warehouse and SQL Endpoint. Column-level security simplifies the design and coding of security in your application, allowing you to restrict column access to protect sensitive data. For example, ensuring that specific users can access only certain columns of a table pertinent to their department. The access restriction logic is located in the database tier rather than away from the data in another application tier. The database applies the access restrictions every time data access is attempted from any tier. This restriction makes your security more reliable and robust by reducing the surface area of your overall security system. In addition, column-level security also eliminates the need for introducing views to filter out columns for imposing access restrictions on the users. You can implement column-level security with the GRANT T-SQL statement. Only Azure Active Directory authentication is supported. Data Warehouse Row Level Security We are thrilled to announce that Row-Level security is now available in Fabric Warehouse and SQL Endpoint. Row-Level Security enables you to use group membership or execution context to control access to rows in a database table. Row-Level Security (RLS) simplifies the design and coding of security in your application. RLS helps you implement restrictions on data row access. For example, you can ensure that workers access only those data rows that are pertinent to their department. Another example is to restrict customers' data access to only the data relevant to their company. The access restriction logic is located in the database tier rather than away from the data in another application tier. The database system applies the access restrictions every time that data access is attempted from any tier. This makes your security system more reliable and robust by reducing the surface area of your security system. Implement RLS by using the CREATE SECURITY POLICY Transact-SQL statement, and predicates created as inline table-valued functions. SQL Projects support for Fabric Data Warehouse We are excited to announce the Microsoft Fabric Data Warehouse as a supported target platform in the SQL Database Projects extension available inside of Azure Data Studio! SQL Database Projects for Azure Data Studio provides a way to design, edit, and publish schemas for SQL databases from a source controlled project. For a complete development workflow, build and deploy your database projects in CI/CD pipelines, such as GitHub Actions or Azure DevOps. A SQL project is a local representation of SQL objects that comprise the schema for a single database, such as tables, stored procedures, or functions. Use the SQL Database Projects to extract and publish warehouse schemas directly from and to Fabric Data Warehouse. Other compatible databases include SQL Server, Azure SQL Database, Azure SQL Managed Instance, and Azure Synapse SQL (serverless and dedicated). Get started today with the insiders build of ADS. Deployment Pipelines In today’s world, analytics is a vital part of decision making in almost every organization. Fabric's deployment pipelines tool provides BI creators with a production environment where they can collaborate to manage the lifecycle of organizational content. Deployment pipelines enable creators to develop and test content in the service before it reaches the users. Supported content types include reports, paginated reports, dashboards, datasets, dataflows, and now warehouses! The deployment process lets you clone content from one stage in the deployment pipeline to another, typically from development to test, and from test to production. The connections between the copied items are kept during the copy process as well. In addition, Fabric applies the configured deployment rules to the updated content in the target stage. You can also deploy content programmatically, using the deployment pipelines REST APIs. You can learn more about this process in Automate your deployment pipeline using APIs and DevOps. Default Dataset Improvements We have released some new capabilities to enhance the default dataset experience: Turn off automatically adding new objects to the default dataset – navigate to the Warehouse artifact settings to toggle this on/off Updates to the manage default dataset dialog in the model view Filter by schema Filter by object (table/view) We’ve heard your feedback about the ability to turn off the default dataset. Thank you for the feedback, we’re planning on delivering this feature in the coming months! Data Engineering VS Code Integration Introducing Notebook File System support in Synapse VS Code extension for Data Engineering and Data Science workload in Microsoft Fabric The Synapse VS Code extension empowers users to develop their notebook artifacts directly within the VS Code environment. Once users download the .IPYNB file to their local desktop, they gain the flexibility to edit, debug, and execute their notebook code locally. Furthermore, they have the option to select the kernel provided by the extension to execute and debug their notebook code on the remote Fabric Spark compute. Python developers commonly structure reusable functions as modules in the form of .py files, a well-established software engineering best practice. Now, with the incorporation of the Notebook File System within the VS Code Synapse extension, developers can apply this principle to their Fabric notebook development. In addition to running and debugging notebook code, developers can also execute and debug code from imported modules. This integration enhances code modularity and facilitates efficient development workflows. When a user clicks "Open Notebook Folder," the VS Code Synapse extension not only opens the selected .IPYNB file but also downloads all associated files, including .PY modules, from the notebook file system to the local VS Code environment. This feature ensures that the entire set of necessary files and modules is readily available for seamless development and collaboration within the VS Code interface. In this example, there is a .PY module named US2Europe.py which contains a function to covert the datetime format from US fashion to Europe’s one. To invoke this function from the notebook side and execute it on the worker node, you need to import the module containing the function and apply into the data frame To synchronize changes between the local environment and the remote workspace in the VS Code Synapse extension, users can take advantage of two key actions: Publish Resource Folder: Clicking the "Publish Resource Folder" button enables users to upload their local changes to the remote workspace. This ensures that any modifications made locally are reflected in the remote environment. Update Resource Folder: The "Update Resource Folder" button allows users to pull changes from the remote workspace into their local VS Code environment. This ensures that any updates made by collaborators or on the remote side are mirrored locally. In the event that a file has been deleted in the remote workspace but still exists in the local environment, the Synapse extension will automatically handle this situation as follows: The deleted file will be moved to a backup folder named "__backup" under the root directory of the building folder. This approach preserves a copy of the deleted file for reference, ensuring that no data is lost during the synchronization process. These actions provide users with an efficient and robust means to manage the synchronization of their files and code between local and remote environments while maintaining data integrity. Import Notebook in Workspace view We are excited to announce that we now have the “Import Notebook” entry on the Workspace -> New -> Import item! You can easily import one or more files (support .ipynb, .py, .sql, .scala, .r file format) as Fabric Notebook items in the target workspace. Mssparkutils new API for fast data copy We now support a new method in mssparkutils that can enable large volume of data move/copy much faster, which is: Mssparkutils.fs.fastcp() You can use mssparkutils.fs.help("fastcp") to check the detailed usage. According to our benchmark testing, we achieved ~7x to ~180x speed up (varies as different source type, data size and regions) compares to the traditional file system copy method mssparkutils.fs.cp(). As the below example shows, it takes ~5s to copy a 10G file from ADLS Gen2 account to Fabric Lakehouse. Support diff versions when saving notebook We now support viewing and comparing the differences between 2 versions of the same notebook when there are saving conflicts. This is particularly useful when more than one users are working on a same notebook instance via different clients, like: Publish notebook from Fabric VSCode extension Some users are using Manual save mode while others are collaborating Collaborating on notebook with unstable network With the diff function the users in collaboration session can get notified and compare the changes from other user’s edits, and they can choose a version to keep, or save a copy of any version, to easily handle the save conflicts without losing the code accidently. Notebook Sharing with Execute-only mode Fabric Notebook now supports checking the “Run” operation separately when sharing a notebook, if you just selected the “Run” operation, the recipient would see a “Execution-only” notebook after opening the item. With the “Execution-only” mode you can easily spread your notebook without worrying about someone messing up the contents, while the recipients can still run the cells. Notebook resources .whl file support We now support uploading the .whl files in the Notebook Resources explorer, you can put your own compiled libs here, use Drag & Drop code snippet to install them in the session and import the libraries in code conveniently. Real-time Analytics OneLake shortcut to delta tables from KQL DB Now you can create a shortcut from KQL DB to delta tables in OneLake, allowing in-place data queries. With this enhancement, you can effortlessly query delta tables in your Lakehouse or Warehouse directly from KQL DB. Learn more. Real-Time Data Sharing: Introducing database shortcuts in Real-Time Analytics A database shortcut in Real-Time Analytics is an embedded reference within a KQL database to a source database in Azure Data Explorer allowing in-place data sharing. The behaviour exhibited by the database shortcut is similar to that of a follower database. If you are a data provider hosting data in Azure Data Explorer and want to share this data with consumers in Microsoft Fabric, you can create a database shortcut. Database shortcut enables sharing in real-time, with no need to build or maintain a data pipeline. All database changes, including schema and data, on the provider side are instantly available on the consumer side. You can then consume data in Microsoft Fabric by using a KQL Queryset, PBI, Notebooks etc. easily . While today, you can share data from an Azure Data Explorer database with a KQL Database, soon you will also be able to share data from a KQL Database with other KQL Database(s) in Microsoft Fabric. Check out the announcement blog to learn more: Real-Time Data Sharing in Microsoft Fabric blog Sample Gallery – Explore what you can achieve in Real-Time analytics Real-Time analytics now offers a comprehensive sample gallery with multiple datasets allowing you to explore, learn and get started quickly. Selecting a sample in Real-Time Analytics automatically creates a KQL Database with data and prewritten KQL Queryset from any of the 6 sample datasets. This will let users to get started with KQL Database and KQL Queryset without the hassle of creating database, finding the right real time dataset, ingesting data, and writing queries. Each sample lets you explore different streaming and real time scenarios. The samples include data and queries for Stock analytics, Weather analytics, IOT analytics, Log analytics, Metrics analytics and Automotive operations analytics. Each dataset is accompanied by a KQL Queryset which includes sample queries introducing you to KQL capabilities such as. aggregations, search , transformations using update policies, user defined functions, geospatial and time-series analytics. Model and Query data as graphs using Kusto Query Language (KQL) We are excited to share with you a new public preview feature of Kusto Query Language (KQL) that allows you to model and query data as graphs. Graphs are useful for representing complex and dynamic data that involve many-to-many, hierarchical, or networked relationships, such as social networks, recommendation systems, connected assets, or knowledge graphs. By using KQL with graph semantics, you get the speed and scale of KQL queries with the flexibility and expressiveness of graphs. You can query graphs using KQL graph semantics, which has a simple and intuitive syntax that works well with the existing KQL features. You can also mix graph queries with other KQL features, such as time-based, location-based, and machine-learning queries, to do more advanced and powerful data analysis. For example, you can use time-aware graphs to examine how the graph evolves over time, or use geospatial queries to analyze the spatial distribution or proximity of nodes and edges. To learn more about graph semantics in KQL, check out the following resources: - Introduction to graph semantics in KQL: https://aka.ms/kusto/graph-overview - Graph operators and functions reference: https://aka.ms/kusto/graph-operators We hope you enjoy this new feature and find it useful for your data analysis needs. As always, we welcome your feedback and suggestions on how to improve KQL and its graph semantics extension. Happy querying! Easily connect to KQL Database from Power BI desktop Power BI desktop just released new ways to easily connect to a KQL database. There are two ways to easily locate your KQL Database: Open the Get Data dialog and select the Microsoft Fabric menu item. Select KQL Databases from the list: Open the OneLake data hub dialog from the main ribbon. Select KQL Databases from the list: Eventstream now supports AMQP format connection string for data ingestion AMQP, stands for Advanced Message Queuing Protocol, is a protocol that supports a wide range of messaging patterns. It's a widely used communication protocol that facilitates efficient and reliable communication among IoT devices and systems. Now in Eventstream, you can create a Custom App source or destination and select AMQP format connection string for ingesting data into Fabric or consuming data from Fabric. Eventstream supports data ingestion from Azure IoT Hub Azure IoT Hub is a cloud-hosted solution that provides secure communication channels for sending and receiving data from IoT devices. With Eventstream, you can easily stream your Azure IoT Hub data into Fabric and perform real-time processing before storing it in Kusto Database or Lakehouse. To add an Azure IoT Hub source, follow these three simple steps: Choose "Azure IoT Hub" as your eventstream source. Give this source a name and enter your Azure IoT Hub details. You may need to enter the share access key from your Azure IoT Hub to establish the cloud connection. On the canvas, select the IoT source and click "Data Preview" to get a preview of the incoming data from your Azure IoT Hub. Improved Eventstream Creation Speed In response to user feedback, we've made significant improvements to the Eventstream creation, enabling you to create an Eventstream item in seconds. Setting up your Eventstream is now faster and smoother than ever before. These improvements save you valuable time and allow you to ingest and transform your data streams more efficiently. Data Factory Data Pipeline The Outlook email & Teams activities in Data Factory Pipelines can now be used in collaborative shared pipeline New pipeline activities are now available as well! If you’ve used the Custom activity type in ADF pipelines in the past, you’ll be very familiar with the new Azure Batch activity in Fabric data pipelines. This will allow you to execute custom executables from Azure Batch for powerful pipeline workflows. We’ve also enabled Azure Functions and KQL scripts as activities in Fabric pipelines. We've updated the Pipeline output monitoring view to focus on simplification, easily representing pipeline run status with customizable output layouts Pipeline lineage now includes lineage for Connections Dataflow Gen2 New Service Principal authentication kind Service Principal has been added as an authentication type for a set of data sources that can be used in Dataset, Dataflow, Dataflow Gen2 and Datamart. Azure service principal is a security identity that is application based and can be assigned permissions to access your data sources. Service principals are used to safely connect to data, without a user identity. Supported data sources include: Azure Synapse analytics Azure SQL database Azure data Lake gen 2 Azure data lake Azure blob storage Web Dataverse SharePoint online Note: service principal is not supported on the on-premises data gateway and virtual network data gateway. Support is planned by the end of 2023. Read more about Service Principal support from the official announcement post. Bug fixes and reliability improvements We continue to listen to your feedback and we’re actively working and making strategic fixes and quality improvements to our service. The following is a short list of the most impactful fixes and improvements delivered in the past couple of weeks. Revert changes for the limit of 50 queries per dataflow. In our previous monthly update we implemented a limit to the number of queries that a dataflow could have. We reverted this limit until further notice due to a number of undesired experiences created, but we will reintroduce this limit again in the near future. If you have dataflows with more than 50 queries currently, we encourage you to start splitting them into multiple dataflows with lower number of queries, as this will give you a head-start when the limit is reintroduced in the future. This documentation article describes several best practices when developing complex dataflows that you may find useful: Best practices for designing and developing complex dataflows - Power Query | Microsoft Learn Multiple fixes and improvements to common error messages and scenarios. Thanks to multiple customers who have reached out to us through our support system and our Community Forum, we’ve been able to address several common issues and improve the error messages that were quite frequent. Some of the error that have been fixed and/or modified to provide more descriptive information are: “The current row is too large to write.” “We cannot convert the value null to type Table.” “Dataflow refresh transaction failed with status Failed” Errors with Dataflow connector when creating linked entities. This issue has been fixed. We want to emphasize our commitment to our customers on improving the product and take your feedback into account in our decision making. We encourage you to engage directly with us through our support channel or through the Community forums. Community New learning path: Implement a Lakehouse with Microsoft Fabric To learn more and get started, check out the Implement a Lakehouse with Microsoft Fabric learning path. That is all for this month! Please continue sending us your feedback and as always, keep voting on Ideas to help us determine what to build next. We are looking forward to hearing from you!Microsoft Fabric October 2023 update
Microsoft Fabric October 2023 update We have a lot of features this month including updates to the Item type icons, Keyword-Based Filtering of Tenant Settings, On-object Interaction updates, Eventstream Kafka Endpoints and many more. Continue reading for more details on our new and updated features! Core Item type icons Admin Keyword-Based Filtering of Tenant Settings Power BI Reporting Power BI Desktop OneDrive and SharePoint integration On-object Interaction Updates (preview) New! Date Hierarchy on data flyout New! Placeholder text for direct text editing New! Ribbon and Funnel charts now support on-object formatting. Power BI Home in Desktop Deduplication rules for composite models on Power BI Datasets and Analysis Services Modeling Edit your data model in the Power BI Service – Updates Model explorer public preview with calculation group authoring and creating relationships in the properties pane. Data connectivity Snowflake (Connector Update) Planview OKR (New Connector) BitSight Security Ratings (Connector Update) Starburst Enterprise (Connector Update) Service OneLake data hub – Quick access to your data Explorer pane - Quick access Favorite items Visualizations New visuals in AppSource ValQ Plan Date Picker by Powerviz Drill Down Network PRO: Show Categorical Relationships with Ease TMap 2.1 Inforiver Enterprise brings writeback to Fabric Lakehouse & Warehouse Synapse Data Warehouse V-Order write optimization SKU guardrails for burstable compute Data Science Semantic Link (Public Preview) Real-time Analytics KQL Database Capacity Reporting KQL Database Auto scale algorithm – improvements Filtering and visualizing Kusto data in local time with special PowerBI optimization Eventstream UX Improvement on Event Processor Eventstream Kafka Endpoints and Sample Code Data Factory Dataflow Gen2 Data connectivity SAP HANA (Connector Update) New certified connector: Emplifi Metrics Bug fixes and reliability improvements Pipelines Activities AML activity Deactivate/reactivate activity state (Preview) Productivity Category redesign of activities Copy runtime performance improvement Integer data type available for variables Pipeline name now supported in System variables. Support for Type editing in Copy activity Mappings Data Activator Data Activator is now in public preview Core Item type icons We’ve heard your feedback that the icons across Fabric are difficult to tell apart from large list views and other areas of the UI, and that the overall appearance of them was overly monochromatic and flat. Our design team has completed a rework of the item type icons across the platform to improve visual parsing and enrich the way users may form a habituated understanding of what icons operate within different semantic categories of working with data. To learn more about the thinking behind this update, see the detailed blog Admin Keyword-Based Filtering of Tenant Settings Microsoft Fabric has recently introduced keyword-based filtering for the tenant settings page in the admin portal. This new feature allows admins to locate the necessary tenant settings quickly and easily by filtering them based on keywords. To use keyword-based filtering, tenant admins simply enter the keywords they are looking for into the search bar at the top of the tenant settings page. The portal will then filter and display only those settings with matching keywords in the title or description. For instance, an admin could search for "preview" to access all tenant settings currently in preview, or "B2B" to view settings associated with the B2B data sharing feature. To learn more about the feature, check out this blog post. Power BI Reporting Power BI Desktop OneDrive and SharePoint integration OneDrive and SharePoint integrations are now easier than ever in Power BI Desktop! While the ability to open, save, and share reports in OneDrive and SharePoint was released to preview in May, the capabilities are now improved and on by default. You can: Open reports stored in OneDrive and SharePoint through the file menu. Save files directly to OneDrive and SharePoint. Share reports stored in OneDrive and SharePoint directly from Power BI Desktop. These are important changes because many Power BI authors use OneDrive and SharePoint to collaborate on their reports before publishing through the Power BI service. These new updates will streamline their workflow. Ad-hoc reporting is made easier and simpler, and new users starting out will be comfortable with the familiar Office interface. The new Desktop features are complimented by the ability to view Power BI reports stored in OneDrive and SharePoint directly in your browser. Previously, viewing a report stored in OneDrive or SharePoint required downloading both the Power BI file, and Power BI Desktop. The new capability allows users to interact with their reports in seconds. We understand that every organization has its unique needs and preferences. If you prefer not to have some of these features available in your organization, learn how to turn them off in the Fabric admin portal. Learn more about these features in our public preview announcement blog post. On-object Interaction Updates (preview) New! Date Hierarchy on data flyout When working with dates, you may choose to swap from a hierarchy to the raw date field. This is still available when right clicking directly on the date field. For greater discoverability, we’ve now added the ability to switch between date hierarchy and raw date on the data flyout as well. Option 1: Swap using right click. New! Option 2: Swap using data flyout. New! Placeholder text for direct text editing Previously, when turning on a text element such as a title for Tables, there was no reserved space to begin typing in characters directly on-object. Users had to go to the format pane to add a title. Now, when turning on a text element that does not have an “auto” value, we show a placeholder for users to begin typing directly on the visual. When using direct text edit, you’ll also see placeholders appear if you accidentally delete all the characters, but the text element is still ON. This again reserves the space for you to be able to come back and a text value using on-object. Placeholders only appear when the visual(s) are selected while editing, deselecting the visual will remove all placeholders so authors can preview what will be published. New! Ribbon and Funnel charts now support on-object formatting. Power BI Home in Desktop Deduplication rules for composite models on Power BI Datasets and Analysis Services Tables and measures in your model must have unique names. If you use composite models on Power BI Datasets and Analysis Services, it’s easy to get into a situation where tables and measure names have duplicate names and are not unique. Up to this point, when that happened one of the tables or measures would be renamed for you. For example, if you created a composite model from two sources and both sources defined a table called ‘Customers’”, one of the tables would be renamed ‘Customer 2’. This resulted in confusing situations as it was not clear which source the ‘Customer 2’s table came from. The same applies to measures: if you had two sources that both contained a measure called ‘Total Sales’ one would be renamed ‘Total Sales 2’ in the composite model. This month we are giving you more control! You can now apply a name disambiguation rule to a source in a composite model when you anticipate name conflicts with tables or measures from another source. You can set up a text to be added as a prefix or suffix to table names, measure names or both. Additionally, you can choose to add that text only when a duplication occurs or if you prefer to have it added all the time. Going back to the example above, let’s say that one of the sources you are combining is for marketing and the other is for sales. You can now set up a deduplication rule on the source connection so the ‘Customer’ table from the marketing source is named ‘Customer (marketing)’: You will find these options under Settings in the dialog that shows when you set up the composite model connection to a Power BI dataset or Analysis Services model: After you make the connections and set up the deduplication rule, your field list will show both ‘Customer’ and ‘Customer (marketing)’ according to the deduplication rule you set up: Note that you can: Specify whether you want the text to be added to the table or measure name as a prefix or a suffix. Apply the deduplication rule to tables, measures, or both. Choose to apply the deduplication rule only when a name conflict occurs or apply it all the time. The default is to apply the rule only when duplication occurs. In our example, any table or measure from the marketing source that does not have a duplicate in the sales source will not get a name change. If you do not specify a deduplication rule or the deduplication rules you specified do not resolve the name conflict the standard deduplication actions are still applied. Read more about composite models on Power BI datasets and Analysis Services in our documentation. Modeling Edit your data model in the Power BI Service – Updates The new data model editing in the Service feature was released to preview in April. We’ve been busy reacting to your feedback and enhancing the experience. Below are the improvements coming later this month: Manage relationships dialog. Now you can easily view and edit all the relationships within your data model in the Service! In the Home tab, simply select the ‘Manage relationships’ button. This will open the revamped ‘Manage relationships’ dialog, which provides a comprehensive view of all your relationships, along with their key properties, in one convenient location. From here you can then choose to create new relationships or edit an existing relationship. Additionally, you have the option to filter and focus on specific relationships in your model based on cardinality and cross filter direction. Mark as date table Within the Service, you can now mark a table in your data model as a date table. Marking a date table in your model allows you to use this table for various date-related elements including visuals, tables, quick measures, and more, with full Time Intelligence support. To set a date table in the Service, right-click on the desired table and choose ‘Mark as date table > Mark as date table’ in the menu that appears. Next, specify the date column by selecting it from the dropdown menu within the 'Mark as date table' dialog. Power BI will then perform validations on the selected column and its data to ensure it adheres to the 'date' data type and contains only unique values. Please continue to submit your feedback directly in the comments of this blog post or in our feedback forum. Model explorer public preview with calculation group authoring and creating relationships in the properties pane. We are excited to announce that the model explorer is now available for public preview in the model view. You can see all your dataset semantic modeling objects in one place and easily navigate between them. Finally, full visibility of the semantic model! An additional properties pane now shows for the dataset semantic model. And new icons are also showing on the Data pane! Additional UI changes will continue through December for the Data pane and properties pane. Also note that measure groups will always show at the top, followed by calculation groups, and then finally the other tables in the model. From the model explorer, not only can you now see the calculation groups with their calculation items, but you can also create and edit them in Desktop! Calculation groups are a powerful feature that allows you to apply dynamic calculations to your existing measures. For example, you can create a calculation group that applies time intelligence functions as calculation items, such as year-to-date, quarter-to-date, or month-to-date, to any measure in your model. Learn more at aka.ms/calculationgroups. To author these in Desktop, you go to the model view and click on the new “Model” tab of the Data pane. If you click on the Calculation groups node, you have three options to create a new one. (1) Ribbon button, (2) Context menu, (3) Properties pane. After clicking “New calculation group”, if you do not have the model property discourage implicit measures turned off, you will be told this setting needs to be on to create the calculation group. Once turned on, the calculation group is created, and you are landed directly in the first calculation item to define the DAX expression you want to apply to your existing measures. You can alter this DAX expression in the DAX formula bar. Optionally, you can also add a dynamic format string to the calculation item from the properties pane. New calculation items can be created from the context menu (right-click) of the Calculation items node or the calculation group, and in the properties pane of the Calculation items node. Controlling the order of the calculation items can be done in the Calculation items node properties pane or using the context menu (right-click) of the calculation items themselves! Known issues to be fixed before public release: Renaming calculation items will not reflect in the report view slicer until another calculation item added or deleted. Changing the order of calculation items in properties pane moves away from the properties pane. Changing precedence of the calculation groups in the properties pane moves focus away from the properties pane. Is hidden showing “Mixed values.” Cursor not being in the calculation item on creation. Another option also now available from the model explorer is the ability to create relationships in the properties pane. Just like edit relationships in the properties pane allows you to simply change table and columns without previewing data or validating until you click Apply change, this experience is now available to create relationships too. Simply choose new relationship from the model explorer Relationships node. This will show an empty relationship properties pane to fill out and then apply when ready! Learn more about adding and editing relationships at Create and manage relationships in Power BI Desktop - Power BI | Microsoft Learn. Known issues to be fixed: Setting the relationship as one too many even in a valid way will result in an error. This has been fixed but waiting for the changes to be included in Desktop, which may be after public release. The model explorer does show you other modeling features that do not yet have authoring paths in Desktop. These features are perspectives and cultures. These will still need to be authored outside of Desktop throughout XMLA write external tools or through XMLA directly. Learn more about XMLA write at Dataset connectivity and management with the XMLA endpoint in Power BI - Power BI | Microsoft Learn and external tools at External tools in Power BI Desktop - Power BI | Microsoft Learn. Finally, learn more about perspectives at Perspectives in Analysis Services tabular models | Microsoft Learn which work well with the personalized visuals feature of Power BI reports at Let users personalize visuals in a report - Power BI | Microsoft Learn. And cultures is the translations features of semantic models, learn more at Translations in Analysis Services tabular models | Microsoft Learn. These features are available in the latest version of Power BI Desktop. To use them please turn on the model explorer public preview switch. Go to File > Options and settings > Options > Preview features in the GLOBAL section. Data connectivity Snowflake (Connector Update) The Snowflake connector has been updated to support better implementation of “LIMIT 1” queries, resulting in performance improvements. Planview OKR (New Connector) We are excited to release the new Planview OKR connector. Here are the release notes from the Planview team. Planview Objectives and Key Results (OKRs) are an outcome-driven framework adopted by organizations who want to define key organizational goals and track progress toward achieving them. Defining OKRs creates organizational clarity by enabling organizations to answer the questions “Where do we want to go?” (objectives) and “How will we measure our efforts to get there?” (key results). OKRs can be created at different levels of an organizational structure – such as enterprise, portfolio, program, or team – and are connected using parent/child relationships. Linking organizational and team goals in a hierarchical way aligns work delivery to company strategy and provides a single line of sight into value delivered by the organization. Connect now to your OKR data with the custom connector Planview OKR. BitSight Security Ratings (Connector Update) The BitSight Security Ratings connector has been updated with minor bug fixes. Starburst Enterprise (Connector Update) The Starburst Enterprise connector has been updated. Here are the release notes from the Starburst team: Added optional Connection string (non-credential properties) in the Advanced Section to allow the use of other ODBC connection parameters. Changed data source display from Starburst Enterprise to a value based on the connection details ({“Host”:”sep.example.com”, “Port”:”443”}) to allow distinction between multiple Starburst Galaxy or clusters connected as separate data sources. Fixed an issue with port 8443 for OAuth 2.0 authentication. Fixed query folding with timestamp columns. Service OneLake data hub – Quick access to your data OneLake data hub is the central location for users to find and reuse existing organizational data. It allows users to browse through their data and discover insights that can help them make better decisions. We are happy to introduce some of the new features that we recently added to enhance the OneLake data hub discovery experience. Explorer pane - Quick access The Explorer pane enables users to navigate through the workspaces hierarchy and scope the data items to a specific workspace. With the recent enhancements, we added a Quick access section at the top of the Explorer pane. The Quick access section contains an active workspace, pinned workspaces, and recently used workspaces. Favorite items Another new feature is the favorite items. Users can now filter to view items that were marked previously as favorite in Power BI. Users can also find favorite items from within the data hub experience, and they will also show up across all the experiences including Home and Browse. This feature helps users to keep track of the data items that are most important for them. Visualizations New visuals in AppSource Step Filter ValQ Plan valQ Plan is a newer and re-architected Power BI-certified edition of valQ with numerous transformational updates and a simplified no-code experience. valQ helps users build complex business plans, what-if simulations, create & compare budgets, forecasts and scenarios – all within Microsoft Power BI. It can be used as your standalone and integrated business planning software tool or used in conjunction with your existing planning platforms. YT video URL - https://www.youtube.com/watch?v=w4AFQzr58j4 Feature highlights: Niche Use Cases such as Scenario Planning, Strategic Planning or Value Drive Trees and What-If Simulations. Typical Planning use cases: Financial planning & analysis (FP&A), 3-statement financial modelling, and Supply Chain Demand Planning etc. Excel-like grid input experience for your rolling forecasts with no-code auto forecasting algorithms, and goal seek. Integrated reporting: One-click IBCS templates for variance reporting, presentation mode with live charts and tables, enhanced export to Excel & PDF Intuitive modelling: Significantly enhanced formula editor experience, dynamic templating and more Time intelligence: Supports more than 12 periods allowing you to create 90-day rolling forecasts, 52-week forecasts, and even 10-year strategic plans. ValQ Plan can be purchased directly from Microsoft AppSource. Date Picker by Powerviz The Ultimate Date Slicer for Power BI. The Date Picker visual comes with a modern calendar view along with highly requested features like Presets, Pop-up mode, Default Selection, Themes, and more. This is a must-have date slicer for all the Power BI reports. It has rich formatting options to match your brand style guide and to meet your business needs. KEY FEATURES: Display Mode: Choose between Pop-up and Canvas modes. Presets: Many commonly used presets like Today, Last Week, YTD, MTD, or create your own preset using field. Default Selection: Enforce the selection of the desired period when a user opens the report without any custom DAX. Filter Type: Choose between Range and Start/End types. Multiple Date Ranges: Flexibility to select multiple date ranges. Themes: 15+ pre-built themes with full customization. Holidays and Weekends: Multiple formatting options. Import/Export JSON: Build templates and share your designs. Many more features and customizable options. Check out the video - Introducing Date Picker by Powerviz. Get Powerviz Date Picker for FREE from AppSource. Download the demo file here. Step-by-step guide and detailed documentation of all features. To learn more, visit Powerviz website. Drill Down Network PRO: Show Categorical Relationships with Ease Drill Down Network PRO by ZoomCharts is designed for effortlessly visualizing categorical data and automatically detects relationships based on category structure. Use your existing category-based data with few adjustments, and quickly create an interactive chart that makes relationships between each data category easy to read. Visit our site to learn more! Main features: Cross-chart filtering – eliminate slicers by selecting data points directly on the charts. Category-based customization – apply image to node and choose its type, shape, color, font. Link styling – configure 'from' and 'to' decorations and show link value. Display up to 9 data categories. Touch-input device friendly – explore data on any device. Popular use cases: Accounting & Finance – show cost attribution. Human Resources – analyze salary data by department. Production – map production volumes by product or factory. Sales & Marketing – visualize marketing campaigns. ZoomCharts Drill Down Visuals are known for interactive drilldowns, smooth animations, and rich customization options. They support interactions, selections, custom and native tooltips, filtering, bookmarks, and context menu. Use them to create visually appealing and intuitive reports that business users will love on every device. Try Drill Down Network PRO now from the AppSource! TMap 2.1 Newly released TMap 2.1 has been added new features for drilling down Donut Map, Choropleth Map, Bar Chart Map, Pie Chart Map and Stacked Bar Chart Map by georegion's names. Drilling down maps by georegion's names doesn't need to prepare a hierarchy of pre-built polygon layers and will save data professional's time and cost to extract deeper insights from the data on different levels of geographic regions. Screenshot 1 (Drilldown donut map for company A's sale in Asia by subregions) Realigning georegions will become much easier because you only need to change georegion's name in the lowest level and the polygon layer for upper level will be automatically generated during the drilling down process. Screenshot2 (Drilldown donut map for company A's sale in Asia by realigned subregions) You can go to Microsoft AppSource( https://appsource.microsoft.com/en-us/product/power-bi-visuals/mylocsinc1648311649136.tmap?exp=ubp8) to download and try it. To learn more on how to use it, please read tutorials (https://www.mylocs.ca/tutorials.html#drilldown-donut-map-name). Inforiver Enterprise brings writeback to Fabric Lakehouse & Warehouse Inforiver now supports three new data writeback destinations from Power BI: Microsoft Fabric Lakehouse, Fabric Datawarehouse, and Dataverse, along with other popular DWH, DL, and Databases. Here is our 2-minute demo and overview The latest Inforiver Enterprise release supports writeback to Fabric for these use cases. Create business projections by entering/editing directly in Power BI reports. Create & manage rolling forecasts by blending actual & forecast data series. Create dynamic simulations at the enterprise, BU, or department level and roll up projections (to the top) or distribute projections (to granular dimensions) based on chosen allocation/distribution methods. Create multiple scenarios, each containing its own set of simulations. Write back data, including comments & threaded conversations Create periodic report snapshots. Support multiple data input types and writeback from multiple Power BI users in reading mode. The above, combined with advanced audit, security & governance capabilities, make Inforiver the most advanced data input and write-back solution in the market. Inforiver supports writeback in the following deployment configurations: (a) Azure SAAS service managed by Inforiver or (b) managed by customers in their private tenant. Visit our FAQ page to learn more. For Fabric Writeback Proof of Concept (PoC), contact Inforiver here. Synapse Data Warehouse V-Order write optimization V-Order optimizes parquet files to enable lightning-fast reads under the Microsoft Fabric compute engines such as Power BI, SQL, Spark and others. Warehouse queries in general benefit from faster read times with this optimization, still ensuring the parquet files are 100% compliant to its open-source specification. Starting this month, all data ingested into Fabric Warehouses use V-Order optimization. SKU guardrails for burstable compute Synapse Data Warehouse on Microsoft Fabric has the desired flexibility to allow better performance under peak demand by providing burstable compute. SKU Guardrails ensure that customers are operating within the right boundaries for their capacity, preventing peak workloads from consuming all capacity units in a short duration. To learn more about SKU guardrails for burstable compute, check out the blog Data Warehouse SKU Guardrails for Burstable Capacity Data Science Semantic Link (Public Preview) We are pleased to introduce the Public Preview of Semantic Link, an innovative feature that seamlessly connects Power BI datasets with Synapse Data Science within Microsoft Fabric. As the gold layer in a medallion architecture, Power BI datasets contain the most refined and valuable data in your organization. With Semantic Link, we unlock this data's potential beyond traditional business intelligence by making it accessible to notebooks and Python in Microsoft Fabric. Real-time Analytics KQL Database Capacity Reporting A KQL Database will utilize capacity via Operations that can be monitored using the Microsoft Fabric Capacity Metrics. KQL Database Consumption - This is the number of seconds that your KQL database is active in relation to the number of virtual cores used by your database. For example, if my database uses 4 virtual cores and is active for 10 minutes then you will utilize 2,400 seconds of Capacity Units. An auto-scale mechanism is utilized to determine the size of your KQL database. This ensures the most cost-optimized and best performance based on your usage pattern. Read more in the blog: Understanding Fabric KQL DB Capacity | Microsoft Fabric Blog | Microsoft Fabric KQL Database Auto scale algorithm – improvements Users do not need to worry about how many resources are needed to support their workloads in a KQL database. KQL Database has a sophisticated in-built auto scaling algorithm. The algorithm ensures that the optimal number of resources are allocated to support the workloads, with minimum cost. The auto scaling algorithm is multi-dimensional, based on the following dimensions: Hot cache- how much data to store in hot cache for immediate response. This metric is impacted by the caching policy defined by the customer. Memory – how much memory required for meta data and data for optimal query performance. CPU usage- how much compute resources are needed to process queries, update policies, materialized views etc Ingestion - how much compute resources are needed to for data ingestion based on ingestion rates and load times. Filtering and visualizing Kusto data in local time with special PowerBI optimization Datetime values in Kusto (aka ADX/KQL database in Fabric) are assumed to be in UTC. There are good reasons why you should always keep it this way. On the other hand, in many cases you want to visualize the datetime values in a specific time zone and filter the data using values expressed in local time. This is correct but may lead to severe performance degradations if not done correctly. We recently implemented some optimizations that will make some such scenarios much more efficient. In addition, for the many users who are using PowerBI with Fabric KQL database for time series analysis, here is the optimal recommendation: Create a function that will receive the time range as parameters, shift it to UTC, filter the table and shift the filtered rows from UTC to the same time zone. For more details and examples refer to the following blog: Filtering and visualizing Kusto data in local time Eventstream UX Improvement on Event Processor The Event Processor within Eventstream is a powerful no-code editor, enabling you to process and manage your real-time data streams efficiently. You can easily aggregate and filter data streams using temporal functions before they reach your lakehouse or Kusto database. The recent UX improvements introduce a full-screen mode, providing a more spacious workspace for designing your data processing workflows. The insertion and deletion of data stream operations have been made more intuitive, making it easier to drag and drop and connect your data transformations. Eventstream Kafka Endpoints and Sample Code We've expanded the Custom App feature with a range of new endpoints in sources and destinations. Now, you can seamlessly connect your applications to Fabric Eventstream using protocols like EventHub, AMQP, and Kafka. To simplify your setup process, we've included sample Java code for your convenience. Simply add it to your application, and you'll be all set to stream your real-time event to Eventstream. Data Factory Dataflow Gen2 Data connectivity SAP HANA (Connector Update) The update enhances the SAP HANA connector with the capability to consume HANA Calculation Views deployed in SAP Datasphere by taking into account SAP Datasphere’s additional security concepts. This enables consumption of Calculation Views in Datasphere and allows customers to connect to HANA Cloud views without the need for additional privileges on the _SYS_BI schema. New certified connector: Emplifi Metrics We are happy to announce the release of the new Emplifi Metrics connector. Please find release notes from the Emplifi team below: “Integrating social media insights alongside the rest of your marketing or business intelligence data gives you a holistic understanding of your entire digital strategy, all in one place. With Emplifi Power BI Connector, you’ll be able to include social media data from the Emplifi Platform in your charts and graphs and combine them with other data you own. The Power BI Connector is a layer between Emplifi Public API and Power BI itself. It helps you work with your data intuitively, directly in the Power BI tool. Most data and metrics available in the Emplifi Public API are also available in the Connector. Please visit the official documentation for more information about Emplifi Public API and a list of available metrics. You will find it here: https://api.emplifi.io/.” Bug fixes and reliability improvements We would like to thank our community for reporting issues and providing feedback through the Data Factory Community Forum. We continue to work on improving the overall experience and reliability of Dataflow Gen2 in Microsoft Fabric. With over 400 work items closed in the last month, you should notice a better experience in the overall Dataflow Gen2 authoring and refresh experience. We encourage you to visit our community forum and provide any feedback or inquire about any possible issues that you might have with Dataflow Gen2. Your feedback is helping us make this a better product each day. Pipelines Activities AML activity We’re excited to announce that the Azure Machine Learning activity is now available to use in your Data Factory data pipelines. In your pipeline, you can use the Azure Machine Learning activity to connect to your Machine Learning pipelines or enable batch prediction scenarios such as identifying possible loan defaults, determining sentiment, and analyzing customer behavior patterns. Deactivate/reactivate activity state (Preview) We’re excited to share that you can now deactivate one or more activities from a pipeline, allowing you to skip activities during pipeline validation and during pipeline runs. This will help to improve developer efficiency, allowing you to comment out parts of your pipeline without deleting anything from your pipeline canvas. Deactivated activities can be reactivated at any time. Learn more here. Productivity Category redesign of activities We’ve redesigned the way activities are categorized to make it easier for you to find the activities you’re looking for with new categories like Control flow, Notifications, and more! Copy runtime performance improvement We've made improvements to the Copy runtime performance. According to our tests results, with the improvements users can expect to see the duration of copying from parquet/csv files into Lakehouse table to improve by ~25%-35%. Integer data type available for variables We now support variables as integers! When creating a new variable, you can now choose to set the variable type to Integer, making it easier to use arithmetic functions with your variables. Pipeline name now supported in System variables. We’ve added a new system variable called Pipeline Name so that you can inspect and pass the name of your pipeline inside of the pipeline expression editor, enabling a more powerful workflow in Fabric Data Factory. Support for Type editing in Copy activity Mappings You can now edit column types when you land data into your Lakehouse table(s). This makes it easier to customize the schema of your data in your destination. Simply navigate to the Mapping tab, import your schemas, if you don’t see any mappings, and use the drop-down list to make changes. Data Activator Data Activator is now in public preview Data Activator reached a big milestone this month, with its release to public preview. This means Data Activator is now available to all Fabric users, without the need to sign up to be a preview user. You can use Data Activator to drive alerts and actions from your Fabric data. Want to try it out right now? Open a Power BI visual and select the “trigger action” menu option, to create a Data Activator alert: Data Activator also works with real-time streaming data in EventStreams. To make a Data Activator alert on your EventStreams items, create a “reflex” destination: To learn more about Data Activator, check out the following: Data Activator demo video Data Activator documentation on Microsoft Learn
Microsoft Fabric November 2023 Update
Welcome to the November 2023 update. We have lots of features this month including Narrative visual with Copilot, cross workspace "save as" for Data Factory, the general availability of Semantic model scale-out, and many more. Continue reading for more details on our new features! Contents Core Microsoft Fabric User API Reporting Button slicer Reference Labels Enhance your Q&A visual with suggested synonyms from Copilot On-object Interaction Updates Azure Maps visual now aggregates multiple data points at the same location Narrative visual with Copilot Modeling Datasets renamed to semantic models DAX query view to write and run DAX queries on your model Edit your data model in the Power BI Service – Updates Data Connectivity Azure Resource Graph (New Connector) Profisee (Connector Update) Bloomberg Enterprise Data and Analytics (Connector Update) Dremio (Connector Update) Celonis (Connector Update) Service Explore Public Copilot for Power BI in Microsoft Fabric OneLake integration for Import-mode semantic models RLS/OLS security and stored credentials for Direct Lake semantic models Sharable cloud connections for semantic models Semantic model scale-out Show visuals as tables Visualizations Editor's pick of the quarter New visuals in AppSource Zebra BI Tables 6.6: Introducing text columns Funnel Chart by Powerviz Create interactive timelines with full control Other Enhanced accessibility in paginated reports authored in Report Builder Skill up on Fabric with the Microsoft Learn Cloud Skills Challenge Dynamic dataset binding for paginated reports Synapse Data Warehouse Query Insights Fabric Warehouse publishing Full DML to Delta Lake Logs Automatic Data Compaction for Fabric Warehouse Fabric Warehouse support for sp_rename Improvements to CSV data ingestion Fabric enables you to read multi-TB results from Warehouse Blazing fast compute resource assignment is ON! SSD Metadata Caching Fabric SQL support for TRIM and GENERATE_SERIES Time-Travelling through data: The Magic of Table clones REST API support for Warehouse SQLPackage support for Fabric Warehouse User experience improvements Dynamic data masking for Fabric Warehouse & SQL Analytics Endpoint Data Engineering Accessibility support for Lakehouse Enhanced multitasking experience in Lakehouse Upgraded DataGrid capabilities in Lakehouse SQL re-provisioning support in Lakehouse Runtime 1.2 (Apache Spark 3.4, Java 11, Delta Lake 2.4) Multiple Runtimes Support Delta as the default table format in the new Runtime 1.2 Intelligent Cache Monitoring Hub for Spark Enhancements Monitoring for Lakehouse operations Spark application resource Usage Analysis (preview) Rest API support for Spark Job Definition (preview) Rest API support for Lakehouse artifact, Load to tables and table maintenance Lakehouse support for git integration and deployment pipelines (preview) Embed a Power BI report in Notebook Mssparkutils new API – reference run multiple notebooks in parallel Notebook resources .JAR file support Notebook Git integration (preview) Notebook in Deployment Pipeline (preview) Notebook REST APIs (preview) Environment (preview) Synapse VS Code extension in vscode.dev (preview) Data Science Copilot in notebooks (preview) Custom Python Operations in Data Wrangler Data Wrangler for Spark DataFrames (preview) MLFlow Notebook Widget New Model & Experiment Item Usability Improvements Recent Experiment Runs Models renamed to ML Models Release of SynapseML v1.0 Train Interpretable Explainable Boosting Machines with SynapseML Prebuilt AI models Reusing Existing Spark Session in sparklyr Rest API Support for ML Experiments and ML Models New Data Science Happy Path tutorial Expansion of Data Science Samples New Data Science Forecasting Sample Real-time Analytics Delta Parquet support in KQL DB Open Source Connectors for KQL DB New and Improved Get Data Experience for Real Time Analytics REST API Support for KQL Database Event Streams Data Transformation for KQL DB (Generally Available) Get Data from Eventstream Anywhere in Fabric Create a Cloud Connection within Eventstream Optimize Tables before Ingesting Data to Lakehouse Data Factory Dataflow Gen2 General availability of Fabric connectors Automatic refresh cancellation Error message propagation through gateway Support for column binding for SAP HANA connector Staging artifacts will be hidden. New Power Query Editor style Support for VNET Gateways (preview) Pipelines Cross workspace “Save as” MongoDB and MongoDB Atlas connectors are now available. Microsoft 365 connector now supports ingesting data into Lakehouse (preview) Multi-task support for editing pipelines in the designer from different workspaces String interpolation added to pipeline return value Community and Learning Skill up on Fabric with the Microsoft Learn Cloud Skills Challenge Core Microsoft Fabric User API We're happy to announce the Public Preview of Microsoft Fabric User APIs. User APIs are a major enabler for both enterprises and partners to use Microsoft Fabric as they enable end-to-end fully automated interaction with the service, enable integration of Microsoft Fabric into external web applications, and generally enable customers and partners to scale their solutions more easily. The set of APIs we’re introducing today includes the following, while there are a lot more to come very soon: API Group API Description Fabric Workspace Management APIs to Create/Delete/Get/List/Update Fabric workspaces Fabric Workspace Permissions Management APIs to manage principal permissions for Fabric workspaces – Add/Delete/Get/Update workspace role assignment Fabric Workspace Capacities Management APIs to manage capacities in the context of a Fabric workspace – Assign/Unassign a Fabric workspace to/from a capacity Fabric Item Management APIs to Create/Delete/Get/List/Update Microsoft Fabric Items Fabric Capacities Management APIs to manage Fabric capacities – List/Get capacities Fabric Job Scheduling APIs to Schedule/Run/Get History/Update/Cancel Fabric jobs More details can be found in Using the Microsoft Fabric REST APIs A sample can be found here Power BI Reporting Button slicer In this update, we are beyond excited to finally show you Power BI’s latest innovation - the Button slicer, the ultimate tool for slicing data with ease and style! This is just the first step in a thrilling 5-stage journey that will revolutionize your data exploration experience, replacing the old tile slicer with a sleek, flexible alternative that lets you customize your insights and make data-driven decisions faster than ever! Brace yourself for the November upgrade, as it redefines customization and user-friendliness. With so many upgrades and enhancements in the new Button slicer, we continue to go beyond the improvements you saw in the new card visual, and the following list of features will have you on the edge of your seat, so let’s have a look! Customize the shape and appearance of your design with more control to modify the corner radius. With an adjustable grid layout, you can now divide your design into rows and columns, and even use pixels to customize card spacing. When grid layouts have more buttons than rows and columns, use overflow styles, and choose Pagination or Continuous scrolling with vertical or horizontal direction – a big step forward in design flexibility! Just like the new card visual, the button slicer revolutionizes alignment and format properties, and the Label feature will spotlight crucial information from within your buttons. Image control allows you to infuse buttons with images creating endless possibilities with URL images. Interactive states make your designs more engaging with options including on hover, on press, and selected to engage users with responsive interactivity! Get ready for the new formatting settings that will open a world of versatility and a new era of data visualization! Single select has been updated with a Force Selection toggle, and a new Select All option! The new multi-select feature is a significant step forward in user convenience and efficiency! A new Tooltips experience awaits you as the new button slicer now supports both default and report page tooltips to enrich your data visualization skills. With this preview update, the new Button Slicer feature has been toggled ON by default for your convenience and is found in either the Visual gallery on the ribbon, or by selecting Build a visual after right-clicking on the canvas, and then choosing the new slicer, or lastly, by selecting new slicer from the on-object dialog. You can also toggle this feature preview ON or OFF by looking under Options > Preview features > New button slicer. What’s next? Remember, this was only the first stage on the new slicer roadmap. So, fasten your seatbelt, and get ready for the exciting journey ahead as our next stage unfolds and reveals even more Power BI updates, enhancements, and features! Coming next, the List and dropdown slicer! To learn more about the New button slicer read our blog post here. Reference Labels Just like the Button slicer update mentioned above in this month’s Feature Summary, we’re equally excited to share Power BI’s second exhilarating advancement – Reference labels, a versatile tool for adding custom labels to new cards, providing relevant information, comparisons, key metrics, benchmarks, goals, and more, in an appealing and concise manner! As the second step in our 5-stage new card visual journey, this November upgrade will keep you smiling from ear to ear! With Reference labels offering an abundance of creative possibilities, you and your users will be astounded by the world of wonder they unlock. Here's an overview of the available features! Not only can you Reference labels to your card visual, but you even add multiple data fields to your Reference labels. With three main components, Title, Value, and Detail, you can also choose Custom content using a different data field or measure and apply various styles to both Title and Value. Extra context is available for the Detail component, showing data from a single data field well, but also customizable with styles and colors. With an active Divider, Reference labels have their own area where you can modify the divider line and use color to differentiate callout and reference labels. Reference labels can have a horizontal or vertical layout and can have custom spacing and padding. With this preview update, the new Reference Labels feature has been toggled ON by default for your convenience and is found in either the Visual gallery on the ribbon, or by selecting Build a visual after right-clicking on the canvas, and then choosing the new slicer, or lastly, by selecting new slicer from the on-object dialog. You can also toggle this feature preview ON or OFF by looking under Options > Preview features > Reference labels. What’s next? With the first and second stage of the new Card visual now delivered, you can imagine what exciting features are waiting for you as we continue this journey together. To learn more about the Reference labels read our blog post here. Enhance your Q&A visual with suggested synonyms from Copilot The Q&A visual allows you to ask questions about your data and get answers in the form of visual. It provides any report viewer with an intuitive means to explore their data without requiring deeper knowledge of the model or report authoring. Today, the Q&A visual doesn't rely on generative AI to function. The Q&A engine processes your natural language input all inside Power BI algorithmically using a variety of linguistic principles, associating words, and phrases you use with data in your model. This makes it good at answering precise questions about your data, but it may not be able to associate everything you input with data in the model. To help authors ensure that the Q&A visual provides consistent and accurate answers based on the unique language their report consumers use, we introduced Q&A setup tools with an emphasis on providing Q&A with synonyms for column and table names in the model. This way, authors can explicitly define different ways people might refer to their data, and users will always receive the correct answers when they ask similar questions in the future. Power BI recognizes two types of synonyms: approved synonyms and suggestions. Approved synonyms either come directly from the names of fields themselves or are explicitly added by the author. When you use an approved synonym in your Q&A input, it will be treated just as though you used the name of the field and the association will be presented with high confidence, signified by a solid blue underline. Suggested terms are words Power BI thinks are likely to refer to their corresponding name. They come from a variety of sources – synonyms from the Office thesaurus show up by default, but you can also connect to your organization's collection of approved terms and add those to your suggestions as well. Suggestions will still be used by Q&A, but with lower priority than approved synonyms, and the lower confidence will be signaled in the results with a dotted orange underline. In the Q&A setup menu, suggestions can be added to the approved synonyms list or removed entirely. Managing synonyms is therefore an important part of improving the quality of the Q&A experience. However, coming up with synonyms for every data entity in your model can be mentally laborious and physically time-consuming. Copilot for Power BI streamlines this process by generating some for you! If you have Copilot enabled, there are a few ways for you to get suggestions from Copilot. But first, you'll have to enable the feature in Power BI Desktop in File > Options > Preview features > Improve Q&A with Copilot. Then, you might be prompted to add synonyms with Copilot via a banner that shows up the first time you make a Q&A visual or open the Q&A setup menu: You'll also be able to get Copilot suggested synonyms via the Q&A setup menu. You can turn on Copilot as a source in the suggestion settings menu in the synonyms tab, then hit apply to get synonyms. Or, if Copilot is already enabled as a source, you can click the refresh button next to the suggestion settings dropdown. After you've gotten these suggestions, you might be prompted to review them. You'll find the new synonyms in the suggestion’s column in the synonyms page of the Q&A setup menu: Copilot-suggested synonyms will function just like any other suggested synonyms. This means that they may be used by Q&A as a fallback when trying to determine what data fields a natural language input may refer to. Carefully review them in the suggestion’s column of the Q&A visual, remove the synonyms which are inaccurate, and approve the ones which best fit the data. Keep in mind that as we scale out Copilot, you might run into throttling, which may cause Copilot to return incomplete results if you send too many requests in a short period of time. If that happens, you can wait a bit and try again. Copilot may also not return results for terms for which it cannot generate synonyms, or when its results are deemed inappropriate by our content filter. As we mentioned in our release of linguistic relationships for Q&A, we see our investment in both Copilot and the Q&A visual as mutually beneficial. There will be more features coming soon, so keep an eye out on the new ways in which we're bringing the two together! On-object Interaction Updates “Always open in new pane” setting: Most requested, this month we are bringing you the ability to configure your pane switcher to stack panes instead of swap. If you preferred the behavior from before where panes opened side-by-side by default, you can now configure this setting by checking the new option for “always open in new pane” from either the Options menu or the View ribbon. To achieve the stacked behavior of panes as before: Turn on the new option within the Options menu: Or select the new option from the View ribbon: Resizing the data flyout: Also highly requested, this month we’ve also added the ability to resize the data flyout (second flyout) from the build button when working with long field names. Note we have a known bug: there may be cases where the resize handles appear to the left of the data flyout if there is not enough space to expand on the right side. We're working on a fix! As a workaround in these cases, you can move the visual to left temporarily on the canvas to resize the data flyout. The table’s add button is back! The table has the add button again! Originally, we had to remove the add button from the Table visual type as currently the only chart element to add from this menu was the title that does not have a default value. This added confusion to the experience because simply turning on the title did not appear to have changed anything in the visual, and users had to go to the format pane to type in their visual title. Last month we shipped placeholder text which allowed us to bring back the add button for Tables. Now, when turning on title, a placeholder will appear to type directly onto the visual. Azure Maps visual now aggregates multiple data points at the same location Previously, when you had multiple data points with the same latitude and longitude field values, those points would be plotted separately, leaving them drawn one over the other at the same location. This could lead to some unclear data visualizations. For example, grouping the points by category using a legend field might leave just one category visible per location due to the overlap. This behavior was also different from visualizing those points using location names, which would aggregate the points together. With this release, Azure Maps now aggregates points with the same latitude and longitude values in the same way that it does with location names, allowing you to see them as one bubble. These aggregated points can then be filtered or grouped as you would normally. Narrative visual with Copilot We’re excited to bring Copilot’s unique ability to summarize data to the rebranded Narrative with Copilot visual - formerly Smart Narratives. This visual allows you to use Copilot to summarize data across your report, or even specific pages or visuals you choose. We offer suggested prompts to get authors started, such as “Give me an executive summary,” “Answer likely questions from leadership,” and “created a bulleted list of insights.” Users can also type in their own custom prompts and questions that return summaries about their data. Users have the ability to choose whether they want to summarize the entire report, select pages, or even specific visuals across their report, giving them flexibility in what their summary looks like. Users will also see references for each portion of the summary that align to visuals on the various pages on the report, from which the summary lines were generated, making it easy to validate the summary’s accuracy and tie it back to the data. The summary can be updated as the data is sliced and diced, so end users can interact with it, without editing the prompts themselves. The narrative visual with Copilot makes it faster to communicate insights about the data that matters to you. The visual is available in the service and in Power BI Desktop. Modeling Datasets renamed to semantic models We’re excited to share that datasets are being renamed to semantic models in Power BI and Fabric. This is necessary for disambiguation from other Fabric items and will make the product clearer and more usable. You will see this change in the most prominent UI elements in the product, and the documentation is being updated. APIs are not currently affected, and we will progressively roll out further changes. The timing of this change is driven by the general availability of Fabric and aligns with the rename for ML models. It reflects the immense progress that Power BI datasets have made in becoming an enterprise-grade semantic modeling technology. The semantic model’s name will help drive awareness of the unparalleled capabilities provided. Power BI semantic models support for Direct Lake on Synapse Data Warehouse We are delighted to announce that semantic models can now leverage Direct Lake mode in conjunction with Synapse Data Warehouses in Microsoft Fabric. Synapse Data Warehouse is the first transactional data warehouse to natively support the open data standard of Delta-Parquet in OneLake for seamless interoperability across all Fabric and Power BI workloads and the Spark ecosystem. By default, all tables and views in the Warehouse are automatically added to a default Power BI semantic model so that you can readily query the data with DAX and MDX in addition to T-SQL and Spark. For more details, see Default Power BI datasets in Microsoft Fabric in the product documentation. Of course, you can also build custom semantic models on top of a Synapse Data Warehouse. Up until now, semantic models were only able to query Synapse Data Warehouses in DirectQuery mode. This has now changed! Default and custom semantic models can now operate in Direct Lake mode, as the below screenshot highlights. Direct Lake mode is a groundbreaking new data access technology for semantic models based on loading Delta-Parquet files directly from OneLake without having to import or duplicate the data. Especially when dealing with large data volumes, Direct Lake combines the advantages of DirectQuery and Import modes to deliver blazing-fast query performance without any data movement. That's why we are so excited to introduce support for Direct Lake semantic models on top of Synapse Data Warehouses. For more information, see Learn about Direct Lake in Power BI and Microsoft Fabric in the product documentation. With Direct Lake mode support, default semantic models and new custom semantic models on top of Synapse Data Warehouses will operate in Direct Lake mode out of the box. You don’t need to take any action. Existing custom semantic models, however, might have to be recreated or manually converted to Direct Lake mode using XMLA-based editing tools or SQL Server Management Studio as in the screenshot above. In order to verify that a semantic model is indeed leveraging Direct Lake mode for query processing, refer to the article Analyze query processing for Direct Lake datasets in the product documentation. DAX query view to write and run DAX queries on your model Quickly explore and analyze your semantic model with DAX queries. The DAX query view is a fourth view in Power BI Desktop which allows you utilize the powerful DAX query language using EVALUATE to discover, analyze, and see the data in your semantic model. Similar to the Explore feature above for the Power BI service, model authors can quickly validate data and measures in their semantic model without having to build a visual, publishing, or using an additional tool. Changes made to measures can be updated back to the semantic model. DAX queries are different from DAX expressions used to create model items such as measures, calculated columns, and calculated tables, and are more like SQL queries returning data in a table. This powerful way to interact with your data model is now available in the new DAX query view. We give you several ways to be as productive as possible. Quick queries can generate DAX queries for you in the Data pane context menu of tables, columns, or measures gives you a head start by generating a DAX query to preview data or show summary statistics. Use DAX queries to help you understand the data without creating visuals and for DirectQuery you no longer must go back to Power Query to see some sample data. Quick queries to get the DAX expression of all, some, or a specific measure in a generated DAX query. It provides the DEFINE block with the measure’s DAX expression and an EVALUATE block to see the measure’s output. You can then add to the Dax query with any additional group by columns. Update the model options will be available to you with any DAX query updates to measures in DAX query’s DEFINE block. Measures that reference other measures can now be seen on the same screen and updated together. Preview the output of your changes then also update the model when ready. Report view’s Performance Analyzer already lets you copy the visual DAX query. Now you no longer need to take that DAX query and use another tool to view and run it – simply run it in DAX query view. All these features in a bigger DAX query editor similar to VS Code, including more keyboard shortcuts and the ability to improve readability by formatting any DAX query. We plan to continue to add functionality to the DAX query view, so your feedback will be critical. We have previously shown a vision demo with a DAX query view copilot, and saw the excitement for this copilot, and it will be coming soon. We also have planned investments in bringing DAX query view to live connect reports and to the Power BI service, as well as building on it to give more insight into debugging and performance of DAX. Learn more about DAX queries at aka.ms/dax-queries and get started today by turning on this public preview feature in Options > Preview features. Edit your data model in the Power BI Service – Updates The new data model editing in the Service feature was released to preview in April. We’ve been busy reacting to your feedback and enhancing the experience. Below are the improvements coming this month: Mark as date table Within the Service, you can now mark a table in your data model as a date table. Marking a date table in your model allows you to use this table for various date-related elements including visuals, tables, quick measures, and more, with full Time Intelligence support. To set a date table in the Service, right-click on the desired table and choose ‘Mark as date table > Mark as date table’ in the menu that appears. Next, specify the date column by selecting it from the dropdown menu within the 'Mark as date table' dialog. Power BI will then perform validations on the selected column and its data to ensure it adheres to the 'date' data type and contains only unique values. Rename and delete tables and columns Within the Service the following functionality is now supported: Renaming and deleting any table Renaming and deleting any column Please continue to submit your feedback directly in the comments of this blog post or in our feedback forum. Multiple or Empty selections If the user makes multiple selections on the same calculation group, the current behavior is to return the same result as if the user did not make any selections. In this preview, this behavior is going to change and instead we will return no results if you did not specify a multiOrEmptySelectionExpression on the calculation group. If you did, then we evaluate that expression and related dynamic format string and return its result. You can, for example, use this to inform the user about what is being filtered: EVALUATE { CALCULATE ( [MyMeasure], 'MyCalcGroup'[Name] = "item1" || 'MyCalcGroup'[Name] = "item2" ) } -- multipleOrEmptySelectionExpression on MyCalcGroup: IF(ISFILTERED ( 'MyCalcGroup' ),\"Filters: \"& CONCATENATEX (FILTERS ( 'MyCalcGroup'[MyCalcGroup] ),'MyCalcGroup'[MyCalcGroup],\", \")) IF ( ISFILTERED ( 'MyCalcGroup' ), "Filters: " & CONCATENATEX ( FILTERS ( 'MyCalcGroup'[Name] ), 'MyCalcGroup'[Name], ", " ) ) -- Returns “Filters: item1, item2” In case of a conflicting or empty selection on a calculation group you might have seen this error before: With our new behavior this error is a thing of the past and we will evaluate the multipleOrEmptySelectionExpression if present on the calculation group. If that expression is not defined, we will return no results. No selections One of the best showcases for this scenario is automatic currency conversion. Today, if you use calculation groups to do currency conversion, the report author and user must remember to select the right calculation group item for the currency conversion to happen. With this preview, you are now empowered to do automatic currency conversion using a default currency. On top of that, if the user wants to convert to another currency altogether, they can still do that, but even if they deselect all currencies the default currency conversion will still be applied. Current Note how both the currency to convert to as well as the “conversion” calculation group item is selected. New Notice how the user only has to select the currency to convert to. Read more about selection expressions in our calculation groups documentation. The selection expressions for calculation groups are currently in preview. Please let us know what you think! Data Connectivity Azure Resource Graph (New Connector) We are excited to announce the release of the new Azure Resource Graph connector! Please find below release notes from the Azure Resource Graph team. Empower your data insights with our cutting-edge Power BI data connector for Azure Resource Graph! Now, seamlessly transform your Azure Resource Graph queries into stunning visualizations within Power BI. Elevate your analytics game and make data-driven decisions with ease. Unlock the synergy of Azure Resource Graph and Power BI today! Profisee (Connector Update) The Profisee connector has been updated. Below are updated release notes from the Profisee team. Profisee's Power BI Connector Version 3.0 exposes new information around data quality, enhancing analytics on data quality improvements for data managed in Profisee with detailed information regarding validation issues flagged within their data. Additionally, the data types have been refined streamlining the experience for users using the Profisee Connector to load data from Profisee to Microsoft Fabric using Data Factory Gen 2 Data Flows. Bloomberg Enterprise Data and Analytics (Connector Update) The Bloomberg Enterprise Data and Analytics connector has been updated. Below are update release notes from the Bloomberg team. This version of the Bloomberg Data and Analytics connector for Power BI includes changes to support underlying infrastructure updates and back-end performance enhancements. All user-facing features remain unchanged. Dremio (Connector Update) The Dremio connectors have been updated to support additional optional values and parameters. Celonis (Connector Update) The Celonis EMS connector has been updated with minor changes. Service Explore Public Within Power BI, many times users need to perform ad-hoc exploration of their data. This could be an analyst who just got access to a new dataset or data source and wants to spend time learning about the data before building a report off it. Or this could be a business user who needs to answer a specific question using the data to include in a PowerPoint presentation, but the report they’re using doesn’t answer the exact question they have. Creating a new report from scratch in these cases is a large hurdle, just to get a quick answer or screenshot for a deck. Introducing the public preview of the new Explore feature, where users have a lightweight and focused experience to explore their data. Similar to exporting and building a PivotTable in Excel, now, directly within Power BI users can quickly launch Explore to begin creating a matrix/visual pair to get the answers they need without all the distractions and extra complexity of reports. Simply find a dataset or report you’d like to explore: Begin building your matrix/visual pair to get to the answers you need: e And, if you’d like to return to your work save it as an exploration: Find more details in the Introducing Explore (Public Preview) blog post. Copilot for Power BI in Microsoft Fabric We are thrilled to announce the public preview of Copilot in Microsoft Fabric, including the experience for Power BI which helps users quickly get started by helping them create reports in the Power BI web experience. We’ve also added the Copilot’s unique ability to summarize data to the Smart Narrative visual, now rebranded as the Narrative with Copilot visual. The visual is available in the Power BI service and in Power BI Desktop. Lastly in Desktop, we’ve added the ability to generate synonyms synonyms for their fields, measures, and tables using Copilot. To use Copilot you’ll need access to a workspace that has a P1 or higher or a F64 or higher capacity. Head over to our Ignite blog Empower Power BI Users with Microsoft Fabric and Copilot to read all the announcements related to Copilot. We’ll share more details in a dedicated blog next week. Check out the Copilot for Power BI Docs for complete instructions and requirements and don't hesitate to leave a comment in the Fabric Community site if you have any questions. OneLake integration for Import-mode semantic models We are absolutely thrilled to introduce yet another groundbreaking semantic model technology to the world! We are announcing the public preview of Microsoft OneLake integration for import models. With the click of a mouse button, you can enable OneLake integration and automatically write data imported into your semantic models to delta tables in OneLake, as depicted in the following diagram. The data is instantaneously and concurrently accessible through these delta tables. Data scientists, DBAs, app developers, data engineers, citizen developers and any other type of data consumer can now get seamless access to the same data that drives your business intelligence and financial reports. You can include these delta tables in your Lakehouses and Synapse Data Warehouses via shortcuts so that your users can use T-SQL, Python, Scala, PySpark, Spark SQL, R, and no-code/low-code solutions to query the data. OneLake integration can even help you if you don’t plan to query the data. Perhaps you only want to export the data to backup files. Thanks to OneLake integration, this is very straightforward now. Ensure that your import-mode semantic model is hosted in a workspace on a Premium or Fabric capacity and that the large dataset storage format is enabled. Then, enable OneLake integration and perform a manual or scheduled data refresh operation. That’s it! The semantic model writes the imported data to the delta tables as part of the refresh operation. Exporting import-mode tables has never been easier. The delta tables are kept up to date without requiring any ETL pipelines copying data. Of course, you can also export the data programmatically via Tabular Object Model (TOM) and Tabular Model Scripting Language (TMSL) if you can access your semantic model through XMLA in read-write mode. For example, you can open SQL Server Management Studio (SSMS) and run the following TMSL command (see also the screenshot below): { "export": { "layout": "delta", "type": "full", "objects": [ { "database": "<Name of your database>" } ] } } If you have installed the latest version of OneLake File Explorer, you can conveniently verify the success of the export process by using Windows File Explorer. In OneLake File Explorer, right click on the workspace folder and select Sync from OneLake. Then, in the workspace folder, look for a subfolder with a name that matches your semantic model and that ends with .SemanticModel, as in the screenshot above. In this semantic model folder, every import-mode table has a subfolder that contains the delta table’s parquet files and delta log. But you don’t need to know these file system details if you add shortcuts for your semantic model’s delta tables to other workloads in Fabric, such as lakehouses etc. Simply launch the Shortcut Wizard UI, pick Microsoft OneLake, select the semantic model, and then pick the tables you want to include, as in the screenshots below, and that’s it. You are ready to read and query the tables using your favorite data tools and APIs. And there you have it! Now you can use Direct Lake mode to read delta tables directly from OneLake and write delta tables thanks to OneLake integration. Fabric is redefining how customers can build their BI solutions for faster performance at big-data scale while at the same time reducing Total Cost of Ownership (TCO) and infrastructure complexity. For example, you no longer need an entire portfolio of homegrown ETL solutions to get data volumes of any size in and out of semantic models. So, don’t delay and see for yourself how OneLake integration can help you maximize the return of your investments into semantic models by making the data instantaneously and concurrently accessible to data scientists, DBAs, app developers, data engineers, citizen developers and any other type of data consumer you may have in our organizations through delta tables added to your Lakehouses and Synapse Data Warehouses via shortcuts. And as always, provide us with feedback if you want to help deliver additional enhancements. We hope you are as excited about OneLake integration as we are. We think this is a massive innovation and are looking forward to hearing from you! RLS/OLS security and stored credentials for Direct Lake semantic models We are thrilled to announce the public preview of RLS/OLS security and stored credentials for Direct Lake semantic models. RLS/OLS security is a Power BI feature that enables you to define row-level and object-level access rules in a semantic model, so that different users can see different subsets of the data based on their roles and permissions. Stored credentials help reduce configuration complexity and are strongly recommended when using RLS/OLS with Direct Lake semantic models. The following screenshot shows how you can add users to RLS roles in a Direct Lake model by using the Web modeling experience. The web modeling security roles dialog will be fully deployed in the coming days or weeks. For more information about how to set up stored credentials, see the Direct Lake product documentation. For RLS and OLS, see the articles Row-level security (RLS) with Power BI and Object level security (OLS). There is (almost) nothing special for RLS/OLS in Direct Lake models. You can define roles and assign users as for any other semantic model type. But keep in mind that by default Direct Lake models use single sign-on (SSO) authentication to the underlying data source. Using RLS/OLS in conjunction with SSO can be challenging because it involves multiple authorization layers—RLS/OLS in the semantic model and user authorization at the data source. For example, if you wanted to authorize a new user, you would have to add that new user to appropriate RLS roles and ensure that the user has access permissions to the underlying delta tables in the lakehouse or data warehouse. Managing user authorization at multiple layers adds complexity and friction. That's why we are excited to introduce support for stored credentials with Direct Lake semantic models. Your semantic models can now access the delta tables at the source with a single, fixed identity on behalf of the users instead of delegating the actual user identities via SSO. When adding new users to an RLS role, you are effectively authorizing them to use the fixed identity. Because this approach avoids SSO-related complexity and friction, we strongly recommend that you switch to a fixed identity whenever you add RLS/OLS to a Direct Lake model. Switching to a fixed identity is as easy as binding the Direct Lake model to a Shareable Cloud Connection (SCC) that has SSO disabled. For more information, see Connect to cloud data sources in the Power BI service in the product documentation. Here are the steps to configure a Direct Lake model with a fixed identity: Display the settings of the Direct Lake model and expand the Gateway and cloud connections section. Note that your Direct Lake model has a SQL Server data source pointing to a lakehouse or data warehouse in Fabric. Under Maps to, open the listbox, and click on Create a connection. This will pull you to the connections management page with the new connection form opened and prepopulated with the data source information. Select OAuth 2.0 or Service Principal as the authentication method and provide the credentials of the fixed identity you want to use. Make sure you disable to checkbox labeled Use SSO via Azure AD for DirectQuery queries, as in the following screenshot. Configure any other parameters as needed and then click Create. This pulls you back to the Direct Lake model settings page. Verify that the data source is now associated with the non-SSO cloud connection. The ability to set up stored credentials is available today! The RLS editor for Direct Lake datasets in the web modeling experience is being deployed and will be visible in the coming days or weeks. And that’s it for this announcement of RLS/OLS with fixed identities for Direct Lake semantic models. For more information see the articles about Direct Lake semantic models in the product documentation. We hope that these exciting new capabilities enable you to create and migrate even more Power BI semantic models to Direct Lake mode so that you can take full advantage of all the data movement, data science, real-time analytics, and Office integration, and AI, and BI capabilities that Fabric and Power BI have to offer. And please provide us with feedback if you want to help shape the future on world’s best and most successful BI service – Power BI on the unified Fabric platform! We always love to hear from you! Learn about Direct Lake in Power BI and Microsoft Fabric - Power BI | Microsoft Learn Sharable cloud connections for semantic models Along with the general availability (GA) of shareable cloud connections (SCC), we are happy to announce that SCC support for semantic models and paginated reports is GA as well. Now, you can use this modern connection type in conjunction with your production semantic models and paginated reports to access cloud data sources and centralize cloud connection management. In enterprise organizations, centralizing cloud connection management in addition to data gateway management can help to lower the overhead of maintaining data connections and credentials. SCCs let you securely share access to cloud data sources through an access-control list. The credentials are protected and cannot be retrieved from the SCCs, but Power BI users with at least Use permissions can connect their semantic models and paginated reports to the cloud data sources through these SCCs. You can also create multiple connections to the same data source, which is particularly useful if you want to use different connection settings, such as different credentials, privacy settings, or single-sign-on settings, for different semantic models, paginated reports, and other artifacts. Semantic model scale-out We are thrilled to announce semantic model scale-out is now generally available (GA). Large-scale production solutions will benefit from high user concurrency, as Power BI automatically scales out read-only replicas to ensure performance doesn’t slow down when lots of users are using the system at the same time. And of course, automatic scale out works for Direct Lake semantic models! Additionally, Import-mode semantic models will benefit from refresh isolation, ensuring business users are unaffected by resource-intensive refresh operations, and continue to enjoy enable blazing-fast queries for interactive analysis. Here's a quick summary of the benefits semantic model scale-out can provide to your reports, dashboards, and other BI solutions: Increased query throughput Power BI can automatically scale read-only replicas when query volume increases and fluctuates. Refresh isolation Refresh and write operations on the read-write replica do not impact the query performance on read-only replicas. More flexibility for advanced data refresh scenarios As a side-benefit of refresh isolation, you can now perform advanced refresh operations on the read-write replica without impacting the read-only replicas. Simply disable automatic replica synchronization, then refresh, refresh, refresh until the read-write replica is fully updated, and then synchronize the read-replicas manually. Semantic-model scale-out is the last of the key features to make Microsoft Fabric and Power BI a superset of Azure Analysis Services (AAS) and is superior in Fabric compared to its equivalent in AAS. Unlike AAS, scale out takes place based on live user demand, and adjusts automatically to changes in usage patterns. AAS, on the other hand, requires detailed analysis to determine peak usage times, creation of automation scripts, and ongoing monitoring to ensure optimum set up. Additionally, cost in AAS increases linearly per replica, unlike Fabric that is usage based. Please refer to the Configure dataset scale-out article in the product documentation for details on how to enable semantic model scale-out.Along with the general availability (GA) of shareable cloud connections (SCC), we are happy to announce that SCC support for semantic models and paginated reports is GA as well. Now, you can use this modern connection type in conjunction with your production semantic models and paginated reports to access cloud data sources and centralize cloud connection management. In enterprise organizations, centralizing cloud connection management in addition to data gateway management can help to lower the overhead of maintaining data connections and credentials. SCCs let you securely share access to cloud data sources through an access-control list. The credentials are protected and cannot be retrieved from the SCCs, but Power BI users with at least Use permissions can connect their semantic models and paginated reports to the cloud data sources through these SCCs. You can also create multiple connections to the same data source, which is particularly useful if you want to use different connection settings, such as different credentials, privacy settings, or single-sign-on settings, for different semantic models, paginated reports, and other artifacts. Show visuals as tables Leveraging our previous accessibility improvements to table and matrix, we are now introducing a new view mode called Show visuals as tables which display report visuals in a tabular format with a single action. Some users may prefer to consume data in a text-based or tabular format depending on their different learning styles and usage of assistive technologies. This provides a supplemental format for visuals that allows users to display the data in the way that best meets their needs. This new view mode is similar to how Show as a table displays underlying data for individual visuals today. Show visuals as tables will display the underlying data for visuals for all pages in the current report, with the added functionality of interaction and cross-filtering capabilities. To activate this view mode, navigate to the view dropdown menu and select Show visuals as tables. To revert, select Show original visuals. Or simply use the keyboard shortcut Control + Shift + F11 to toggle between the two views. Learn more details about this feature, including limitations, in our documentation: Consuming reports in Power BI with accessibility tools Visualizations Editor's pick of the quarter Performance Flow - xViz Galigeo For Power BI Calendar by Datanau Image Pro by CloudScope Sparkline by OKViz New visuals in AppSource Apex Gantt Chart Zebra BI Tables 6.6: Introducing text columns With Zebra BI Tables 6.6, you can add multiple text columns inside the table. Using this feature, you can bridge the gap in data visualization by having relevant information side by side in the world's best table/matrix custom visual for Power BI. Watch the video of the new functionality! With this feature, you can now display multiple text columns in a tabular way, which leads to a better understanding of the data when displaying more attributes of the same Category. Additionally, there is no need to apply any complex DAX functions. Simply add additional text columns into the ‘Values’ placeholder. SOME POPULAR USE CASES: In one-to-one mapping, you usually need to add additional information next to the descriptive column to ensure data accuracy, consistency, and ease of reference. Product ID: Product name Product ID: SKU: Shipping/Order ID Job ID: Job title The new feature also works in cases with one-to-many mapping. For example, Customer: Sales representatives because the same person can represent multiple customers. You can also add multiple text columns when presenting data in a hierarchical table or using cross-tables for quarterly performance. Try it on your data today for free. Funnel Chart by Powerviz The Funnel Chart by Powerviz is a stunning and informative visualization. It has 4 chart types in 1 visual, including a pyramid chart. The Power BI-certified visual is useful for tracking progress through different stages. It can also group data with legends for detailed insights. Key Features: Funnel Settings: Multiple display types are available, including vertical and horizontal orientation. Data Colors: Offers 7 schemes and 30+ color palettes. Labels: Select from multiple display styles with a custom label option included. Conversion Rate: In a single click measure the percentage of top and bottom stages to identify bottlenecks. Fill Patterns: Highlight stages with custom or pre-filled patterns. Conditional Formatting – Create rules based on measure or category rules. Many other features included are ranking, annotation, grid view, show condition, and accessibility support. Business Use Cases: Sales Funnel Analysis: Track sales stages. Marketing Campaigns: Assess lead generation and conversion rates. User Onboarding: Monitor steps to product adoption. Website Traffic: Visualize visitor drop-offs. Try Funnel Chart for FREE from AppSource. Check out the visual features in the demo file. Step by Step instructions and documentation. To learn more, visit the Powerviz website. Funnel Chart by Powerviz Feature Video on YouTube. Create interactive timelines with full control Drill Down Timeline PRO lets you easily create timelines with a date/time hierarchy. Click directly on the chart to drill down to examine specific periods in detail. Combine multiple series and choose between multiple chart types (line, column, area). Learn more. Main features include: On-chart interactions – click on chart to drill down to months, days, hours, or milliseconds Customize up to 25 series DAX measure support Take customization to the next level with conditional formatting Use series defaults and value labels defaults to customize multiple series Static and dynamic thresholds – set up to 4 thresholds to demonstrate targets Full customization – colors, gradients, backgrounds, fonts, and more Touch device friendly – explore data on any device Popular use cases: Banking & Finance – stock exchange indices, capital ratios, transaction volumes Sales & Marketing – web traffic, audience reach, sales revenue Information Technologies – network traffic, response times, syslog and error trends Manufacturing – quality metrics, uptime and downtime, production output and cost Get Drill Down Timeline PRO from AppSource! ZoomCharts Drill Down Visuals are known for their interactive drilldowns, smooth animations, rich customization options and support: interactions, selections, custom and native tooltips, filtering, bookmarks, and context menu. Other Enhanced accessibility in paginated reports authored in Report Builder The StructureTypeOverwrite property has been added to the .rdl model. You can use it to improve accessibility for paginated reports in Microsoft Report Builder and Power BI Report Builder. You can then publish these reports to the PBI service. Read more about improving accessibility of paginated report. Skill up on Fabric with the Microsoft Learn Cloud Skills Challenge We are excited to announce the Microsoft Ignite: Microsoft Fabric Challenge as part of the Microsoft Learn Cloud Skills Challenge. Skill up for in-demand tech scenarios and enter to win a VIP pass to the next Microsoft Ignite. The challenge is on until January 15, 2024. In this challenge, you will learn how to connect to data, ingest it with Data Factory and notebooks, store it in the lakehouse or data warehouse, and create Power BI reports to turn your data into competitive advantage. The challenge will help you prepare for the Microsoft Certified: Fabric Analytics Engineer Associate certification and new Microsoft Applied Skills credentials covering the lakehouse and data warehouse scenarios, which are coming in the next months. Dynamic dataset binding for paginated reports Use dynamic binding to maintain a single template of an RDL report that can be connected to multiple datasets across workspaces, instead of copying and maintaining hundreds of report duplicates. You don't need to create copies of the same report with a different dataset. You can bind datasets dynamically to a paginated report as outlined in the “Bind datasets dynamically to a paginated report” documentation. You can also bind datasets dynamically to a paginated report visual as outlined in the “Bind datasets dynamically to a paginated report visual” documentation. Synapse Data Warehouse Query Insights Query Insights (QI) is a scalable, sustainable, and extendable solution to enhance the SQL analytics experience. With historical query data, aggregated insights, and access to actual query text, you can analyze and tune your SQL queries. Query Insights provides a central location for historic query data and actionable insights for 30 days, helping you to make informed decisions to enhance the performance of your Warehouse or SQL Endpoint. When a SQL query runs in Microsoft Fabric, Query Insights collects and consolidates its execution data asynchronously, providing you with valuable information. Admin, Member, and Contributor roles can access the feature. Query insights provide the following: Historical Query Data: Query Insights stores historical data about query executions, enabling you to track performance changes over time. System queries aren't stored in Query insights. Aggregated Insights: Query Insights aggregates query execution data into insights that are more actionable, such as identifying long-running queries or most frequent queries. There are three system views to provide answers to your key query performance analysis and turning related questions: queryinsights.exec_requests_history (Transact-SQL) Returns information about each completed SQL request/query. queryinsights.long_running_queries (Transact-SQL) Returns the information about queries by query execution time. queryinsights.frequently_run_queries (Transact-SQL) Returns information about frequently run queries. Autogenerated views are available under the queryinsights schema in SQL Endpoint and Warehouse. Read more about Query Insights here: Query Insights - Microsoft Fabric | Microsoft Learn Fabric Warehouse publishing Full DML to Delta Lake Logs We are excited to announce that the Data Warehouse now publishes all Inserts, Updates and Deletes for each table to their Delta Lake Log in OneLake! Our vision is to break down data silos and make it easy to share data from your Data Warehouses with other teams who use different services without having to create copies of your data in different formats. What does this mean? Today, team members have a wide set of skills and varying comfort levels with different tools and query languages such as Python, T-SQL, KQL and DAX. Instead of having to create copies of your data in different formats for each tool and service, Fabric leverages Delta Lake as a common format across all its services. By only having one copy of your data, this makes it more secure, easier to manage, ensures the data is consistent across reports and it makes it faster and easier to share your data. The Data Warehouse supports this by publishing Delta Lake Logs for every table that you create in your Data Warehouses. When you modify data within a Data Warehouse table, those changes will be visible in the Delta Lake Log within 1 minute of the transaction being committed. For example, say you want to use Python to query a Data Warehouse table by using a Notebook in a Lakehouse. All you would need to do is to create a new shortcut in the Lakehouse and point it to the Data Warehouse Table. That table is now directly accessible by your Notebook and no data has been copied or duplicated! Data Scientists and Data Engineers are going to love how easy it is to incorporate your Data Warehouse Tables into their projects like Machine Learning and training AI models. To learn more about how to create shortcuts that point to Data Warehouse Tables, please see this documentation article: Create a OneLake shortcut - Microsoft Fabric | Microsoft Learn. Conclusion You might wonder, how do I enable this? The answer is that you do not have to do anything! This all happens automatically with your Data Warehouses. Note, only tables created going forward will have all DML published. If you have an older table that you wish to be fully published, you will need to use CTAS (Create Table as Select) to create a new copy of the table with all its data or drop the table and reload it. To learn more about how to leverage your Data Warehouse’s data through its published Delta Lake Logs, please see our documentation Delta Lake logs in Warehouse - Microsoft Fabric | Microsoft Learn. Automatic Data Compaction for Fabric Warehouse We are excited to announce automatic data compaction for Data Warehouses! One of our goals with the Data Warehouse is to automate as much as possible to make it easier and cheaper for you to build and use them. This means you will be spending your time on adding and gaining insights from your data instead of spending it on tasks like maintenance. As a user, you should also expect great performance, which is where Data Compaction comes in! Why is Data Compaction important? To understand what Data Compaction is and how it helps, we need to first talk about how Data Warehouse Tables are physically stored in OneLake. When you create a table, it is physically stored as one or more Parquet files. Parquet files are immutable which means that they cannot be changed after they are created. When you perform DML (Data Modification Language), such as Inserts and Updates, each transaction will create new Parquet files. Over time, you could have 1000s of small files. When reading parquet files, it is faster to read a few larger files than it is read many small files. Another reason for Data Compaction is to remove deleted rows from the files. When you delete a row, the row is not actually deleted right away. Instead, we use a Delta Lake feature called Delete Vectors which are read as part of the table and let us know which rows to ignore. Delete Vectors make it faster to perform Deletes and Updates because we do not need to re-write the existing parquet files. However, if we have many deleted rows in a parquet file, then it takes more resources to read that file and know which rows to ignore. How does Data Compaction happen? As you run queries in your Data Warehouse, the engine will generate system tasks to review tables that potentially could benefit from data compaction. Behind the scenes, we then evaluate those tables to see if they would indeed benefit from being compacted. The compaction itself is actually very simple! It is basically just re-writing either the whole table or portions of the table to create new parquet files or files that do not have any deleted rows and/or have more rows per file. Conclusion Data Compaction is one of the ways that we help your Data Warehouse to provide you with great performance and best of all, it involves no additional work from you! This helps give you more time to work on leveraging your Data Warehouse to gain more value and insights! Please look forward to more announcements about more automated performance enhancements! Fabric Warehouse support for sp_rename We are excited to announce that the Data Warehouse now supports sp_rename. With sp_rename, you can rename user objects like Tables, Stored Procedures, Functions etc. Here is an example of how to use sp_rename that fixes a spelling mistake for city: sp_rename [dbo.dimension_ctiy], [dimension_city]; When you use these features, change the schema an object belongs to or drop a table, the changes will be reflected in OneLake within 1 minute. For More information, please see our documentation: sp_rename (Transact-SQL) - SQL Server | Microsoft Learn Improvements to CSV data ingestion We’re excited to announce a new, faster way to ingest data from CSV files into Fabric Warehouse: introducing CSV file parser version 2.0 for COPY INTO. The new CSV file parser builds an innovation from Microsoft Research’s Data Platform and Analytics group to make CSV file ingestion blazing fast on Fabric Warehouse. The performance benefits you will enjoy with the new CSV file parser vary depending on the number of files you have in the source, the size of these files, and the data layout. Our testing revealed an overall improvement of 38% in ingestion times on a diverse set of scenarios, and in some cases, more than 4 times faster when compared to the legacy CSV parser. The new CSV file parser is now available and is the new default file parser for CSV files during ingestion, so you do not need to do anything to enjoy its benefits. For more details, refer to our documentation on https://learn.microsoft.com/sql/t-sql/statements/copy-into-transact-sql?view=fabric&#parser_version---10--20-. Fabric enables you to read multi-TB results from Warehouse We are excited to announce that the Fabric warehouse enables you to execute queries that return the huge result sets. This capability is useful for scenarios where you need to export data into other systems, refresh large caches, or import substantial amounts of data from a Warehouse into Power BI datasets or other systems. You can handle multi-terabyte results per query without impacting your workload. Warehouse leverages One Lake storage to store and buffer large temporary results before delivering them to the client apps. Smaller results are directly returned to the clients, while larger results are temporarily offloaded into lake storage and then streamed to the client. This feature allows you to work with exceptionally large data warehouses without worrying about the effect of large result sets on your queries and potential size limits of your results. Blazing fast compute resource assignment is ON! All query executions in Fabric Warehouse are now powered by the new technology recently deployed as part of the Global Resource Governance component that assigns compute resources in milliseconds! Warehouse workloads are dynamic and can unpredictably change leading to spikes and dips in number of resources needed for optimal execution. To meet this demand in real-time Global Resource Governance keeps track of all the compute resources in the region and keeps them in a ready state. This enables assignment in milliseconds providing seamless scale up experience allowing workloads to burst. This is not all, Global Resource Governance improves reliability, efficiency, and performance and you can read all about it in a separate blog post covering all these benefits. SSD Metadata Caching Previously, Fabric Warehouse utilized in-memory and SSD cache to store frequently accessed data on local disks in a highly optimized format. This significantly reduced IO latency and expedited query processing. As a result of this enhancement, file and rowgroup metadata are now also cached, further improving performance. Fabric SQL support for TRIM and GENERATE_SERIES You can now remove spaces or specific characters from strings by using the keywords LEADING, TRAILING or BOTH in the TRIM SQL command. Syntax: TRIM ([ LEADING | TRAILING | BOTH] [characters FROM] string) Generates a series of numbers within a given interval with the GENERATE_SERIES SQL command. The interval and the step between series values are defined by the user. Syntax: GENERATE_SERIES (start, stop [, step]) Time-Travelling through data: The Magic of Table clones On July 5 th , we announced the ability to clone data warehouse tables within Microsoft Fabric as of current point in time. The ability to clone tables is a powerful technique that not only empowers businesses to streamline reporting and analytics but also helps expedite development and testing processes. While data warehouses constantly evolve, it is often necessary to capture a snapshot of data as it existed at a particular moment in time. We are now excited to introduce the ability to clone tables with time travel, up to a default data history retention period of seven calendar days. Table clones can be created within and across schemas in the data warehouse within Microsoft Fabric. Businesses can now unlock the ability to perform historical trend analysis, enabling them to compare data from various historical points. It empowers them to identify trends and facilitates making well-informed, data-driven decisions. Cloning a table at previous time points offers the advantage of preserving historical data records, serving a valuable role in meeting various audit and compliance requirements. When data discrepancies occur, these clones not only assist in generating older table versions for root cause analysis but also help create older versions of the table for seamless business continuity. Get started with table clone by creating your first clone either through T-SQL or through UX in the Microsoft Fabric portal. REST API support for Warehouse We're excited to announce the launch of RESTful Public APIs for Warehouse! With the warehouse public APIs, SQL developers can now automate their pipelines and establish CI/CD conveniently and efficiently. The warehouse REST Public APIs makes it easy for users to manage and manipulate Fabric Warehouse items. Here is the Warehouse REST API’s that are supported: Create Item Delete Item Get Item Get Item Definition List Item Update Item Update Item Definition You can learn the detailed usage following the Fabric REST APIs public documentation. SQLPackage support for Fabric Warehouse We are excited to announce SQLPackage support for Fabric Warehouses! SqlPackage is a command-line utility that automates the following database development tasks by exposing some of the public Data-Tier Application Framework (DacFx) APIs: Version: Returns the build number of the SqlPackage application. Added in version 18.6. Extract: Creates a data-tier application (. dacpac) file containing the schema or schema and user data from a connected SQL database. Publish: Incrementally updates a database schema to match the schema of a source. dacpac file. If the database does not exist on the server, the publishing operation creates it. Otherwise, an existing database will be updated. DeployReport: Creates an XML report of the changes that would be made by a publish action. DriftReport: Creates an XML report of the changes that have been made to a registered database since it was last registered. Script: Creates a Transact-SQL incremental update script that updates the schema of a target to match the schema of a source. The SqlPackage command line tool allows you to specify these actions along with action-specific parameters and properties. User experience improvements We are excited to announce the following user experience features to increase your productivity and provide seamless experience within the Warehouse and Lakehouse SQL analytics endpoint: Clone table user experience – Easily create zero-copy clone of tables by selecting table’s current or past point-in-time state. Save as view/table in visual query editor – No-code experience to save your query as a view or save results into the table via Visual query editor enables you to save your analysis for future use. Endorsement – Promote or certify your Warehouse or SQL analytics endpoint of Lakehouse to make it discoverable within your organization. Sample loading experience – Improved sample loading experience with performance improvements and visibility into steps to load sample data into your Warehouse. Viewers/shared recipients can save queries – We automatically save your queries for all users of the Warehouse, including viewers and shared recipients. For more information on these, check out the blog: Insert blog link Dynamic data masking for Fabric Warehouse & SQL Analytics Endpoint Dynamic Data Masking is a powerful security feature that enables organizations to protect sensitive data while preserving the functionality of their applications. DDM allows you to define masking rules for specific columns in your database, ensuring that sensitive information is never exposed in its raw form to unauthorized users or applications. With DDM, you can maintain data privacy and confidentiality without altering your core data structure or application logic. Data Engineering Accessibility support for Lakehouse We are thrilled to announce a significant step toward enhancing accessibility in our Lakehouse experience to provide a more inclusive and user-friendly interaction. Here are the key initiatives and improvements we have implemented so far to support accessibility: Screen Reader Compatibility: Work seamlessly with popular screen readers, enabling visually impaired users to navigate and interact with our platform effectively. Text Reflow: Responsive design that adapts to different screen sizes and orientations. Text and content reflow dynamically, making it easier for users to view and interact with our application on a variety of devices. Keyboard Navigation: Improved keyboard navigation to allow users to move through Lakehouse without relying on a mouse, enhancing the experience for those with motor disabilities. Alternative Text for Images: All images now include descriptive alt text, making it possible for screen readers to convey meaningful information. Form Fields and Labels: All form fields have associated labels, simplifying data input for everyone, including those using screen readers. We encourage all our users to share their thoughts and suggestions for further improvements. We'll continue to monitor feedback from users and make ongoing improvements to maintain the highest standards of inclusivity. Enhanced multitasking experience in Lakehouse We’ve introduced new capabilities to enhance the multi-tasking experience in Lakehouse. Our goal is to make your data management journey as efficient and user-friendly as possible. This latest enhancement includes the following changes designed to supercharge your productivity and streamline your daily tasks: Preserve Running Operations: Have an upload or data loading operation running in one tab and need to check on another task? No problem. With our enhanced multi-tasking, your running operation will not be canceled when you navigate between tabs. Focus on your work without interruptions. Retain Your Context: Selected objects, data tables, or files remain open and readily available when you switch between tabs. The context of your data Lakehouse is always at your fingertips. Non-Blocking List Reload: We've introduced a non-blocking reload mechanism for your files and tables list. You can keep working while the list refreshes in the background, ensuring that you work with the most up to date data without any interruption. Clearly Defined Notifications: Toast notifications will now specify which Lakehouse they are coming from, making it easier to track changes and updates in your multi-tasking environment. We understand that managing a Lakehouse can involve numerous complex tasks, and this upgrade is designed to help you get more done in less time. Stay productive, stay efficient, and happy multi-tasking! Upgraded DataGrid capabilities in Lakehouse We are excited to introduce an upgraded DataGrid for Lakehouse table preview experience with advanced features designed to make working with your data even more seamless and powerful. Here's what you can look forward to in this upgrade: Sorting Data: Whether you're working with large datasets or need to quickly identify trends, this feature will be a game-changer. Sort columns in ascending or descending order with a simple click, giving you full control over your data's organization. Filtering Data: Finding specific data within large datasets is now a breeze. The DataGrid allows you to filter your data in two convenient ways: Type in a keyword, and the DataGrid will instantly display matching results, helping you narrow down your search. Easily filter data by selecting from a list of available values. It's a fast and efficient way to locate the exact data you're looking for. Resizing Columns: Whether you want to prioritize certain data or view a wide range of fields, the ability to resize columns gives you full flexibility. We are dedicated to providing you with the best tools and features to simplify your data management tasks. Stay tuned for more exciting updates in the future! SQL re-provisioning support in Lakehouse We understand that one of the core features of the Lakehouse you rely on for a successful end-to-end experience is the SQL endpoint, and that it's crucial that it functions seamlessly to support your day-to-day needs. We have been listening to your feedback, and today, we are delighted to introduce a significant improvement that empowers you to self-mitigate issues related to SQL endpoint provisioning. Our goal is to provide you with the tools to address any potential hiccups in a user-friendly manner, reducing any inconvenience you might encounter. We now offer you the ability to retry SQL endpoint provisioning directly within the Lakehouse experience. This means that if your initial provisioning attempt fails, you have the option to try again without the need to create an entirely new Lakehouse. We hope that this new feature will provide you with more seamless and reliable experience for your data management needs. Runtime 1.2 (Apache Spark 3.4, Java 11, Delta Lake 2.4) We are thrilled to introduce Microsoft Fabric Runtime 1.2, representing a significant advancement in our data processing capabilities. Microsoft Fabric Runtime 1.2 includes Apache Spark 3.4.1, Mariner 2.0 as the operating system, Java 11, Scala 2.12.17, Python 3.10, Delta Lake 2.4, and R 4.2.2, ensuring you have the most cutting-edge tools at your disposal. In addition, this release comes bundled with default packages, encompassing a complete Anaconda installation and essential libraries for Java/Scala, Python, and R, simplifying your workflow. Multiple Runtimes Support With the introduction of Runtime 1.2, Fabric is adjusted to support multiple runtimes, offering users the flexibility to seamlessly switch between them, minimizing the risk of incompatibilities or disruptions. To change the runtime version at the workspace level, go to Workspace Settings > Data Engineering/Science > Spark Compute > Workspace Level Default, and select your desired runtime from the available options. Once you make this change, all system-created items within the workspace, including Lakehouses, SJDs, and Notebooks, will operate using the newly selected workspace-level runtime version starting from the next Spark Session. If you are currently using a notebook with an existing session for a job or any lakehouse-related activity, that Spark session will continue as is. However, starting from the next session or job, the selected runtime version will be applied. Delta as the default table format in the new Runtime 1.2 What changed? The default Spark session parameter spark.sql.sources.default is now 'delta’. All tables created using Spark SQL, PySpark, Scala Spark, and Spark R, whenever the table type is omitted, will create the table as Delta by default. If scripts explicitly set the table format, that will be respected. The command ‘USING DELTA’ in Spark creates table commands becomes redundant. Scripts that expect or assume parquet table format should be revised. The following commands are not supported in Delta tables: ANALYZE TABLE $partitionedTableName PARTITION (p1) COMPUTE STATISTICS ALTER TABLE $partitionedTableName ADD PARTITION (p1=3) ALTER TABLE DROP PARTITION ALTER TABLE RECOVER PARTITIONS ALTER TABLE SET SERDEPROPERTIES LOAD DATA INSERT OVERWRITE DIRECTORY SHOW CREATE TABLE CREATE TABLE LIKE Intelligent Cache We are excited to announce that by default we have enabled the newly revamped and optimized Intelligent Cache feature in Fabric Spark. The Intelligent Cache works seamlessly behind the scenes and caches data to help speed up the execution of Spark jobs in Microsoft Fabric as it reads from your OneLake or ADLS Gen2 storage via shortcuts. It also automatically detects changes to the underlying files and will automatically refresh the files in the cache, providing you with the most recent data and when the cache size reaches its limit, the cache will automatically release the least read data to make space for more recent data. This feature lowers the total cost of ownership by improving performance by up to 60% on subsequent reads of the files that are stored in the available cache. You can learn more about this feature here: https://learn.microsoft.com/fabric/data-engineering/intelligent-cache Monitoring Hub for Spark Enhancements We are thrilled to share our latest enhancements in the monitoring hub, designed to provide customers with a comprehensive and detailed view of Spark and Lakehouse activities. Executor allocations: The executors' information has been added as an additional column, responding to one of the top requests from customers. This feature allows users to view Spark executor allocations and utilization for optimization and core usage visibility. Runtime version: To support multiple Spark runtime versions, users can now view the Spark runtime version used for a Spark application in the monitoring hub and on the Spark application L2 page. Related items link: The detail page has been updated to include related item links corresponding to the snapshot or notebook with a refined UI. Columns customization: Users can now customize your preferred Spark columns in the monitoring hub, making it easier to sort, search, and filter. Monitoring for Lakehouse operations To enhance the monitoring hub as a unified platform for viewing all Spark activities, including Lakehouse, users can now view the progress and status of Lakehouse maintenance jobs and table load activities. Users can also drill down to view more details on Lakehouse table-level operations. Spark application resource Usage Analysis (preview) Responding to customers' requests for monitoring Spark resource usage metrics for performance tuning and optimization, we are excited to introduce the Spark resource usage analysis feature, now available in public preview. This newly released feature enables users to monitor allocated executors, running executors, and idle executors, alongside Spark executions. Users can zoom in and out as needed for both running and completed Spark applications. The feature also provides a calculated utilization efficiency, allowing users to assess the health of their resource utilization. Additionally, users can drill down to view the related jobs and tasks of the executors. Rest API support for Spark Job Definition (preview) We're excited to announce the launch of RESTful Public APIs for Spark Job Definition! The SJD REST Public APIs makes it easy for users to manage and manipulate SJD items. Here are the Spark Job Definition REST API’s that are supported: Create SJD with Definition Delete SJD Get SJD Get SJD Definition List SJD Update SJD Definition Run SJD Get SJD run status Rest API support for Lakehouse artifact, Load to tables and table maintenance As a key requirement for workload integration, we are announcing the launch of RESTful Public APIs for Lakehouse! The Lakehouse REST Public APIs makes it easy for users to manage and manipulate Lakehouse artifacts items programmatically. The key capabilities of the Load to tables feature and table maintenance feature are also supported, guaranteeing programmability to data ingestion and that Delta tables are maintained in top performance for Fabric consumption. Create Lakehouse Creates Lakehouse inside a workspace. A SQL Analytics endpoint also gets provisioned along with the lakehouse. Update Lakehouse Updates name of the Lakehouse artifact and of the SQL Analytics endpoint. Delete Lakehouse Delete lakehouse artifact and associated SQL Analytics endpoint. Get Properties Gets the properties of the lakehouse artifact and SQL Analytics endpoint. List tables List tables in the Lakehouse. Table load Creates Delta table from CSV and parquet files and folders. Table maintenance Apply bin-compaction, V-Order, and unreferenced old files cleanup. Lakehouse support for git integration and deployment pipelines (preview) The Lakehouse artifact now integrates with the lifecycle management capabilities in Microsoft Fabric, providing a standardized collaboration between all development team members throughout the product's life. Lifecycle management facilitates an effective product versioning and release process by continuously delivering features and bug fixes into multiple environments. Summary of git integration and deployment pipelines capabilities: Serialization of the Lakehouse object metadata to a git JSON representation. Apply changes directly or use pull requests to control changes to upstream or downstream workspaces and branches. The renaming of Lakehouses is tracked in git. The update of renamed Lakehouse will also rename default semantic data model and SQL Analytics endpoint. Deployment across Dev-Test-Production workspaces. Lakehouse can be removed as a dependent object upon deployment. Mapping different Lakehouses within the deployment pipeline context is also supported. Updates to Lakehouse name can be synchronized across workspaces in a deployment pipeline context. Embed a Power BI report in Notebook We are thrilled to announce that the powerbiclient Python package is now natively supported in Fabric notebooks. This means you can easily embed and interact with Power BI reports in your notebooks with just a few lines of code. You can also create stunning reports based on a Pandas dataframe or a Spark dataframe in the context of a notebook run. Power BI reports in Fabric Notebooks are a good way to tell compelling data stories and share insights with others. To learn more about how to use the powerbiclient package, check out this article how to embed a Power BI component. Remember that you don’t need to do any extra set up on Fabric notebooks using Spark 3.4, just import the package and things will work! Mssparkutils new API – reference run multiple notebooks in parallel We are thrilled to introduce a new API in mssparkutils called mssparkutils.notebook.runMultiple() which allows you to run multiple notebooks in parallel, or with a pre-defined topological structure. With mssparkutils.notebook.runMultiple(), you can: Execute multiple notebooks simultaneously, without waiting for each one to finish. Specify the dependencies and order of execution for your notebooks, using a simple JSON format. Optimize the use of Spark compute resources and reduce the cost of your Fabric projects. View the Snapshots of each notebook run record in the output, debug/monitor your notebook tasks conveniently. To learn more about this new API and how to use it, please refer to the documentation here. You can also try to run the mssparkutils.notebook.help("runMultiple") to find out more. Notebook resources .JAR file support We now support uploading the .jar files in the Notebook Resources explorer. You can put your own compiled libs here, use Drag & Drop to generate a code snippet to install them in the session and load the libraries in code conveniently. Notebook Git integration (preview) Fabric notebooks now offer Git integration for source control using Azure DevOps. It allows users to easily control the notebook code versions and manage the Git branches by leveraging the Fabric Git functions and Azure DevOps. Users can set up a connection to their Repo from the workspace settings following the Fabric Git integration Instructions. Once connected, your notebook items will appear in the “Source control” panel together with other items inside the workspace. After the notebook artifact is committed to the Git repo successfully, from Azure DevOps Repos Files view, you’ll be able to see the artifact folder structure and source code there. Update Additionally, when committing the notebook artifact to the Git repo, the code will be converted to a source code format (e.g., PySpark notebook to a 'notebook-content.py' file) instead of a standard ‘. ipynb’ file. This approach allows for easier code reviews using built-in diff features. In the artifact content source file, the artifact's metadata including lakehouses, markdown cells, and code cells will be preserved and distinguished, allowing for precise recovery when synced back to a Fabric workspace. Notebook in Deployment Pipeline (preview) Now you can also use notebooks to deploy your code across different environments, such as development, test, and production. This enables you to streamline your development process, ensure quality and consistency, and reduce manual errors. You can also use deployment rules to customize the behavior of your notebooks when they are deployed, such as changing the default Lakehouse of a Notebook. You can follow the instructions Get started with deployment pipelines to set up your deployment pipeline, Notebook will show up in the deployment content automatically. You will be able to compare content in different stages, deploy content from one stage to another, monitor the deployment history, and configure deployment rules. Notebook REST APIs (preview) We're excited to announce the launch of RESTful Public API for the Notebook item! With the notebook public APIs, data engineers/data scientists can now automate their pipelines and establish CI/CD conveniently and efficiently. The notebook Restful Public API can make it easy for users to manage and manipulate Fabric notebook items and integrate notebook with other tools and systems. Here is the list of supported Notebook REST APIs: Notebook Management: Create Item/Delete Item/Get Item/Get Item Definition/List Item/Update Item/Update Item Definition Notebook Job Scheduler: Run on Demand Item Job (with parameterization support)/ Cancel Item Job Instance/ Get Item Job Instance You can learn more by checking out the Fabric REST APIs public documentation. Environment (preview) We are thrilled to announce public preview of the environment item in Fabric. The environment is a centralized item that allows you to configure all the required settings for running a Spark job in one place. It provides a unified interface for managing, maintaining, and sharing the configurations. In an environment, you can select different Spark runtime, refine the compute resources, install libraries, and more. Once you have configured your environment, you can attach it to your Notebooks, Spark job definitions or even the workspace, serving as workspace default. The configurations will be effective, and the libraries will be available once the Spark session is started with the environment attached. Synapse VS Code extension in vscode.dev (preview) We are excited to introduce the public preview of the Synapse VS Code extension on vsocde.dev. With support for vscode.dev, a lightweight version of VS Code that operates entirely within your web browser, users can now seamlessly edit and execute Fabric Notebooks without ever leaving their browser window. Additionally, all the native pro-developer features of VS Code are now accessible to end-users in this environment. Any code modifications made within vscode.dev will be instantly applied to the workspace. If users still prefer working within their local development environment, they can opt to download and edit their notebooks using the desktop version of VS Code. Data Science Copilot in notebooks (preview) We’re excited to announce the public preview of Copilot in Fabric Data Science and Data Engineering notebooks. Copilot is designed to accelerate productivity, provide helpful answers and guidance, and generate code for common tasks like data exploration, data preparation and machine learning with. You can interact and engage with the AI from either the chat panel or even from within notebooks cells using magic commands to get insights from data faster. Note that this Copilot public preview does not require any sign-up and will gradually be rolled out to customers during the coming months. Learn more about Copilot in Microsoft Fabric Copilot Chat panel: The chat pane experience makes seeking help and insights into data easy on top of notebooks. Users can use natural language to ask questions about their data, generate code snippets based on prompt inputs, and even ask Copilot to provide helpful next steps in the notebook. The Copilot chat panel shares all background context with the notebook inside of a spark session, so any executed cells or dataframes added to the notebook are available to the chat panel. Chat magics: Chat magics are magic commands, such as %%chat and %%code, which can be used inside of a notebook to make requests to Copilot to provide insightful recommendations. Magics can provide assistance directly from within the notebook cell output and can be persisted as output into the notebook. Custom Python Operations in Data Wrangler Data Wrangler, a notebook-based tool for exploratory data analysis, has always allowed users to browse and apply common data-cleaning operations, generating the corresponding code in real time. Now, in addition to generating code from the UI, users can also write their own code with custom operations. Like every transformation in Data Wrangler, custom operations update the display grid, offering a live preview of each step before it is applied. Getting Started: You can initiate a custom operation either by selecting “Custom operation” from the lefthand Operations panel or by typing directly into the code editor below the display grid. Modifying the DataFrame: Your custom code will be previewed in the display grid in real time. Note that you can modify the DataFrame variable (df’) directly or set it equal to a new value (df = . . .’). Previewing the Transformation: Once you’ve stopped typing, the effect of your custom operation will be previewed in the display grid, as with any other Data Wrangler transformation. You can apply or discard the step or edit the code to modify the transformation. Data Wrangler for Spark DataFrames (preview) We’re pleased to announce that Data Wrangler now supports Spark DataFrames in public preview. Until now, users have been able to explore and transform pandas DataFrames using common operations that can be converted to Python code in real time. The new release allows users to edit Spark DataFrames in addition to pandas DataFrames. In both cases, you can now customize the sample that Data Wrangler displays, specifying the number of rows to show and choosing from among three sampling methods (first records, last records, or a random subset). In order to maintain performance, Spark DataFrames are automatically converted to pandas samples in Data Wrangler’s preview grid, but all the generated code will be converted to PySpark when it is exported back to your notebook. This built-in sampling ensures that Data Wrangler feels consistent no matter what kind of data you’re working with. If you’re already familiar with Data Wrangler for your pandas use cases, you’ll notice only a few changes when you open a Spark DataFrame. Launching Data Wrangler: Beneath the list of pandas variables in the Data Wrangler dropdown, you’ll now see a list of Spark variables and an option to choose a custom sample. Exploring and Transforming Data: Once the tool loads, an informational banner above the preview grid will remind you, if you’re working with Spark data, that all the code generated by Data Wrangler will be converted to PySpark when it’s added back to your notebook. Exporting Code: Browsing operations and applying transformations remain the same, but when you’re ready to export the committed steps as code, Data Wrangler will display a preview of the final PySpark and provide the option to save the interim pandas code for further exploration. MLFlow Notebook Widget We've introduced the MLflow inline authoring widget, enabling users to effortlessly track their experiment runs along with metrics and parameters, all directly from within their notebook. Additionally, users can access a Run Comparison view, where they can visually compare runs using various plot types and customizations. Last, for those seeking even more insights, you can dive into the Experiment item directly from your notebook. New Model & Experiment Item Usability Improvements We have made significant enhancements to our model and experiment tracking features based on valuable user feedback. The new tree-control in the run details view makes tracking easier by showing which run is selected. Furthermore, we've enhanced the comparison feature, allowing you to easily adjust the comparison pane for a more user-friendly experience. Run names are now highlighted across plots, making them easier to spot. Update In addition, transitioning to the Run Details view is now effortless – simply click on the run name. Plus, users can now apply filters to any column, offering increased flexibility and user-friendliness, even beyond their customized columns view. Recent Experiment Runs We've made it simpler for users to check out recent runs for an experiment directly from the workspace list view. This update makes it easier to keep track of recent activity, quickly jump to the related Spark application, and apply filters based on the run status. Models renamed to ML Models We’re excited to share some important updates regarding a terminology shift in Fabric. We're evolving our terminology from "Models" to "ML Models" to ensure clarity and avoid any confusion with other Fabric elements. This change will be visible in key UI elements and updated documentation, though APIs remain unaffected. While we understand that this may introduce some initial adjustments, it's a necessary step towards enhancing the overall usability and comprehensibility of the product. This transition aligns with the upcoming general availability of Fabric and mirrors the rename of Power BI Datasets to Semantic Models, making Fabric even more user-friendly. To learn more, you can visit: Machine learning experiments in Microsoft Fabric. Release of SynapseML v1.0 We are pleased to announce the release of SynapseML v1.0, our open-source library that simplifies the creation of massively scalable machine learning (ML) pipelines. SynapseML makes it easy to build production ready machine learning systems on Fabric and has been in use at Microsoft for over 6 years. This milestone release introduces new integrations with Vector Search engines for efficient management of GPT embeddings, advanced conversational AI and LLM capabilities, Orthogonal Forest DoubleML for robust causal inference, key-free authentication for Azure AI services on Fabric, and much more. This release underscores our commitment to providing powerful open-source tools for your machine learning needs and supporting the SynapseML community. Train Interpretable Explainable Boosting Machines with SynapseML We've introduced a scalable implementation of Explainable Boosting Machines (EBM) powered by Apache Spark in SynapseML. EBMs are a powerful machine learning technique that combines the accuracy of gradient boosting with a strong focus on model interpretability. With this addition, you can now leverage the EBM model in your machine learning endeavors within SynapseML, making it easier to gain insights and transparency in domains where understanding model decisions is crucial, such as healthcare, finance, and regulatory compliance. You can learn how to train EBMs for classification or regression scenarios. Prebuilt AI models We are excited to announce the public preview for prebuilt AI models in Fabric. Fabric seamlessly integrates with Azure AI services, allowing you to enrich your data with prebuilt AI models without any prerequisites. Azure OpenAI Service, Text Analytics, and Azure AI Translator are available out of the box in Fabric, with support for both RESTful API and SynapseML. You can also use the OpenAI Python Library to access Azure OpenAI service in Fabric. For more information on available models, see prebuilt AI models in Fabric. Reusing Existing Spark Session in sparklyr We have added support for a new connection method called "synapse" in sparklyr, which enables users to connect to an existing Spark session. Additionally, we have contributed this connection method to the OSS sparklyr project. Users can now use both sparklyr and SparkR in the same session and easily share data between them. Rest API Support for ML Experiments and ML Models We’re pleased to begin rolling out a set of public-facing REST APIs for the main data science artifacts: the ML Experiment and the ML Model. These APIs begin to empower users to create and manage machine-learning artifacts programmatically, a key requirement for pipeline automation and workload integration. Stay tuned in the coming months for updates about more robust API support. For now, the following APIs are supported: List ML Experiments in a Workspace List ML Models in a Workspace Create ML Experiment Create ML Model Get ML Experiment Get ML Model Update ML Experiment Delete ML Experiment Delete ML Model New Data Science Happy Path tutorial We're thrilled to announce an update to the Data Science Happy Path tutorial for Microsoft Fabric. This new comprehensive tutorial demonstrates the entire data science workflow, using a bank customer churn problem as the context. It's the perfect resource to kickstart your data science journey on Fabric. The updated tutorial includes the following key steps: Ingest data into a Fabric lakehouse using Apache Spark. Load existing data from delta tables in the lakehouse. Clean and transform data with tools like Apache Spark and Fabric Data Wrangler. Create experiments and runs to train various machine learning models. Register and track trained models using MLflow and the Fabric UI. Perform scalable scoring with the PREDICT function and save results to the lakehouse. Visualize predictions and business insights in Power BI using Direct Lake. Expansion of Data Science Samples We are excited to announce that we have expanded our collection of data science samples by introducing new categories of samples that include end-to-end workflow samples in R and Quick tutorials samples. The two new end-to-end R samples demonstrate the entire data science workflow for the bank customer churn problem and the credit card fraud detection problem using the R language. The two new "Quick tutorial" samples also include "Explaining Model Outputs" and "Visualizing Model Behavior." The first tutorial will walk you through the process of using SHapley Additive exPlanations to gain deeper insights into model outputs. The second tutorial shows how to leverage Partial Dependence and Individual Conditional Expectation plots to interpret the impact of various features on the model outputs. Please check out these new samples and let us know your thoughts, as we are committed to continually improving your data science experience on Microsoft Fabric. New Data Science Forecasting Sample We are happy to introduce our newest Data Science sample – Sales Forecasting - developed in collaboration with Sonata Software. This new sample encompasses the entire data science workflow, spanning from data cleaning to Power BI visualization. It is designed to forecast product sales in a superstore, harnessing the power of the SARIMAX algorithm. Within this sample, you will discover comprehensive guidance on model training and parameter optimization, helping you gain a deep understanding of how to make accurate forecast predictions. The sample also includes a Power BI report that enables you to transform your data-driven forecasts into compelling visualizations that are both informative and visually appealing. Real-time Analytics Delta Parquet support in KQL DB To achieve seamless data access across all compute engines in Microsoft Fabric, Delta Lake is chosen as the unified data lake table format. As part of the one logical copy promise, we are excited to announce that data in KQL Database can now be made available in OneLake in delta parquet format. The data streamed to KQL Database is stored in an optimized columnar storage format with full text indexing and supports complex analytical queries at low latency on structured, semi-structured, and free text data. Enabling KQL DB data availability in OneLake means that customers enjoy best of both worlds: they can query the data with high performance and low latency in KQL DB and query the same data in delta parquet available in OneLake via any other Fabric engines (Direct Lake mode in Power BI, Data warehouse, Lakehouse, Notebooks etc.). KQL DB provides a robust mechanism to batch the incoming streams of trickling data into Delta parquet files suitable for analysis. The Delta representation is provided to keep the data open and reusable, however, it is managed once, it is paid for once and users should consider it a single data set. Users will only be charged once for the data storage after enabling the KQL DB data availability in OneLake. Enabling the feature is quite simple. All that is needed is for the user to enable the Data Availability option of the selected KQL Database: Once the feature is enabled, you should be able to see all the new data added to your database at the given OneLake path in Delta parquet. You can now access this Delta table by creating a OneLake shortcut from Lakehouse, Data warehouse or directly via Power BI Direct Lake mode. You can enable the OneLake availability at a table or database level. Open Source Connectors for KQL DB Microsoft Fabric has formally announced the release of several open-source connectors for real-time analytics. These connectors enable users to ingest data from various sources and process it using KQL DB. New and Improved Get Data Experience for Real Time Analytics We have released a new Get Data Experience to simplify data ingestion process In KQL DB! Designed with simplicity and efficiency in mind, this update streamlines the way you bring data into KQL DB. Whether you're a seasoned data professional or just starting your data exploration journey, this new user experience is crafted to empower you every step of the way. Simplifying Your Data Flow The primary objective of the new Get Data Experience is to streamline your workflow. We understand the importance of efficiency in data exploration, and that's why we've reimagined the process to minimize the number of clicks required, ensuring that you can bring data from all your familiar data sources including Event streams, One Lake, local file, Azure storage, Event Hubs, and Amazon S3. Guided Wizard The Get Data experience is powered by a new Guided Wizard. This step-by-step assistant takes you through the entire data ingestion process, ensuring that you're never left in the dark. Whether you're extracting data from various sources, transforming it to suit your needs, or loading it into KQL DB, the Guided Wizard is your knowledgeable co-pilot, offering insights and guidance at every turn. The supported data resources that are supported by dedicated guided wizards are: Event Streams: Route Event Streams real-time events to KQL database destinations with no-code experience Onelake: OneLake comes automatically with every Microsoft Fabric tenant and is designed to be the single place for all your analytics data. You can seamlessly ingest data from OneLake into your KQL Database with just a few clicks. Local File: This is the simplest way to insert data into a KQL Database. You can upload a file from your local machine using the Get Data UX. This option is useful when you have a small amount of data that you want to analyze quickly. Azure Storage: If your data is stored in Azure Blob Storage or Azure Data Lake Storage Gen2, you can use built-in support for Azure storage to ingest it. Event Hubs: If you are using Azure Event Hubs to stream your data, you can seamlessly create a connection to your Azure resource from the wizard. Amazon S3: If your data is stored in Amazon S3, you can use built-in S3 support to ingest it. All you need is the URI of S3 bucket or individual blobs. REST API Support for KQL Database We're excited to announce the launch of RESTful Public APIs for KQL DB. The Public REST APIs of KQL DB will enable users to manage and automate their flows programmatically. Here is the list of support REST APIs for the KQL DB: List KQL DBs Get KQL DB details Update KQL DB definition Delete KQL DB You can learn the detailed usage following the Fabric REST APIs public documentation. Splunk Add-on (preview) Microsoft Fabric Add-On for Splunk allows users to ingest logs from Splunk platform into a Fabric KQL DB using the Kusto python SDK. When we add data to Splunk, the Splunk indexer processes it and stores it in a designated index. Searching in Splunk involves using the indexed data for the purpose of creating metrics, dashboards and alerts. This Splunk add-on triggers an action based on the alert in Splunk. We then use Alert actions to send data to Microsoft Fabric KQL DB using the specified addon. The add-on uses Kusto python SDK (https://learn.microsoft.com/azure/data-explorer/kusto/api/python/kusto-python-client-library) to send log data to Microsoft Fabric KQL DB. The addon supports queued mode of ingestion by default. The addon boasts of a durable feature which helps to minimize data loss during any unexpected network error scenarios. But durability in ingestion may impact throughput, so it is advised to use this option wisely. The following describes the stages needed: Download and install the Microsoft Fabric Addon Create an Splunk Index Create a KQL DB and table Create a Splunk Alert that defines the data to be ingested into KQL DB and select the Trigger Actions = “Send to Microsoft”. Configure the connection details to point to the KQL DB Event Streams is now Generally Available We're pleased to announce Event Streams GA with a slew of enhancements aimed at taking your data processing experience to the next level. These updates, focused on enhancing efficiency and user experience, include data transformation to the KQL Database, seamless data ingestion from Eventstream in other Fabric items, and optimization of Lakehouse tables. These changes are made to transform the way you manage your data streams. Event Streams Data Transformation for KQL DB (Generally Available) Now, you can transform your data streams in real time within Eventstream before they are sent to your KQL Database. When you create a KQL Database destination in the Eventstream, you can set the ingestion mode to “Event processing before ingestion” and add event processing logic such as filtering and aggregation to transform your data streams. With this feature, you have greater control and flexibility over your data streams, ensuring that it's perfectly structured for your database. Get Data from Eventstream Anywhere in Fabric If you're working on other Fabric items and are looking to ingest data from Eventstream, our new "Get data from Eventstream" feature is here to help you. This feature enables you to seamlessly ingest Eventstream data to various Fabric items, offering both flexibility and convenience. It simplifies the process of routing your data streams efficiently throughout the entire Fabric environment. The following screenshots show how to "Get data" from Eventstream while you are working with a KQL database and Lakehouse: Create a Cloud Connection within Eventstream We've simplified the process of establishing a cloud connection to your Azure services within Eventstream. When adding an Azure resource, such as Azure IoT Hub and Azure Event Hubs, to Eventstream, you can now create the cloud connection and enter your Azure resource credentials right within Eventstream. This enhancement significantly improves the process of adding new data sources to Eventstream, saving you time and effort. Two Ingestion Modes for Lakehouse Destination To cater to your specific requirements, we've introduced two distinct ingestion modes for your Lakehouse destination. You can select one of these modes to optimize how Eventstream writes to Lakehouse based on your scenario. Rows per file – You can now specify the minimum number of rows that Lakehouse ingests in a single file. The smaller the minimum number of rows, the more files Lakehouse will create during ingestion. Duration – You can now specify the maximum duration that Lakehouse would take to ingest a single file. The longer the duration, the more rows will be ingested in a file. Optimize Tables before Ingesting Data to Lakehouse Our customers have inquired about compacting numerous small streaming files generated on a Lakehouse table. We have a solution for you! Table optimization shortcut is now available inside Eventstream Lakehouse destination - This helps the user by opening a Notebook with Spark job which would compact small streaming files in the destination Lakehouse table. Data Factory Dataflow Gen2 General availability of Fabric connectors The connectors for Lakehouse, Warehouse and KQL Database are officially out of public preview and generally available. We encourage you to use these connectors when trying to connect to data from any of these Fabric experiences. Automatic refresh cancellation When you set Dataflow for a refresh, the dataflow refresh process has a timeout of 8 hours. This means that your Dataflow cannot run for more than 8 hours for a single refresh job. However, during those 8 hours you could be consuming resources and the operation might be in a loop due to an unforeseen situation with your data source or one of the logics in your Dataflow. To prevent unnecessary resources from being consumed, we’ve implemented a new mechanism that will stop the refresh of a Dataflow as soon as the results of the refresh are known to have no impact. This is to reduce consumption more proactively. Error message propagation through gateway We made diagnostics improvement to provide meaningful error messages when Dataflow refresh fails for those Dataflows running through the Enterprise Data Gateway. Support for column binding for SAP HANA connector Column binding is described as follows: Data fetched from the data source is returned to the application in variables that the application has allocated for this purpose. Before this can be done, the application must associate, or bind, these variables to the columns of the result set; conceptually, this process is the same as binding application variables to statement parameters. When the application binds a variable to a result set column, it describes that variable - address, data type, and so on - to the driver. The driver stores this information in the structure it maintains for that statement and uses the information to return the value from the column when the row is fetched. Enabling support for column binding for SAP HANA as an optional parameter will result in significantly improved performance for users that opt into the feature. Staging artifacts will be hidden. When using a Dataflow Gen2 in Fabric, the system will automatically create a set of staging artifacts which support the Dataflow Gen2 operation by providing storage and compute engine for processes that require it such as staging of queries or the Data destination feature. You can find more details regarding this in this earlier blog post: Data Factory Spotlight: Dataflow Gen2 | Microsoft Fabric Blog Starting today, these artifacts will be abstracted from the Dataflow Gen2 experience and will be hidden from the workspace list. No action is required by the user and this change has no impact on any Dataflows that you’ve created before. New Power Query Editor style The Power Query Editor user interface used inside of Dataflows Gen2 has been updated to match the style used throughout Microsoft Fabric. Support for VNET Gateways (preview) We’re also very excited to announce that we’re releasing VNet Data Gateway support for Dataflows Gen2 in Fabric as a Public Preview feature today. The VNet data gateway helps customers to connect from Fabric Dataflows Gen2 to their Azure data services within a VNet without the need of an on-premises data gateway. The VNet data gateway securely communicates with the data source, executes queries, and transmits results back to Fabric. You can learn more about VNET Gateways from the following link. Pipelines Cross workspace “Save as” You can now clone your data pipelines across workspaces by using the “Save as” button. This makes it easier to develop pipelines collaboratively inside Fabric workspaces without having to redesign your pipelines from scratch. Dynamic content flyout integration with Email and Teams activity In the Email and Teams activities, you can now add dynamic content with ease. With this new pipeline expression integration, you will now see a flyout menu to help you select and build your message content quickly without needing to learn the pipeline expression language. Copy activity now supports fault tolerance for Fabric Data Warehouse connector. The Copy activity in data pipelines now supports fault tolerance for Fabric Data Warehouse. Fault tolerance allows you to handle certain errors without interrupting data movement. By enabling fault tolerance, you can continue to copy data while skipping incompatible data like duplicated rows. MongoDB and MongoDB Atlas connectors are now available. We’re excited to announce that the MongoDB and MongoDB Atlas connectors are now available to use in your Data Factory data pipelines as sources and destinations. In your data pipeline, you can create a new connection to your MongoDB or MongoDB Atlas data source to copy, extract, and transform your data. Microsoft 365 connector now supports ingesting data into Lakehouse (preview) We’re excited to share that the Microsoft 365 connector now supports ingesting data into Lakehouse tables. Multi-task support for editing pipelines in the designer from different workspaces You can now open and edit data pipelines from different workspaces and navigate between them using the multi-tasking capabilities in Fabric. String interpolation added to pipeline return value You can now edit your data connections within your data pipelines. Previously, a new tab would open when connections needed editing. Now, you can remain within your pipeline and seamlessly update your connections. Community and Learning Skill up on Fabric with the Microsoft Learn Cloud Skills Challenge We are excited to announce the Microsoft Ignite: Microsoft Fabric Challenge as part of the Microsoft Learn Cloud Skills Challenge. Skill up for in-demand tech scenarios, and enter to win a VIP pass to the next Microsoft Ignite. The challenge is on until January 15, 2024. In this challenge, you will learn how to connect to data, ingest it with Data Factory and notebooks, store it in the lakehouse or data warehouse, and create Power BI reports to turn your data into competitive advantage. The challenge will help you prepare for the Microsoft Certified: Fabric Analytics Engineer Associate certification and new Microsoft Applied Skills credentials covering the lakehouse and data warehouse scenarios, which are coming in the next months.
