Securing Outbound AI Connectivity in SQL Server 2025 (Generally Available)
AI is now part of the data layer SQL Server is evolving beyond a system that only stores and queries data. In SQL Server 2025, the engine can participate directly in AI-enabled workflows through governed integrations. Database-side workloads can connect to external AI services for scenarios such as retrieval-augmented generation (RAG), where model outputs are grounded in authorized, enterprise data context. This shift helps organizations reduce custom middleware and ad- hoc integration paths while keeping governance, security, and access control closer to the data platform. For example, a financial institution analyzing suspicious transaction patterns can execute approved outbound AI calls under database governance instead of exporting sensitive data through scattered application services. However, that shift creates a valid security question: when SQL Server can reach external AI endpoints, how do you maintain control—especially for outbound connectivity to external AI services? SQL Server 2025 addresses this by extending its existing defense-in-depth model. AI connectivity is not treated as an exception path; it is enforced within SQL Server’s existing trust and control model. Why this matters Many teams adopted AI by pushing sensitive data out of the database into application code or integration services. That approach often leads to three issues: Data moves more than necessary. Security policy becomes inconsistent and fragmented across systems. Audit trails are more challenging to correlate. SQL Server 2025 supports a more controlled operating model. AI integration can happen closer to the data, under database governance, while still enabling modern AI scenarios. This approach reduces unnecessary data movement while preserving existing application patterns. This benefits security, platform, and engineering teams by simplifying control and reducing fragmentation. Strong controls over external connectivity SQL Server 2025 supports outbound AI connectivity through features such as direct REST invocation and external model objects. Both rely on explicit SQL Server permissions and approved authentication, but the controls apply differently depending on the feature. External REST invocation uses sys.sp_invoke_external_rest_endpoint to call approved HTTPS endpoints from T-SQL. External model registration uses CREATE EXTERNAL MODEL to define model objects for embedding and vector search scenarios. The shared security idea is simple: define what can be used, grant only the required permissions, and authenticate access with approved credentials or identities. Enablement applies to REST invocation, which must be turned on before use. Permissions control who can enable REST calls, create or alter model objects, and execute approved operations. Authentication determines which credential, identity, or secret is used when SQL Server reaches an external service. For sys.sp_invoke_external_rest_endpoint, SQL Server provides a direct way to invoke HTTPS REST endpoints. The feature is disabled by default in SQL Server 2025 and must be enabled through sp_configure. Changing that setting requires ALTER SETTINGS server permission (implicitly held by sysadmin and serveradmin), which keeps enablement under administrative control. After REST invocation is enabled, runtime use still depends on permissions, endpoint policy, and credential handling, including EXECUTE ANY EXTERNAL ENDPOINT for callers. Organizations should grant invocation rights only to required principals, restrict allowed destinations, and require authenticated requests. Depending on the target service, authentication can use managed identity, certificates, API keys, or other supported mechanisms. For CREATE EXTERNAL MODEL, SQL Server creates a database object that stores the model endpoint, authentication method, and model purpose. Creating or changing that object requires CREATE EXTERNAL MODEL or ALTER ANY EXTERNAL MODEL, while using it requires EXECUTE on the model. This makes model access explicit without implying it uses the same enablement switch as REST invocation. The result is a clearer security boundary: REST calls require explicit enablement and controlled invocation, while external models require governed object creation and execution. In both cases, least privilege, approved authentication, and auditability remain central. High-privilege access is still the real security boundary AI features do not change the core rule of SQL Server security: the real boundary is who has permission to do what. When privileges are tightly scoped, features remain safe to adopt. That’s why least privilege matters. Users and applications should have only the permissions they need for their specific tasks. Nothing more. In practice, this means: Avoid broad administrative role membership wherever possible. Grant only the minimum permission required for the specific operation. Separate operational roles (configuration, model management, invocation). Review role assignments and grants on a defined cadence. Remove stale access quickly after ownership changes. Strong security is not about blocking capabilities. It is about controlling who can use them, when they can use them, and under what conditions, while integrating with existing governance processes. Visibility and accountability are built in Prevention is only one part of security. Detection and response matter just as much. Because these operations execute through SQL Server interface, they can be observed with established SQL monitoring and auditing patterns, including SQL Server Audit and Extended Events. That helps teams answer operational questions such as: Who enabled external connectivity? Which principal invoked an external endpoint? Which external model object was executed? When did behavior deviate from baseline? Governed SQL-side AI execution can improve both incident response and audit readiness. By centralizing execution and logging, organizations can more easily collect, trace, and explain evidence when investigating incidents or responding to compliance reviews. Architecture still determines risk Traditional AI patterns often move data out of SQL Server into other runtime layers before calling external models. This increases the attack surface and fragments policy enforcement. SQL Server 2025 enables a different operating model: Keep data closer to where governance already exists. Use explicit model and endpoint objects with permission checks. Apply outbound network controls and endpoint allow patterns. Centralize audit trails for security and compliance review. In addition, network connectivity and SQL permissions alone should not be the final security boundary. External AI endpoints should also enforce independent authorization decisions, such as RBAC, allowing access only to approved managed identities. This creates an additional layer of protection and helps limit the impact of credential misuse or configuration errors. Secure usage best practices Use this checklist before production rollout: Enable only the connectivity features required for the scenario. Grant least privilege for configuration, model management, and invocation. Restrict outbound traffic to approved destinations. Use managed identities where supported; otherwise use centrally managed, least-privilege credentials. Enforce endpoint-side authorization and RBAC controls for service identities and credentials. Audit configuration changes and invocation activity. Establish behavioral baselines and alert on deviations. Run periodic access and configuration reviews. Include incident response runbooks for external AI call paths. These practices align with standard SQL security guidance and are not unique to AI features. Most control failures come from weak operating discipline around the feature. Strong defaults help, but repeatable processes are what keep systems safe over time. Next Steps Review permission design for AI connectivity operations in a non-production environment. Define approved endpoint policy and outbound control requirements. Pilot one AI scenario with audit-first instrumentation. Document operational ownership for enablement, credential management, and monitoring. Get Started Refer to sp_invoke_external_rest_endpoint, apply the recommended risk-mitigation controls for unauthorized access and data transfer, then validate your CREATE EXTERNAL MODEL permissions in a dev environment using the best practices checklist above, and finalize your SQL Server Database Engine permission design before production rollout.
Change Event Streaming for SQL database in Microsoft Fabric (Preview)
SQL database in Microsoft Fabric already brings operational and analytical workloads together—now it can stream changes as they happen. Change Event Streaming sends inserts, updates, and deletes directly to Fabric Eventstream or Azure Event Hubs. Build near-real-time pipelines from operational data without leaving Fabric.A new analytics frontier: GPU-accelerated Fabric Data Warehouse (Early Access Preview)
As data volumes grow, concurrency rises, and analytics workloads become more dynamic and AI-driven, performance becomes harder to predict and harder to scale. Every query sits in the critical path, adding pressure to the warehouse, and every second counts. This is the core tension in analytics today. The expectations have changed, but the underlying technology has not, leaving agents, applications, and AI systems waiting on data. To meet this moment, analytics needs a new kind of execution engine.Microsoft recognized as a Leader in The Forrester Wave™: Data Lakehouses
For years, organizations have invested in data platforms to understand what happened across their business. Dashboards, reports, and KPIs are now table stakes. But as AI becomes central to how organizations operate, the bar for the data lakehouse is getting much higher. The next generation of applications and agents needs governed access to every kind of data: structured, semi-structured, and unstructured; batch, streaming, and real-time—working together on one open foundation. When that data remains spread across fragmented systems, teams are left reconciling copies, duplicating governance, and stitching together context before they can create value. Today, we're proud to share that Microsoft has been recognized as a Leader in The Forrester Wave™: Data Lakehouses, Q3 2026. In the report, Forrester describes Microsoft Fabric as “a strong fit for enterprises seeking a unified, AI-enabled lakehouse platform integrated with the Microsoft ecosystem.” We believe this recognition reflects the bold vision behind Microsoft Fabric and Microsoft OneLake: helping organizations eliminate the integration tax of fragmented data estates by bringing data, analytics, governance, and AI together on one open lakehouse foundation. Fabric: A unified foundation for the AI-era lakehouse The Forrester report frames the modern lakehouse as more than a system of record for analytics. As agentic AI systems begin to reason, plan, and act on enterprise data, the lakehouse is becoming the operational foundation where intelligence is grounded and activated in real time. Microsoft Fabric was built for this shift. With OneLake, Fabric gives organizations a single, governed data lake and one SaaS platform where data teams, analysts, developers, and business users can work from the same trusted foundation. Forrester notes that Microsoft’s approach emphasizes deep integration across Power BI, Copilot, Microsoft 365, OneLake, databases, and AI services—helping unify analytics, operational, and AI workloads within a single ecosystem. Forrester also highlights Microsoft’s “bold vision of a unified, AI-powered, open data platform that brings together analytics, data engineering, business intelligence, and operational data.” Furthermore, they add that “innovations such as OneLake shortcuts, mirroring, AI-powered transformations, and cross-cloud interoperability support this vision by reducing silos and simplifying access to distributed data.” One open foundation for data and analytics OneLake is the governed data lake at the heart of Fabric, designed to unify your entire multi‑cloud data estate. It connects data across clouds and on‑premises systems using zero‑copy, zero‑ETL access, so teams work from a single, governed copy of data. With native support for open formats like Delta Lake and Iceberg, this data remains accessible from any analytics engine or platform, including Microsoft Fabric, Snowflake, and Azure Databricks. Once data is connected or stored in OneLake, the OneLake catalog helps secure, govern, and organize it into a logical data mesh, making trusted data easy for everyone to discover and use. Govern once, across every engine Security, identity, lineage, and governance are built into Fabric rather than bolted on tool by tool. State of the art OneLake security can define object-, row-, and column-level controls once and enforce them consistently across Spark, SQL, KQL, Power BI, Copilot, and third-party engines through OneLake security APIs. The OneLake catalog centralizes sensitivity labels, classification, and end-to-end lineage, helping organizations simplify governance while giving users trusted access to the data they need. Every workload on one lakehouse Fabric brings relational, real-time, analytical, document, and vector workloads into one platform experience on OneLake. Spark powers data engineering with the Native Spark Execution Engine in Microsoft Fabric, which accelerates workloads by running much of the execution in highly optimized native C++ code with vectorized processing, while preserving the same Spark APIs, notebooks, and DataFrame code users already know. With the native execution engine, Spark in Fabric delivers up to 6x faster performance than open-source Apache Spark, helping improve price performance by completing the same workloads with less compute and lower costs. Additionally, the warehouse engine and lakehouse SQL endpoint serve interactive queries over the same Delta tables; Real-Time Intelligence supports streaming and event-driven scenarios; and Power BI queries OneLake directly through Direct Lake without importing or moving data. AI native to the data platform Fabric’s integration with Copilot and agents enable natural language analytics and intelligent automation at scale. Vector embeddings can sit alongside structured data for AI retrieval, so teams can build analytics, AI, and applications on one governed copy of data. Instead of moving data to each workload, organizations can bring more workloads and more AI-powered experiences to the same open lakehouse. Why organizations choose Fabric Customers are seeing real impact from using Fabric: less duplication, cleaner governance, faster development, and a simpler path from data to AI. At the foundation is a common pattern: organizations are consolidating fragmented data estates onto a single governed lakehouse. London Stock Exchange Group (LSEG), a partner to the world's leading financial institutions, set out to simplify a complex, fragmented data landscape and give its customers a consistent, unified experience. Using Fabric, LSEG is building a unified data platform that consolidated 30 systems, 1,200 datasets, and 33 petabytes of data, accelerating product development, improving data quality, and advancing AI readiness. Product development timelines have moved from years to months, delivering faster, cleaner data to everyone from global firms to individual traders. "When you need to pull data together across disparate sources that are in different formats and varying levels of modernisation and maturity, it makes it difficult to react to market demand quickly. We knew it would be far more efficient to bring everything into a single, modern platform. It would mean we could run the organisation leaner and react to market demand faster.” Dave Byrne, Group Head of Data Platforms at LSEG Once data is unified, organizations can apply governance and analytics at enterprise scale. UNC Health standardized its enterprise data estate on Fabric, creating a single, governed lakehouse foundation for clinical analytics, operations, population health, and research. Fabric powers UNC Health’s AI solutions, which reduced care-gap chart review time by nearly 50% and provides the governed data foundation for a secure research environment supporting 25 active studies. That same foundation also creates new opportunities for AI-driven innovation. Eastman, a global specialty materials company, adopted Fabric to modernize its legacy data architecture and create a unified, governed lakehouse foundation for analytics and AI. Using OneLake shortcuts and data mirroring, Eastman shares data across domains without unnecessary duplication, ingested roughly one billion rows from eight systems, and established a scalable platform for analytics, machine learning, and AI-powered applications. “We operate in a lot of markets that are fundamentally different from one another. Being able to aggregate all this loose, unstructured data into something that’s actionable is really helping our commercial organization build better strategies for the year ahead.” —Andrew Ervin, Manager of Generative AI, Eastman Taken together, these examples illustrate the evolution of the modern lakehouse: first unifying data, then governing it consistently, and ultimately turning it into a foundation for AI-powered innovation. This is the shift that many organizations are making as they prepare for the next generation of applications and agents. Strategic takeaways for enterprise leaders Three shifts stand out for leaders preparing their organizations for the next generation of AI: The lakehouse is now the default foundation for AI. Agents and AI applications need governed access across every data type; not siloed systems stitched together after the fact. Openness prevents lock-in. Open table formats and bi-directional interoperability help organizations unify their estate without walking away from the tools and platforms they already run. Reducing duplication changes the economics. Bringing workloads to a shared, governed foundation helps organizations simplify operations, strengthen consistency, and accelerate innovation. As organizations prepare for the next generation of AI-powered applications and agents, the need for a unified, open, and governed data foundation will only grow. Microsoft Fabric was built to bring data, analytics, governance, and AI together on that foundation, and we’re excited to keep innovating alongside our customers. Forrester’s recognition reinforces what we’ve believed from the start: the organizations that succeed in the AI era will be those that eliminate fragmentation, simplify governance, and build on one open lakehouse. Learn more Read the complimentary report. Explore Microsoft OneLake and Microsoft’s vision of an open data lake ecosystem. Join us at the next Fabric + SQL Community Conference in Barcelona to hear directly from our product and teams and community members. Statement from Forrester Forrester does not endorse any company, product, brand, or service included in its research publications and does not advise any person to select the products or services of any company or brand based on the ratings included in such publications. Information is based on the best available resources. Opinions reflect judgment at the time and are subject to change. This report is part of a broader collection of Forrester resources, including interactive models, frameworks, tools, data, and access to analyst guidance. For more information, read about Forrester's objectivity here.Fabric November 2024 Feature Summary
Welcome to the November 2024 update for Microsoft Fabric! This month, we're excited to bring you a host of new features and improvements designed to enhance your experience and productivity. From the introduction of Copilot in Power BI mobile apps to the new Fabric Databases, there's something for everyone. Whether you're looking to streamline your data analysis, improve your reporting capabilities, or simply stay up to date with the latest innovations, this update has you covered. To learn more, read about all these announcements, and more in Arun’s blog post Accelerate app innovation with an AI-powered data platform | Microsoft Fabric Blog Be one of the first to use SQL database on Fabric In this series on SQL database on Fabric, you will learn how Fabric brings together both transactional and analytical workloads, creating a truly unified data platform. You’ll also learn how developers can build reliable, highly scalable applications where cloud authentication and encryption are secured by default. Starting December 3rd, join us for six sessions with database experts and see just how easy it is to get started. Sessions are available live and on-demand. View the sessions and register for the series. Don’t miss Microsoft Ignite 2024 and FabCon 2025 Attend Microsoft Ignite 2024 online for free November 19 – 21, 2024 to learn about the latest innovations in Data & AI. Join sessions that will cover solutions that help modernize and manage intelligent applications, safeguard data, and accelerate productivity. Join us at FabCon Las Vegas from March 31 to April 2, 2025, for the ultimate Microsoft Fabric, Power BI, SQL, and AI community-led event. With more than 144 sessions, 18 pre- and post-conference workshops, unique community experiences, a dedicated pre-day for partners, all-day Ask-The-Experts hours, 20+ expo booths, plus after-hours events and socials you don't want to miss. Contents Be one of the first to use SQL database on Fabric Don’t miss Microsoft Ignite 2024 and FabCon 2025 Certifications Get certified in Microsoft Fabric - for free! Explore the newest Fabric certification for Data Engineers Copilot and AI Copilot in Power BI mobile apps (Preview) Copilot summaries in subscriptions (Preview) Reporting Path Layer for the Azure Map visual Visual calculations (Preview) ‘Set Alert’ with Activator and Real-Time Intelligence (Generally Available) Small multiples for the new card visual (Preview) New visual - text slicer (Preview) Modeling Define new measure in DAX query view quick queries Metric sets: a new era of metric management in Fabric (Preview) Performance improvements for models with calculation groups and format strings in Excel DLP policies restrict access action for semantic models (Preview) Semantic modeling in Visual Studio Code with the new TMDL extension (Preview) Developers + APIs Fabric Git: TMDL format for semantic model export Semantic model client library updates Visualizations KPI by Powerviz Zebra BI Tables 7.3 Waterfall PRO by ZoomCharts: the most interactive waterfall visual for financial data Lollipop bar chart by Nova Silva Sales velocity chart Donut Chart by JTA New book: Data Visualization with Microsoft Power BI Other Support for Power BI language settings when a paginated report is viewed on the Power BI service Platform Introducing OneLake catalog Tenant switcher control Automate GitHub integration with Microsoft Fabric REST APIs Switch branches from the source control pane Announcing general availability of the Fabric Workload Development Kit OneLake External data sharing is now generally available Mirroring Introducing Open Mirroring Fabric Database Mirroring Public REST APIs are now generally available Mirroring for Azure SQL Database now Generally Available Introducing Mirroring for Azure SQL Managed Instance (Preview Databases Introducing Fabric SQL database (Preview) Data Warehouse Cold query performance improvement Service principal support for Fabric Data Warehouse Data Engineering Notebook display chart upgrade Fabric API for GraphQL is now generally available with exciting new features Esri’s ArcGIS GeoAnalytics integration with Fabric Spark (Preview) Jar libraries are now supported in Fabric Environments Support of spaces and special characters in Delta table names Data Science Introducing low code AutoML Real-Time Intelligence Ingest & Process Announcing the general availability of Real-Time Hub Announcing the general availability of Enhanced Eventstream Announcing the general availability of connector sources in Eventstream Introducing Azure Service Bus Connector for Eventstream (Preview) New Fabric events (Preview) Eventstream Data Preview on database CDC sources Monitoring experience on connector sources with Runtime Logs and Data Insights in Eventstream Processing and routing events to Activator with Eventstream (Preview) Introducing Eventstream’s CI/CD support Automate Eventstream Item Operations with Eventstream REST APIs Stream Data to Eventstream Securely using Entra ID Authentication Analyze & Transform Eventhouse monitoring (Preview) Eventhouse Query Acceleration for Shortcuts (Preview) Synapse Data Explorer to Eventhouse migration (Preview) New explorer for database objects in KQL Queryset Entity Diagram view in KQL Database Visualize & Act Easily share Real-Time Dashboards with others Announcing the general availability of Real-Time Dashboards Announcing the general availability of Activator Data Factory Table and partition refreshes added to semantic model refresh Import and export your Fabric Data Factory pipelines New connectors available Simplify data ingestion with Copy Job - CI/CD upsert & overwrite New capabilities in Copilot for Data Factory to efficiently build and maintain your Data pipelines OneLake datahub is now the OneLake catalog in Modern Get Data experience Dataflows now support CI/CD (Preview) https://youtu.be/eyjVj-k8m1M?si=Jt1-TkIY58r8LyKr Certifications Get certified in Microsoft Fabric - for free! Get ready to fast-track your career by earning your Microsoft Certified: Fabric Analytics Engineer Associate certification. For a limited time, we will be offering 5,000 free DP-600 exam vouchers to eligible Fabric Community members. Complete your exam by the end of the year and join the ranks of certified experts. Don’t miss this opportunity to get certified. Explore the newest Fabric certification for Data Engineers We are excited to announce a brand-new certification for data engineers. The new Microsoft Certified: Fabric Data Engineer Associate certification will help you demonstrate your skills with data ingestion, transformation, administration, monitoring, and performance optimization in Fabric. To earn this Certification, pass Exam DP-700: Implementing Data Engineering Solutions Using Microsoft Fabric, currently in beta. Copilot and AI Copilot in Power BI mobile apps (Preview) We’re excited to announce the release of Copilot in Power BI mobile apps (Preview)! This new feature brings the power of AI directly to your fingertips, enhancing your mobile experience when you’re on the go, offering a quick and simple way to dive into your data. With Copilot in Power BI mobile apps, you no longer need to analyze data yourself. Copilot provides report summaries and insights that allow you to make informed decisions anytime and anywhere. Imagine a sales manager effortlessly pulling up an executive summary of the latest sales report with a single tap, or a maintenance technician getting real-time machine-performance insights while on the factory floor. To start using Copilot on your mobile app, simply tap the Copilot button located in the report header (that meets Copilot requirements in Power BI) and choose whether you want to get a summary or to look into insights. Copilot will deliver a response based on your request. You can then copy and share the response or keep interacting with Copilot by choosing from the suggestions at the bottom. These suggestions can help you tweak the response or create new requests. For more details about Copilot in Power BI mobile apps, check out our full blog post. Copilot summaries in subscriptions (Preview) Do you need to extract insights from Power BI report images in your email or quickly digest a summary of your Power BI report? Subscribe to Copilot summaries for Power BI reports. This feature is available with Standard subscriptions and for reports in a copilot-eligible capacity. Learn more about using Copilot in Power BI and Fabric. Set up the copilot summaries for Power BI reports that you subscribe to as follows: Select the ‘Subscribe’ option from the ribbon for the Power BI report that you are interested in. 2. Select ‘Standard Subscription’. 3. You can subscribe to the report. Learn more about creating report subscriptions. 4. Add a copilot summary to the email delivered by the subscription. If you are eligible, your subscription will receive the Copilot summary by default. Learn more about setting up Copilot summaries for subscriptions. 5. You can ‘Preview summary’ to view a sample of what the summary might look like. 6. Test your subscription by selecting ‘Send Now’ after you Save the subscription. Note: ‘Send Now’ will deliver the email with the copilot summary to all recipients. Email Sample: Learn more about Copilot summaries in subscriptions from our documentation. This feature will roll out gradually over the next few weeks and is not available in Gov clouds. Copilot and AI demo https://youtu.be/eyjVj-k8m1M?si=MYJkFPH5FFdNtEBT&t=160 Reporting Path Layer for the Azure Map visual This month we’re introducing a new feature to the Azure Map visual that takes geospatial analytics to the next level -- the Path layer. The Path layer provides users with the ability to visualize geographic connections between multiple points. Whether you’re managing logistics, analyzing network traffic, or tracking asset shipment across the globe, this feature allows you to visualize connections between multiple geographic points in an intuitive and interactive way. The Path layer is ideal for several key scenarios, for example: Network Analysis: For industries like telecommunications, the Path Layer enables you to map intricate network connections. It helps identify inefficiencies, monitor data flow, and strengthen critical infrastructure. Flight Path Analysis: Airlines can leverage the Path Layer to visualize and analyze flight routes, improving air traffic management. It helps identify new route opportunities and enhances the overall passenger experience by optimizing existing routes. To get started, add the location for each point using either a geocoded location field, such as city names, or latitude and longitude. Then, differentiate between the paths by adding a field to the Path ID field well and indicate the order of connection through the Point Order field well. For example, you could create a map showing the path of two ships with the latitude and longitude of their positions for each point, a path field with a unique identifier for each ship, and a timestamp for each location to make sure the points are connected in the correct order. You can also format the visual by controlling the color, transparency and width of the lines, and even turning off the bubble markers for each point. If you turn off the bubble layer, you’ll only see a bubble on hover showing you the closest point to your pointer location. Paths are interactive as well, so you’ll get tooltips on hover and be able to cross-highlight other visuals by clicking on points of the lines. There are a couple of unique behaviors to be aware of with this new layer. First, when using a drill hierarchy with the path layer, the visual will automatically drill down to the lowest level and will not allow you to drill up, as points in the path would be aggregated at higher drill levels. Next, if you have a location that’s part of multiple paths, the bubbles for that location show up on top of each other. If you want to click on the bubbles underneath, just hover on the line associated with its path, and it will float to the top and be selectable. Lastly, you can further break down the paths by adding a legend, which will create unique lines for each legend value of a given path ID. An additional point to consider: Currently, the path layer operates mainly in conjunction with the bubble layer. Once you add a path to your map, you’ll see that the filled, cluster bubbles, heat map, and 3D column layers are all disabled. Additionally, while you can use the path layer in conjunction with reference layers, the reference layer will be static. It’s currently unsupported to mix data bound reference layers with the path layer. The path layer is still actively rolling out to all regions. Depending on what region your tenant is in, you might not see the path layer in the Power BI service through the weekend. Be sure to check the report after publishing, and if you don't see the layer, it should be accessible within a week. We’re excited to see what you create with this new path layer. Give it a try and let us know what features you’d like to see next! Visual calculations (Preview) The work on visual calculations continues as usual and this month we are adding a highly requested item: support for exporting! You can now export data from visuals that contain one or more visual calculations or hidden fields. If you export data, hidden fields on a visual are not included in the export, unless you export the underlying data. The results of visual calculations are always included in the export, except when exporting underlying data, since visual calculations are not part of the underlying data as they only exist on the visual. Learn more about visual calculations in our documentation. ‘Set Alert’ with Activator and Real-Time Intelligence (Generally Available) Back in December we announced the preview of alerting capabilities within Power BI reports using Real-Time Intelligence Activator, part of Microsoft Fabric. We are excited to announce that this capability is now generally available! With GA, you’ll be able to: Stay on top of your critical metrics by monitoring your business objects. You can track and analyze key business objects such as individual packages, households, refrigerators, and more in real-time, ensuring you have the insight needed to make informed decisions. Whether it’s understanding how individual instances of your business objects impact sales figures, inventory levels, or customer interactions, our monitoring system provides detailed insights, helping you stay proactive and responsive to changes in your business environment at a fine-tuned level of granularity. Unlock the full potential of creating business rules on your data with advanced data filtering and monitoring capabilities. This update offers a wide array of options for filtering, summarizing, and scoping your data, allowing you to tailor your analysis to your specific needs. You can set up complex conditions to track when data values change, exceed certain thresholds, or when no new data has arrived within a specified timeframe. Ensure your communications are perfect before hitting send by seeing a preview of your Email and Teams messages. This will allow you to see a preview of your message exactly as it will appear to the recipient. Review your content, check formatting, and make any necessary adjustments to ensure clarity. With this feature, you can confidently have Data Activator send messages on your behalf knowing they look just the way you intended. Set up rules that trigger automatically with every new event that comes in on your stream of data. Whether you need to send notifications or initiate workflows, this feature ensures that your processes are always up-to-date and responsive. We renamed our feature to help create clarity about what it is and what it does. If you are used to seeing Reflex, please note that it is now called Activator. The items you create to set up rules and actions are, therefore, activators. We also enabled capacity usage reporting, to help you better understand your capacity consumption and future charges. Our billing is based on Storage used for events retention, and Compute resources: the number of rules running, the number of events per second ingested, rules evaluation and activation. For more on Activator meters and billing stay tuned for the detailed RTI Billing Blog post coming soon. You can learn more about the updates in GA through our blog. We continue to improve our capabilities, and we’d love to hear your feedback. Please share your ideas or suggestions. Small multiples for the new card visual (Preview) With this month’s update, we’re enhancing the Card visual with a new version that retains all familiar features and updates, while adding advanced functionality and an improved user experience with small multiples. This new feature is currently in preview with the new Card visual, offering an excellent opportunity to experience the capabilities of the feature. Small multiples are a series of similar card tiles displayed together in a grid format, each representing a different category or dimension of data, allowing for easy OKR comparison and analysis across multiple fields. This newly added feature enhances data organization, visual clarity, and performance, making it easier to analyze and present data effectively. To try it, navigate to Options and settings > Options > Preview features > New card visual, and make sure it’s enabled. Another advantage of the new Small multiples feature is the extensive customization it offers, including: Small multiples layout: Choose from single column, single row, or grid, and customize the number of small multiples, rows, or columns displayed. Advanced formatting options: Enhanced features such as font styles, color-coding, and conditional formatting. Border and gridlines: When enabled, individual controls for borders and gridlines permit the customization of style, width, color, and transparency. Overflow style: Options include continuous scroll or paginated, to smoothly navigate through multiple cards without overwhelming visual space. Headers: Choose from horizontal or vertical orientation, top or left position, customizable alignment, font, color, transparency, padding, plus background color or image. To create a card visual with Small multiples, first select the Card (new) icon from the visual gallery on the Build visual tab in the Visualizations pane, then select some data fields from the data model to add them to the data field well. To categorize your cards using small multiples, choose a data field from the data model and add it to the Small multiples data field well. This new feature provides extensive customization options, such as layout, advanced formatting options, conditional formatting, borders and gridlines, overflow style, and customizable headers. Small multiples for the Card visual in Power BI offer another great enhancement that significantly improves data organization, visual clarity, and performance. The Core Visuals team continues to add new features and greater functionality, and we’re committed to improving our capabilities. We invite you to explore this new feature and share your feedback with us in the comment section below as we continue to improve our Card visual capabilities. For more information we encourage you visit the Core Visuals blog on LinkedIn. New visual - text slicer (Preview) Introducing the new text slicer, now available in our core visuals gallery. This month brings the arrival of the new text slicer in Power BI offering new possibilities for both users and the organization. Enable the new text slicer by navigating to Options and settings > Options > Preview features > text slicer visual to ensure its selected, and restart Power BI. The text slicer works by allowing users to input specific text that acts as a filter, targeting a designated data field. By entering the desired text in the slicer’s input box, the slicer effectively narrows down the dataset to display only the relevant information that contains the entered text. This functionality is particularly useful for handling large datasets, where quick and precise filtering is essential for efficient data analysis and presentation. To create a text slicer visual, select the text slicer icon from the visual gallery on the Build visual tab in the Visualizations pane. This adds a visual placeholder to the report canvas. To filter a dataset, add a text field from the data model to the Field well to establish the text slicer's functionality, allowing it to filter the dataset based on user input. Simply add text to the slicer’s input box, select the apply icon, or press enter, and the slicer immediately filters the dataset, displaying results on the visual. As shown here, the new text slicer introduces a powerful and customizable filtering tool in Power BI: Improved user experience: The text slicer provides users with a straightforward and efficient method to filter input. Unmatched customization: It offers numerous options for users to tailor their filter experience to their needs and preferences. The Core Visuals team is dedicated to enhancing our features and functionality continuously. We are committed to advancing our capabilities and highly value your feedback. Kindly share your insights regarding this capability in the comments section below. For more information we encourage you visit the Core Visuals blog on LinkedIn. Reporting demos https://youtu.be/eyjVj-k8m1M?si=kH1kS1oSDCy5_xZK&t=516 Modeling Define new measure in DAX query view quick queries Creating measures in DAX query view just became even easier. The quick queries option available from the context menu of tables, columns, or other items in the Data pane, now includes Define new measure. This will create a new query table with the syntax started for you to create a query-scoped measure DAX formula ready for you to add your own DAX formula and then run when ready. Learn more about DAX query view and the other quick queries available at DAX query view - Power BI | Microsoft Learn. Metric sets: a new era of metric management in Fabric (Preview) The preview of metric sets is now officially available for both service and desktop. This is a transformative new feature designed to redefine how organizations manage and consume metrics. The Fabric Metric Layer’s home base is the Metrics Hub in Power BI and brings powerful capabilities to streamline metric management, ensure consistency, and foster trust in data across your organization. Metric sets will be available for both consumers to browse, and creators to use in reporting. A service experience consisting of visualized metrics and data exploration will allow end users to answer their data questions. The desktop experience will allow creators to connect to the most authoritative metrics to visualize in reports. Key Features: Curated Collection of Metrics: metric sets will serve as a collection of measure pointers to source semantic models and include key dimensions so end users and authors alike can unambiguously understand how a metric should be grouped or used. Rich Consumption Experiences: Users can explore and consume metrics from the metric set itself, allowing for deep insights and understanding. Copilot summaries and multiple visuals will be available for users to scroll through and go from data to insights in seconds. Efficiency: Consumers no longer need to rely on report creators to answer questions or build custom reports for specific needs. Consumers can leverage the Explore dialog to dig deeper into a given metric in an environment where everything in the data pane ‘just works’ because the dimensions have been curated specifically for the metric. Discoverability and Reuse: Consumers - Metrics are discoverable via search, and metric sets can be promoted, endorsed, certified just like any artifact so that users trust it. Consumers can also leverage the Explore dialog to dig deeper into a given metric in a safe environment where everything in the data pane ‘just works’ because the dimensions have been curated specifically for the metric. Authors- Metrics in Desktop: In the November release of desktop, metric sets will be available to connect to and use in desktop reporting. You can access the metric you want to include in your model via OneLake datahub / data catalog and connect there. This ensures your reports use the most up to date and authoritative measures available. Stay tuned for the upcoming milestones and get ready to transform your metric management experience with Metrics Hub! Performance improvements for models with calculation groups and format strings in Excel We’re excited to announce significant performance improvements for MDX queries on models with Calculation Groups and Format Strings! The latest changes should greatly improve the performance and reliability of operations in Analyze in Excel on models that include one or both of: Dynamic Format Strings for Measures. Calculated Items with Format Strings. This extends to other MDX scenarios as well, so all client applications that use MDX to query semantic models with the above will experience the same performance benefits. DLP policies restrict access action for semantic models (Preview) Purview data loss prevention policies for Fabric now enable admins to restrict access based on the sensitive information detected within their semantic models’ data. When Purview compliance admins configure DLP policies for Fabric, they now have the option to decide if upon detecting sensitive information they would like to block access to the data. They have the option to prevent guest users from accessing the data or to restrict access for all users except the data administrator. In Fabric, data admins will see an indication that their data is restricted, and can act, such as reporting an issue to the compliance admin or override the policy rule. Consumers, such as guest users who have been now restricted from seeing this information, also see an indication letting them know that an organization policy revoked their access, and if they attempt to see its content, they will not be able to. With restrict access action, compliance admins get further control and enforcement when uncovering sensitive data in their Fabric tenant. Semantic modeling in Visual Studio Code with the new TMDL extension (Preview) Power BI developers, the new TMDL Extension for Visual Studio Code in public preview enhances your TMDL editing experience, boosting semantic model development. The Tabular Model Definition Language (TMDL) is designed to make model representations readable, editable, collaborative, and reusable. The TMDL Extension builds on these strengths of TMDL with several key features that create a rich development experience: Semantic Highlighting: Improves readability by applying different colors to parts of your code based on meaning, making it easier to understand the structure and functionality of your TMDL briefly. Error Diagnostics: Helps you identify and fix issues in your code by clearly highlighting errors and providing you with detailed messages that guide you on how to resolve them. Autocomplete: Offers intelligent suggestions while you type to speed up your workflow, reduce the chance of errors, and help you understand your code options More features on the way! By working in Visual Studio Code, you can also take advantage of other fantastic tools on the platform such as: Source Control: Seamless integration with Git, allowing you to track changes, collaborate with team members, and version control your semantic models. GitHub Copilot: An AI coding assistant that will help you write code faster, generate TMDL from natural language, and quickly apply advanced bulk edits to your models. Download the TMDL Extension on the Visual Studio Marketplace and see how you can accelerate your semantic model development today! Modeling demos https://youtu.be/eyjVj-k8m1M?si=IvBsLYzRyRaOZKI2&t=1104 Developers + APIs Fabric Git: TMDL format for semantic model export As part of our commitment to providing a developer-friendly experience that enhances team collaboration, Fabric Git integration will begin exporting semantic model definitions as Tabular Model Definition Language (TMDL) in January 2025. This change will replace the use of a single JSON file (model.bim) with Tabular Model Scripting Language (TMSL). Due to its folder representation and readable format, TMDL offers a significantly improved source control experience. This enhancement facilitates tracking commit history and simplifies the resolution of merge conflicts, particularly when compared with TMSL. If necessary, you can continue to obtain the TMSL representation of your semantic model by using the Get Semantic Model Definition REST API or XMLA Endpoint. Semantic model client library updates Client applications, such as Excel or Power BI Desktop, connecting to Power BI semantic models now benefit from better performance due to an automatic conversion of legacy connection strings (e.g. pbiazure://*) to the XMLA endpoint. Requests are routed directly through the XMLA endpoint, reducing intermediary steps, speeding up request processing, and decreasing the likelihood of errors. You may need to update your firewall rules. See the troubleshooting document for details. Please ensure that you are using the latest Analysis Services client libraries for optimal performance when connecting to Power BI semantic models. Visualizations KPI by Powerviz KPI by Powerviz (Power-BI Certified) is a powerful custom visual for Power BI that allows users to visualize and create eye-catching and advanced Key Performance Indicators (KPIs). Key Features: 100+ Prebuilt KPI templates within visual and option to create own templates. Design: 16 layers and 40+ chart variations to create infographic designs. Rich customization, formatting options, and color styles. Create KPI objects in layers, combining charts, metrics, and icons. Analytical: Data Visualization Types: Categorical: Compare values across categories. Comparison: Analyze differences between values. Composition: Show parts of a whole. Progression: Display trends over time. Actual vs Target: Compare actual against targets. Formatting Features: Configure the Ranking, Sorting, Axis, Number-Formatting, Tooltip, Gridlines, Data Labels and Series Labels for visuals. IBCS Theme Support: Includes deviation bars, series labels, and consistent color scheme. Small Multiples: Support for all chart types – Fixed/Fluid with change chart feature. Other features include multi-categories comparison, Highlight values, Layer Flexibility, and more. Business Use Cases: Sales Performance, Financial Health, Customer Satisfaction. Try KPI Visual for FREE! Check out all features of the visual Step-by-step instructions YouTube Video Link Learn more about visuals Follow Powerviz Zebra BI Tables 7.3 With Zebra BI Tables 7.3, users can harness the power of a rich text editor to create and update visual comments with remarkable efficiency. This feature empowers you to style and format your text, add bullet points, and insert hyperlinks, making your report a one-stop shop for the entire team by just leaving links to reports and documentation different stakeholders might have an interest in. Well-structured comments can streamline communication within your reports, enabling readers to quickly grasp essential insights. By emphasizing what’s important and explaining why it matters, you guide your audience towards critical information and promote clarity and understanding. This clarity is crucial in any business environment, where time is often limited, and strategic decisions must be made swiftly. Effective comments reduce the time and effort required to generate actionable insights, which ultimately improves report quality and effectiveness. Incorporating thoughtful commentary can transform a standard report into a powerful tool for decision-making. With Zebra BI Tables, enhancing your reports with meaningful comments has never been easier -- all so you can communicate your message more effectively and engage your audience better. Learn more from our video example of the rich text editor in Zebra BI Tables 7.3. Waterfall PRO by ZoomCharts: the most interactive waterfall visual for financial data Waterfall PRO by ZoomCharts is the most user-friendly and insightful way to visualize financial data, combining incredible user experience with customizability and powerful features. It also seamlessly cross-filters data across multiple visuals, allowing you to create truly interactive Power BI reports. Main Features: Custom Sequence: Have full control over the column order with the Sequence field. Drill Down: Use multiple categories to enable drill down directly on the waterfall chart. Automatic Subtotal Calculation: Display subtotals even if you don’t have them in your data. Rich Customization: Customize X and Y axes, legends, tooltip content, and adjust the appearance settings for positive, negative and total columns separately Thresholds: Display up to four constant or dynamic thresholds as lines or areas. Cross-chart filtering: Dynamically filter data across multiple visuals. Get Drill Down Waterfall PRO on AppSource Visit Product Page Lollipop bar chart by Nova Silva We’re thrilled to continue receiving your valuable feedback, and we appreciate your contributions in helping us improve our visuals. In our latest Lollipop Bar Chart release for Power BI, we’ve added a much-requested feature: secondary markers. This allows you to display not only the primary value but also add context by including a secondary value marker. This new feature integrates seamlessly with all other Lollipop Bar Chart functionalities, such as transforming the Lollipop Bar Chart into a dot plot by removing the connecting bars, as shown in the second image. This also removes the requirement to start your numeric scale at 0, allowing you to have a closer look at the values and their differences. While standard bar charts are great for comparing a single measure across categories, they can become cluttered with larger datasets (>10 categories). The colored bars may fill too much of the chart space. To address this, the Lollipop Bar Chart offers a cleaner, more efficient alternative, minimizing clutter without sacrificing clarity. Try the Lollipop Bar Chart for FREE now on your own data by downloading it from the AppSource. Questions or remarks? Visit us at: https://visuals.novasilva.com/. Sales velocity chart The Sales Velocity chart is a unique tool for analyzing product sales and profitability in specific countries. It uses a combination of pie charts, needles, and color coding to visually represent key metrics. Key Features: Needles: Length indicates sales percentage; width reflects profit margin. Pie & Circle Size: Reflects overall current sales in a country. Color Coding: Green (high profit), Yellow (moderate), Red (low profit). Sales Trend Dot: Gray (no data), Red (decreasing sales), Green (increasing sales). Benefits: Visual Clarity: Easy to understand data representation. Dynamic & Scalable: Handles large datasets and adapts to screen size. Interactive Features: Tooltip displays details, premium options offer filtering and logo removal. Use Cases: Businesses can identify top sales regions and areas needing improvement. Financial analysts can pinpoint high and low profit contributors. Note: For more information, visit our website. Watch a short video. Read this Sales Velocity Chart documentation. For any queries, questions, or requests, please write to us. Donut Chart by JTA An innovative visualization tool that segments data into three clear categories: Positive, Neutral, and Negative. This format is particularly effective for sentiment analysis, offering clear insights into the overall distribution of opinions or data points. Enhance your data visualization effortlessly with this versatile tool. Key Features: Personalize Colors: Tailor the look of your chart by adjusting the color scheme of each slice to reflect your brand or style. Customize Text: Make it uniquely yours! Modify titles, legends, values, and percentages, adjusting margins, colors, fonts, and alignments to perfectly match your design preferences. Shape the Visual: Personalize the entire chart—adjust the circumference, tweak the colors, and refine the overall look and feel to suit your needs. Target Comparison: Easily compare your metrics against specific targets for clearer insight. Icon Customization: Set your own indicators! Choose custom icons to represent performance below or above your target. Conditional Formatting: Effortlessly apply color-coded formatting to highlight how values measure up against their target. Animation Control: Smooth transitions! Enable or disable animations to enhance or streamline your visual experience. Download Decomposition Tree by JTA for free: AppSource Try Decomposition Tree by JTA: Demo Learn more about us: JTA The Data Scientists New book: Data Visualization with Microsoft Power BI We recommend the new book ‘Data Visualization with Microsoft Power BI’ by Alex Kolokolov & Maxim Zelensky, the first book that delivers DataViz best practices for Power BI! 25 chapters about different chart types. 40 visuals: from default to advanced from the AppSource gallery. 400 color pages of an exceptional quality. The book is suitable for non-technical professionals as well as for experienced data analysts, it consists of 3 parts: Classic Visuals - Authors explain how to choose charts for basic types of analysis and avoid common mistakes. How to set up interactions and put visuals together on a dashboard. Trusted Advanced Visuals - Different options and data requirements for waterfall and bullet charts, Gantt, tornado, funnel, Sankey, etc. Risky Advanced Visuals - ‘Eye-catching’ charts that may confuse the average user. We explain use cases and offer simpler alternatives. Book features: Beautiful examples, specific use cases for charts. Step-by-step guides on how to set it up in the app. Data preparation tips and tricks. Quizzes to consolidate the learning material. “I want to inspire people to use Power BI for more than just reporting. I want them to create brilliant dashboards and tell interactive data stories!” - Alex Kolokolov The book is now available on Amazon. Other Support for Power BI language settings when a paginated report is viewed on the Power BI service When a localized paginated report is published to the Power BI service, the viewer of the report will now see the report in the preferred language that they have selected in the Power BI/ Fabric Settings page. Previously, the rendering of the report was determined by the server settings. Learn more about viewing localized paginated reports on the Power BI service. Platform Introducing OneLake catalog OneLake catalog is the next evolution of the OneLake data hub. Providing a unified experience, where data engineers, data scientists, analysts, and decision-makers can browse, manage, and govern all their data from a single, intuitive location. The OneLake catalog now includes various item types in Fabric, such as dashboards and reports (available by the end of November), dataflows, pipelines, and more. Streamlined for collaboration OneLake catalog offers filtering capabilities to help users find specific items efficiently. Business users can uncover reports and dashboards to answer their questions, while analysts can explore data items and processes for deeper analysis. In-place data management OneLake catalog allows you to view and manage any item directly within the catalog itself, simplifying navigation and enhancing efficiency. This contextual management ensures that you can handle your data ecosystem more effectively. In-depth item metadata Clicking on any item in the OneLake catalog reveals relevant metadata, including descriptions, tags, endorsement and sensitivity labels. The catalog also provides a granular view of your data items schemas and objects (e.g. warehouse tables and views), enabling better insight and control. Unified management and governance OneLake catalog combines crucial functionalities such as cross-workspace item lineage, access permissions, and real-time activity monitoring within a single interface. This unified approach makes governance and management tasks more accessible for every user. Explore the catalog today Explore OneLake catalog to experience the future of data management in Fabric. OneLake catalog is available in more than 40 scenarios where users connect to data within Fabric. It is also accessible in services and applications outside of the Fabric service application, such as Power BI Desktop and Azure Ibiza, with plans to expand to Excel soon. To learn more about this update, find detailed information that covers all the features and benefits of the catalog in depth. Tenant switcher control The tenant switcher is now available in the Fabric portal. Users with access to more than one Fabric tenant can easily switch between tenants directly from the account manager in the top right corner of the Fabric portal. This is in addition to the existing From External Orgs tab that can be found in the home page of the Power BI experience. Automate GitHub integration with Microsoft Fabric REST APIs Introducing the new REST APIs for Git integration with GitHub! These APIs enable you to automate Git integration tasks, such as connecting to GitHub, retrieving connection details, committing changes to your connected GitHub repository, updating from the repository, and more. For more information about the APIs and find available code samples. Switch branches from the source control pane Switching connected Git branches is now available directly through the source control pane. All branching actions can now be accessed in one place within the branches tab. This allows you to: Branch out to new workspace to create a new workspace with a new connected branch. Checkout branch to create a new branch while keeping the current workspace state, useful for resolving conflicts. Switch branch to replace the current workspace content with another branch, new or existing. Learn more details for these actions. Announcing general availability of the Fabric Workload Development Kit The Microsoft Fabric Workload Development Kit is now generally available. This feature allows Fabric to extend additional workloads and offers a robust developer toolkit for designing, developing, and interoperating with Microsoft Fabric using frontend SDKs (Software Development Kits) and backend RESTful APIs (Application Programming Interfaces). See feature blog to learn more. This release includes new features and enables users to start using Partner Workloads, which will be available in the workload hub in the coming weeks. Go to the Workload hub OneLake External data sharing is now generally available The external data sharing feature announced earlier this year is now generally available. External data sharing enables the sharing of OneLake tables and folders across tenant boundaries. In the current release, each share may include a single folder or a table from a Lakehouse. In the coming releases, support will be added for multiple folders and tables in a single share as well as sharing from Warehouses and Eventhouses. For more information check out the documentation. Mirroring Introducing Open Mirroring Introducing Open Mirroring, our new Mirroring capability. When we created Microsoft Fabric, we designed our platform to be extensible, customizable, and open. With that in mind, Open Mirroring, now in preview, is a powerful feature that enhances Fabric’s extensibility by allowing any application or data provider to bring their data estate directly into OneLake with minimal effort. By enabling data providers and applications to write change data directly into a mirrored database within Fabric, Open Mirroring simplifies the handling of complex data changes, ensuring that all mirrored data is continuously up-to-date and ready for analysis. For those looking to expand their data processing and analytics capabilities within Microsoft Fabric, Open Mirroring brings a flexible and powerful solution to ensure your data remains in sync, accessible, and analytics-ready within OneLake. Our Fabric partners such as Striim, OCI Golden Gate, and MongoDB already have capabilities to integrate with Open Mirroring, with DataStax integration coming soon. This enables any organizations to leverage a broader ecosystem of tools, enriching their data processing and analytics within the Fabric environment. Learn more about Open Mirroring in the Introducing Open Mirroring in Microsoft Fabric blog post. Fabric Database Mirroring Public REST APIs are now generally available Announcing the general availability of Fabric Databese Mirroring Public Rest APIs. Users can now utilize Microsoft Fabric REST APIs to perform CRUDLE operations: Create a new Mirrored database in your Fabric workspace. Read existing Mirrored database to get the definition of the item. Update Mirrored database with changes to the definition. Delete existing Mirrored database to clean up your workspace. List all Mirrored database in a workspace to get all available mirrored database in your workspace. With a Mirrored database ID, you can also get additional status for the Mirrored database and its tables mirroring status. In addition, you can start and stop existing mirrored databases with public REST APIs as well. To learn more read mirrored database REST API. Mirroring for Azure SQL Database now Generally Available Mirroring for Azure SQL Database is now generally available. Mirroring is a simple, free and frictionless way to replicate a snapshot and incremental data changes from Azure SQL database to Fabric OneLake with data sync in near-real time. With the GA release, the following new features are now available: Support for Truncate Table in source database when Mirroring is active Address issues related to schema hierarchy and column mapping in Data Warehouse and Lake House experience To learn more Announcing the general availability (GA) of Fabric Mirroring for Azure SQL Database. Introducing Mirroring for Azure SQL Managed Instance (Preview) The Preview of Mirroring for Azure SQL Managed Instance is now available. Mirroring is a simple, free and frictionless way to replicate a snapshot and incremental data changes from Azure SQL Managed Instance to Fabric OneLake with data sync in near-real time. Fabric Mirroring offers a great alternative to running a project to set up an ETL process to enable insights into an operational database that is the subject of an analytics scenario. You can set up Fabric mirroring in just a few steps, choose tables to mirror and the data will start flowing. To make changes to which tables are mirrored, you just need to make a few clicks. At any point, it is easy to see the status of replication for all mirrored tables. All this setup, management and monitoring is integrated directly into Fabric UI. Before mirroring, ETL setups would require additional tooling for data replication, people expertise to set up, configure, monitor and maintain the ETL- and this is just to keep the replication going. Any changes to replication would again require queuing up and waiting for ETL experts to modify your pipelines. To learn more about this new and exciting capability of Mirroring for Azure SQL Managed Instance in Microsoft Fabric, please read more in the blog. Databases Introducing Fabric SQL database (Preview) SQL database is now available as a native solution in Microsoft Fabric (Public Preview) and is the first database offering to land in the new databases workload. This new offering is seamlessly integrated with the Fabric platform and includes unified billing through the Capacity units (CU) model. Whether you are working on small or large analytics projects, we have heard your feedback: you need database support in Microsoft Fabric. With the addition of Fabric databases, we are evolving Microsoft Fabric from an analytics platform into a data platform. Fabric now has everything you need for your GenAI apps: operational database support, analytical storage, real-time intelligence for data in motion, and top-tier visualization. This week, we also announced the preview of a new vector type and functions in Fabric SQL database and Azure SQL Database, making building AI apps much simpler. We have samples for how you can easily integrate with frameworks like LangChain, Semantic Kernel, and more. You can get started with SQL database in Fabric today. For more information, please see the Announcing Fabric SQL database Preview. Data Warehouse Cold query performance improvement Running a query with a cold cache presents several challenges. When data is not cached, it must be fetched from OneLake and transcoded from parquet file format structures into in-memory structures for query processing. This process can be time-consuming and impact overall performance. With our latest improvement, we have optimized both fetching data from the storage and the transcoding process, observing median cold query overhead reduction of 40%. Service principal support for Fabric Data Warehouse We’ve made a major enhancement in the way you can authenticate and manage your Fabric Data Warehouses: the introduction of service principal (SPN) support. This new feature empowers developers and administrators to automate processes, streamline operations, and increase security for their data workflows. Earlier we launched service principal support for various Microsoft Fabric items, including Lakehouses and Eventhouses. Now, this support extends to Fabric Data Warehouses, making it easier to connect, manage, and deploy warehouse solutions in a secure, scalable way without needing to rely on user identities. The feature provides the following benefits: Automation-friendly API Access: You can now create, update, read, and delete Warehouse items via Fabric REST APIs using service principals Seamless Integration with Client Tools: You can use tools like SQL Server Management Studio (SSMS) to connect to your Fabric Data Warehouses using service principals and run TSQL features like COPY INTO. Granular Access Control: Ability to provide granular-level access by using T-SQL commands like GRANT, administrators can assign specific permissions to service principals to control precisely which data and operations an SPN has access to. Improved DevOps and CI/CD Integration: By using service principals, developers can automate the deployment and management of data warehouse resources in their DevOps and Continuous Integration/Continuous Deployment (CI/CD) pipelines to ensure rapid and reliable delivery of data solutions. Data Engineering Notebook display chart upgrade The new and improved chart view is the latest enhancement to our notebook display. This update is designed to provide a more intuitive and powerful experience for visualizing your data by leveraging the built-in visualization tool on Fabric Notebook. Key Features: Multiple charts view: Now you can add up to 5 charts in one display() output widget, allowing you to create multiple charts based on different columns, and compare charts easily! Rich chart recommendation: Get chart suggestions when creating new charts or clicking the suggestion button, easy to get started with the rich chart template and summarized title and insightful recommendations of key-value pairs. Advanced Chart Editing: You can add, rename, delete charts, and configure chart options. A lot of new configurations are provided in this upgrade, like chart title and subtitles, legend, theme, label etc. All your configurations are saved immediately. Global Configuration: Easily filter and apply custom ranges to your data. These settings will be applied to both tables and charts. Interactive Toolbar: Hover over a chart to access a toolbar for exploring the chart, like zoom in, zoom out, select to zoom, reset, panning, etc. Toolbar settings won't be saved, allowing for temporary adjustments. Benefits of the Enhanced Chart View: Improved Data Visualization: The new chart view offers a more dynamic and interactive way to visualize your data, making it easier to identify trends and insights. User-Friendly Interface: The enhancements provide a seamless experience, allowing you to switch between table and chart views effortlessly. Customization Options: With the ability to configure chart options and apply global settings, you can tailor the visualizations to meet your specific needs. We’ll gradually add more advanced chart types based on the new UX framework, stay tuned! Getting Started: To access the Enhanced Chart View, just open your Fabric notebook and run the display(df) statement. If you're seeing the legacy UX, use the switch to go to the new UX. Fabric API for GraphQL is now generally available with exciting new features The Microsoft Fabric API for GraphQL is now generally available, marking a significant milestone in providing powerful, flexible, and efficient data access APIs in Fabric. In addition to important features made available last month (Service Principals support and code generation from the API editor), this release introduces several new capabilities aimed at enhancing your experience and expanding the possibilities of what can be achieved with your GraphQL API in Fabric, making it easier to harness the power of Fabric data in your business applications. New data sources: Azure SQL and Fabric SQL DB (Preview) integration for seamless data access. Access data sources with connections and saved credentials: Enhanced security and simplified access management. Logging and Monitoring Dashboard: Visual insights into API activity and detailed logging for better performance monitoring and troubleshooting. CI/CD Support: Git Integration and Deployment Pipelines for consistent and automated deployments. You can find more information about these exciting new features in our GA announcement blog. Esri’s ArcGIS GeoAnalytics integration with Fabric Spark (Preview) Esri is recognized as the global market leader in geographic information system (GIS) technology, location intelligence, and mapping, primarily through its flagship software, ArcGIS. Esri empowers businesses, governments, and communities to tackle the world's most pressing challenges through spatial analysis and location insight. Microsoft and Esri have collaborated to integrate spatial analytics in Fabric, with a preview set to launch soon. Our collaboration with Esri will introduce cutting-edge visual spatial analytics right within Microsoft Fabric Spark notebooks and Spark job definitions (across both Data Engineering and Data Science experiences). With its integrated product experience, it empowers Spark developers or data scientists to natively use ArcGIS capabilities to run GeoAnalytics functions and tools within Fabric Spark for transformation, enrichment, and pattern / trend analysis of data across different use cases without any need for separate installation and configuration. Here is an example to transform the data with ArcGIS spatial function to uncover the pattern of interest, for instance summarizing the total number of policies of insured properties by hexagonal bins. Here is another example to understand the impact of natural hazards or current events on insured properties by bringing a dataset with probabilities of hurricane force winds and spatially joining it with insured properties. Spatial join links insured properties with wind speed probabilities, and with that for each property we would know the likelihood of hurricane force winds and can run predictive models to assess potential insurance claims. To learn more about ArcGIS GeoAnalytics integration within Microsoft Fabric Spark, please refer to the documentation. Jar libraries are now supported in Fabric Environments Java Archive (JAR) files are a popular packaging format used in the Java ecosystem. They allow developers to bundle multiple files—such as Java class files, metadata, and resources—into a single, compressed archive for distribution. JAR files simplify the distribution and execution of Java applications and libraries by consolidating everything into one file, which can be easily shared and managed. Previously, integrating JAR files into Fabric requires inline commands within notebooks. This approach, while functional, posed a challenge of reproducibility. And now, you can upload your JAR files as the custom libraries to the Environment. These custom libraries will be effective in the Notebooks and Spark jobs once attached to the Environment. Embracing JAR files within Fabric Environments can streamline your development and deployment processes, enhancing the overall efficiency and scalability of your applications. Support of spaces and special characters in Delta table names Support of spaces and special characters in Delta table names in Microsoft Fabric is now available! This is a highly desired enhancement requested by Fabric customers. Now, you can name Delta tables using spaces, special characters and the encoding of your natural language in all Fabric experiences. Everything will work: Spark, Lakehouse, Notebooks, Warehouse, Power BI, Shortcuts creation, Metadata discovery, etc. Some restrictions apply, learn all about it in the documentation. The feature will be available over the next weeks across all Fabric regions worldwide. Data Engineering demos https://youtu.be/eyjVj-k8m1M?si=CPy-GGf3uHHGks4a&t=2175 Data Science Introducing low code AutoML AutoML, or Automated Machine Learning, is a process that automates the time-consuming and complex tasks of developing machine learning models. It simplifies the workflow by handling data preprocessing, feature engineering, model selection, and hyperparameter tuning, allowing users to focus on interpreting results and making decisions. We are introducing the new low code AutoML user experience in Fabric, designed to empower analysts and data scientists to quickly prototype and build machine learning models with ease. This innovative interface supports a variety of tasks, including regression, forecasting, classification, and multi-class classification. Getting started with the AutoML user experience is incredibly simple. Users can begin with an existing experiment, model, or notebook. All it takes is selecting the relevant files or tables from your lakehouse and specifying the desired ML task. For those who want more control, there are optional configurations available. You can choose your parallelization mode, deciding whether to train one Spark-based model at a time or to parallelize trials with Pandas by distributing them across all nodes on your Spark cluster. Additionally, the auto-features setting enables us to generate useful features for model training. One of the key features of the AutoML experience is its integration with MLflow. All generated models are tracked using MLflow and the existing Experiment items. This allows users to monitor all the details, such as metrics, parameters, model types, and model files, making it easy to compare different models generated from the AutoML trial. Learn more about Automated Machine Learning in Fabric. Data Science demo https://youtu.be/eyjVj-k8m1M?si=KqDXTvHWwHw9LqML&t=2602 Real-Time Intelligence Real-Time Intelligence is now generally available (GA)! Announced at Build 2024, Real-Time Intelligence includes a wide range of capabilities across ingestion, processing, analysis, transformation, visualization and taking action. All of this is supported by the Real-Time hub, the central place to discover and manage streaming data and start all related tasks. This month includes a wide range of improvements, read on for more information on each capability and stay tuned for a series of blogs describing the features in more detail. Please submit any feedback on our features at RTI ideas. Ingest & Process Announcing the general availability of Real-Time Hub Fabric Real-Time Hub is now generally available! The one enterprise-wide catalog that enables users to discover, connect to, explore and act upon streaming data & events from anywhere. Seamless integration with all Real-Time Intelligence services like Fabric Eventstreams, Eventhouse and Activator greatly accelerates time to insights. Fabric Real-time Hub was originally released to Public Preview at //Build 2024 and has since become one of the most broadly adopted features within Real-time Intelligence suite. At the same time, our customers continue to give us valuable feedback to make Real-Time Hub even better. And we are listening! Here are some of the recent improvements: What’s new? Simplify Azure Event Hubs source connection: We have simplified the experience when connecting to an existing Azure Event Hub. For users with the right permission to access the available Azure Event Hubs, a single click is all it takes for Fabric to automatically establish the connection to the source New Sources: Azure Service Bus, Apache Kafka, CDC from SQL Server on VM DB and CDC from Azure SQL Managed Instance are added to the “Connect data source” options. Rich Sample Scenarios: For users who are new to Fabric Real-Time Intelligence, we provide three streaming data samples for you to get started. Streams and KQL tables with read (or higher) permission: Users can discover streams and KQL tables that they have read (or higher) access to within Real-Time Hub, which allows them to discover more data streams shared with them. Generate Real-time Dashboards (preview): Users can now quickly and automatically create real-time dashboards by selecting ‘Create Real-Time dashboards’ on KQL Tables. This CoPilot assisted feature can take users input to generate the most common real-time dashboards within seconds. Fabric Events (preview): Customers will soon be able to build event-driven applications, trigger Notebooks and workflows or send emails and Teams IM when OneLake files/tables are created, deleted or renamed (OneLake events) and Jobs are started or completed (Job Events). Explore Data action on KQL tables (coming soon): customers will soon be able to explore the data of their KQL tables with no-code experience, to allow them to interact with the data without leaving the context that they are in Azure Data Explorer (ADX) Database Shortcut (coming soon): customers will soon be able to create database shortcut for their ADX clusters. This will allow customers to manage their ADX clusters more efficiently directly from Fabric. Real-Time Hub serves as the starting point for your Real-Time Intelligence journey. Please feel free to try it out and give us feedback through Ask Fabric Real-time Hub [email protected]. Announcing the general availability of Enhanced Eventstream Enhanced Eventstream is now generally available! This offers new features that improve your experience in building stream flows within Fabric Real-Time Intelligence. The enhancements include Edit and Live View modes, Default and Derived Streams, and Smart Routing, transforming how data engineers handle real-time data streams with greater flexibility and efficiency. Edit Mode and Live View: Eventstream now offers two separate modes, Edit Mode and Live View, to give you flexibility and control over your data streams. Edit mode lets you design and modify your data streaming flow without interrupting the active data streams. Live View gives real-time insight into the data flow, allowing you to monitor the ingestion, processing, and distribution of data streams within the Fabric. You can switch between the two modes using the button in the top-right corner. To learn more, visit: Edit and publish Microsoft Fabric eventstreams - Microsoft Fabric | Microsoft Learn Default and Derived Streams: Data stream is a dynamic and continuous flow of data, allowing you to set up real-time alerts, and feed into different types of data stores. A data stream is a continuous flow of dynamic data that allows for real-time alerts and diverse data storage options. Default stream is created automatically when a streaming source is added to Eventstream, capturing raw event data directly from the source and preparing it for transformation or analysis. Derived stream is a specialized stream that users can set up as a destination within Eventstream. After performing operations such as filtering and aggregating, the derived stream is ready for further analysis or consumption by other organization members through the Real-Time Hub. To learn more, visit: Create default and derived Fabric eventstreams - Microsoft Fabric | Microsoft Learn Content-based Routing: Customers are now able to design stream operations directly within Eventstream’s Edit mode, transforming and routing of real-time data streams. It lets you create stream processing logic and direct data streams based on their content right in the Eventstream editor. To learn more, visit: Route events based on content in Fabric event streams - Microsoft Fabric | Microsoft Learn Announcing the general availability of connector sources in Eventstream Connector sources in enhanced Eventstream are now generally available! This feature enables seamless connection of external real-time data streams to Fabric, allowing for an optimal out-of-the-box experience and more choices for real-time insights from a variety of sources. It supports well-known cloud services like Google Cloud and Amazon Kinesis, as well as database change data capture (CDC) streams through our new messaging connectors. These connectors utilize Kafka connect and Camel Kafka connectors for a flexible approach to data integration, ensuring broad connectivity across leading platforms. Additionally, Debezium is integrated for precise CDC stream capture. Below is the list of generally available connector sources: Confluent Cloud Kafka Amazon Kinesis Data Streams Google Cloud Pub/Sub Amazon MSK Kafka Azure SQL Database Change Data Capture (CDC) Azure SQL Managed Instance (CDC) SQL Server on VM DB (CDC) PostgreSQL DB (CDC) Azure Cosmos DB (CDC) MySQL DB (CDC) To learn more about sources regarding the details of the configuration, visit: Add and manage eventstream sources - Microsoft Fabric | Microsoft Learn. To ask for new connector sources, please contact [email protected]. Introducing Azure Service Bus Connector for Eventstream (Preview) Many enterprise customers rely on Azure Service Bus as a key message broker for managing queues and publishing subscribe topics. They want to integrate their messaging infrastructure with Fabric to enable seamless data streaming, high-performance processing, and real-time dashboards. Now, we're introducing the Azure Service Bus Connector for Eventstream! This connector allows you to stream messages from Azure Service Bus topics and queues directly into Eventstream. Once the messages are in Eventstream, you can process them in real time and route them to multiple destinations within Fabric. This new connector simplifies the integration process and empowers real-time, scalable data streaming from your Azure messaging sources. Below, you will find how to add an Azure Service Bus source in Eventstream's edit mode. New Fabric events (Preview) New Fabric event categories, namely OneLake events and job events, will be available at the end of November, in Preview in Real-Time Hub. These events can be used for real-time alerting and data processing through Reflex triggers and sending them to other destinations via Eventstreams. OneLake events allow you to get alerted when changes occur in your OneLake. For example, when new files or folders are created or deleted. Users can use these events to automate workflows such as triggering a Data pipeline via the Reflex. Job events provide detailed information about various job activities and statuses within Fabric. For example, status when a Data pipeline or Notebook is run. These events can include updates on job initiation, completion, failures, and any intermediary states or changes. Learn more about Fabric events. You can incorporate the two new Fabric events into Eventstream as a source if you want to direct these events to various destinations, including Eventhouse, Lakehouse, or your custom application via Eventstream's custom destination endpoint. To learn more about how to add and configure these sources, please visit Add and manage eventstream sources - Microsoft Fabric | Microsoft Learn. Eventstream Data Preview on database CDC sources The enhanced Eventstream now includes Data Preview for database CDC sources. This feature allows you to view a snapshot of your data from the source in both Edit mode and Live View mode. In Eventstream's Edit mode, the data preview captures a snapshot from the CDC source you configured, enabling you to infer the schema for configuring subsequent operators or destinations without having to publish it first and then return to Edit mode. You can access the data preview in the 'Test result' tab at the bottom pane by selecting the source node on the canvas in Edit mode. Similarly in Live View mode, the data preview provides you with a snapshot from your sources so that you can understand what the data looks like inside your sources. The supported connector sources are: Azure SQL Database Change Data Capture (CDC) Azure SQL Managed Instance (CDC) SQL Server on VM DB (CDC) PostgreSQL DB (CDC) Azure Cosmos DB (CDC) MySQL DB (CDC) Monitoring experience on connector sources with Runtime Logs and Data Insights in Eventstream Eventstream now offers Runtime Logs and Data Insights for the connector sources in Live View mode. With Runtime Logs, you can examine detailed logs generated by the connector engines for the specific connector, which assist in identifying failure causes or warnings. You can access this feature in the bottom pane of Eventstream by selecting the relevant connector source node on the canvas in Live View mode. Data Insights provides metrics that are from the connector engine, aiding users in monitoring the connector sources' status and performance. The Source Incoming/Outgoing Events display the number of records polled or produced by the task assigned to the specified source connector in the worker. To learn more, please visit: Monitoring status and performance of an Eventstream item Processing and routing events to Activator with Eventstream (Preview) Fabric Activator (previously Data Activator) is a no-code experience for automatically taking action when patterns or conditions are detected in data. You use the activator item (previously reflex item) to manage the rules and actions. Fabric event streams under Real-time Intelligence, represented by 'Eventstream' as a Fabric item, aims to establish a centralized place on the Fabric platform for seamlessly capturing real-time events from diverse sources, transforming it, and routing it to various destinations. Now, Eventstream supports processing and transforming events with business requirements before routing the events to the destination: Activator. When these transformed events reach Activator, you can establish rules or conditions for your alerts to monitor the events. To add this destination, simply choose Activator from the Destination menu in the ribbon while in Edit mode. To learn more about sources regarding the details of the configuration, visit: Add an Activator destination to an eventstream Introducing Eventstream’s CI/CD support Collaborating on data streaming solutions can be challenging, especially when multiple developers work in the same environment. Conflicts, versioning issues, and deployment inefficiencies often arise. The integration of Fabric CI/CD tools for Eventstream in Microsoft Fabric has been developed to address these challenges and improve team collaboration. Fabric offers complete CI/CD experience with a variety of tools, including Git integration and Deployment pipelines. By integrating Eventstream with these tools, developers can efficiently build and maintain Eventstream items from end-to-end in a web-based environment, while ensuring source control and smooth versioning across projects. Key features include: Git Integration for Eventstream: Developers can collaborate freely using versioning and branching with their favorite git tools e.g., GitHub and Azure DevOps, preventing conflicts and enabling seamless teamwork. Deployment Pipeline for Eventstream: Accelerate and standardize Eventstream deployments to various stages, such as testing and production, with minimal manual effort in the Fabric UI. Below, you will find how to commit an Eventstream change to a git repository: With these powerful CI/CD capabilities, you can streamline your development workflow for Eventstream, isolate your development environments, and collaborate effortlessly with your team. Experience faster, more reliable development with Eventstream's CI/CD support. Automate Eventstream Item Operations with Eventstream REST APIs Introducing Eventstream REST APIs, these APIs allow you to automate and manage Eventstream items programmatically, simplifying CI/CD workflows and making it easier to integrate Eventstream with external applications. With Eventstream REST APIs, you can: Automate Eventstream deployments within your CI/CD pipeline. Perform full CRUD (Create, Read, Update, Delete) operations on Eventstream items programmatically. Seamlessly integrate Fabric Eventstream into external applications. Scale your streaming solutions quickly and efficiently. By leveraging these REST APIs, you can create fully automated workflows that enhance the quality, reliability, and productivity of your Eventstream items. Stream Data to Eventstream Securely using Entra ID Authentication (Coming Soon) Introducing Entra ID authentication for Eventstream’s Custom Endpoint! This feature enhances security by allowing users to stream data to Eventstream without relying on SAS keys or connection strings, reducing the risk of unauthorized access. Entra ID authentication ties user permissions directly to the Fabric workspace access, ensuring that only authorized users can access the workspace and stream data to Eventstream. Check out the screenshot below to see how this feature appears in Eventstream’s Custom Endpoint! Additionally, Tenant Admins now have the option to disable Eventstream’s key-based authentication in tenant settings, further securing the eventstream by enforcing the use of Entra ID authentication only. Analyze & Transform Eventhouse monitoring (Preview) Fabric workspace monitoring is the centralized logging solution of Fabric. Workspace monitoring is designed to provide a seamless and consistent monitoring experience with end-to-end visibility across all Fabric items. Workspace monitoring is based on the Real-time Intelligence Eventhouse KQL database. Once Fabric Monitoring is enabled, a KQL database is created to store all the workspace items event logs. KQL databases are ideal for time series logs and metrics monitoring solutions. For each one of the supported items, one or more events or metrics tables are created. Here you can see the tables supporting Eventhouse query, command and ingestion monitoring, and semantic model query logs. The Eventhouse Monitoring offers 5 events and metrics tables: EventhouseQueryLogs – logs all Eventhouse KQL queries. EventhouseCommandLogs- logs all Eventhouse commands. EventhouseDataOperations – logs all successful data operations including Batch ingestions, Streaming seal operations (operations that store streaming data to database extents), Materialized views updates, and Update policy table updates. EventhouseIngestionResultLogs – logs all successful and failed ingestions. EventhouseMetrics- set of metrics that provide in depth monitoring of ingestions, materialized views, and continuous exports. Users can explore and directly query workspace monitoring tables using KQL or SQL, with example queries available in the documentation. Here is an example of monitoring queries stored in an Eventhouse KQL QuerySet: The workspace monitoring solution centrally monitors all the Power BI reports, semantic models and Eventhouses items created in the workspace. In some cases, the Power BI semantic models read data from KQL database sources, however semantic models queries are logged in workspace monitoring solution regardless of the Power BI report data source. Realtime Dashboards can be created on top of the workspace monitoring KQL database, providing an easy graphical monitoring user experience. Real time dashboard templates can be imported to provide an out of the box monitoring experience. In this example real time dashboard, you can see Semantic Model CPU usage monitoring, with a direct link to the underling KQL queries being run by the Power BI report. Users can troubleshoot issues by correlating events across semantic models and their underlying databases. Users can troubleshoot spikes and activities and investigate who is consuming these resources. The user can easily zoom into the exact spike time and determine which user, or application generated the usage peak. They can then drill down to the specific query log record. In summary, workspace Monitoring offers a centralized monitoring solution, allowing users to efficiently monitor and troubleshoot their workspace items. Specifically, for Eventhouse, query, command and ingestions events and metrics logging enable advanced Eventhouse monitoring capabilities. Learn more about Eventhouse Monitoring Eventhouse Query Acceleration for Shortcuts (Preview) Shortcuts are embedded references within OneLake that point to other files' store locations without moving the original data. Previously, you could create a shortcut to OneLake delta tables using Eventhouse and query the data, but performance lagged direct ingestion in Eventhouse, as shortcut queries lacked the powerful indexing and caching capabilities of Eventhouse. Query acceleration indexes and caches data landing in OneLake on the fly, allowing customers to run performant queries on large volumes of data. Customers can use this capability to analyze real-time streams coming directly into Eventhouse and combine it with data landing in OneLake either coming from mirrored databases, Warehouses, Lakehouses or Spark. Customers can expect significant improvements by enabling this capability, in some cases up to 50x and beyond. How to enable Query Acceleration? You will now see an option to enable Acceleration while creating a new shortcut from Eventhouse. Learn more about Real-Time Intelligence Synapse Data Explorer to Eventhouse migration (Preview) Synapse Data Explorer (SDX), part of Azure Synapse Analytics, is an enterprise analytics service that enables you to explore, analyze, and visualize large volumes of data using the familiar Kusto Query Language (KQL). SDX has been in public preview since 2019. The next generation of SDX offering is evolving to become Eventhouse, part of Fabric Real-Time Intelligence. Eventhouse offers the same powerful features and capabilities as SDX, but with enhanced scalability, performance, and security. For customers looking to migrate to Eventhouse from SDX, we are happy to announce a seamless migration capability. Customers can use the migration API to seamlessly move their SDX cluster to Eventhouse with minimal disruption, learn more. New explorer for database objects in KQL Queryset Effortlessly browse through the database your current Queryset is connected to, viewing tables, functions, materialized views, and more. Double-click any object to instantly copy its name to the query editor, making query writing easier than ever. When opening the data source switcher, you can easily refresh the data or disconnect it from the KQL Queryset if it’s no longer needed: In addition, you can now apply actions directly from the explorer. Simply click the ellipsis next to any object to access a menu with options tailored to your selection. Entity Diagram view in KQL Database (Coming Soon) A new feature in the KQL Database page enables you to visually explore relationships between database entities—such as tables, functions, materialized views, update policies, external tables, and continuous exports—through an interactive graph visualization. This helps you efficiently manage your database and gain a clearer understanding of how these entities interact. Sample Scenarios Proactively manage dependencies With this visual representation, you can easily manage dependencies between entities such as tables and functions. For instance, when renaming a table or modifying its schema, you can immediately see which functions are using that table as part of their KQL body. This proactive approach helps prevent unintended consequences and ensures smoother updates to your database structure. Track data sources behind Materialized Views The new feature also lets you trace the relationships between materialized views and their underlying source tables. This makes it simple to identify original data sources, allowing you to track and troubleshoot data flow more effectively. Interact with elements and act You can click on any element in the graph to see its related items, while the rest of the graph is greyed out, making it easier to focus on specific relationships. For tables and external tables, additional options become available, such as querying the table, creating a Power BI report based on the table, and more. Track record ingestion Additionally, you can easily track how many records have been ingested into each table and materialized view. This clear view of data flows helps you stay on top of ingestion size and volume, ensuring your database processes data correctly. Summary This visual enhancement simplifies database management and helps you optimize your data structures, making it easier to track dependencies and take actions quickly. Visualize & Act Easily share Real-Time Dashboards with others Microsoft Fabric’s new real-time dashboard permissions feature brings granular control to how users interact with real-time analytics. With the introduction of separate permissions for dashboards and underlying data, administrators now have the flexibility to allow users to view dashboards without giving access to the raw data. This separation is key for organizations that need to ensure data security while providing actionable insights to a broader audience. Fabric permissions focus on how users interact with the dashboard itself, determining who can view, edit, or share the dashboard. Meanwhile, data source permissions ensure that only authorized users can access the raw data that powers these visualizations. This division improves the overall security posture by ensuring users have only the necessary level of access. An added benefit of this feature is the option to choose between pass-through and editor’s identity for handling data access. Pass-through allows users to access data using their own credentials, while editor’s identity uses the dashboard editor’s permissions. This ensures that the system is adaptable to different collaboration scenarios and aligns with organizational needs. Overall, these enhancements to real-time dashboard permissions in Microsoft Fabric promote secure, efficient, and tailored access to data and dashboards. This flexibility empowers teams to collaborate more effectively while maintaining strict control over data access, making it a valuable addition for organizations leveraging real-time intelligence. Learn more about Real-Time Dashboards permissions (Preview). Announcing the general availability of Real-Time Dashboards Real -Time Dashboards is now generally available in Microsoft Fabric, bringing fast, actionable insights to your fingertips. Real-Time Dashboards make it easier than ever for organizations to track and act on key metrics in real-time, empowering faster decisions and deeper insights without the need for complex coding. Unlocking the Power of Live Insights With Real-Time Dashboards, you can monitor critical data events as they happen. Users can now set auto-refresh rates as low as 10 seconds or even continuous updates for real-time data streams, ensuring you stay up to date on every key metric. Flexible, Secure Data Sharing A new feature announced as part of the general availability of Real-Time Dashboards is the separation of permissions for dashboards and underlying data. Administrators can now grant dashboard access without exposing raw data, allowing teams to make data-driven decisions while maintaining strict data security. This separation of permissions is particularly valuable for organizations that need to ensure compliance and protect sensitive information while still sharing key insights broadly. Effortless, No-Code Data Exploration Our no-code ‘Explore Data’ functionality empowers users of all technical backgrounds to go beyond the dashboard’s insights. Now, anyone can dive deeper into metrics, explore underlying data, and analyze trends—all without needing to know KQL or write queries. With "Explore Data," you can filter, drill down, and interact with data using a user-friendly UI, gaining a clear understanding of what’s driving changes or fluctuations. Start Gaining Real-Time Insights Today Real-Time Dashboards in Microsoft Fabric are designed for users who want to transform data into action, faster. By delivering continuous updates, enhancing security, and offering intuitive data exploration, Real-Time Dashboards provide everything you need to harness the power of live data. Try it today and start unlocking the full potential of your data! Announcing the general availability of Activator Real-Time Intelligence Activator is now generally available! We would like to extend our gratitude for your invaluable partnership and feedback throughout Data Activator’s development as we help your organizations go from insights to action. With GA, you’ll be able to: Stay on top of your critical metrics by monitoring your business objects. You can track and analyze key business objects such as individual packages, households, refrigerators, and more in real-time, ensuring you have the insight needed to make informed decisions. Whether it’s understanding how individual instances of your business objects impact sales figures, inventory levels, or customer interactions, our monitoring system provides detailed insights, helping you stay proactive and responsive to changes in your business environment at a fine-tuned level of granularity. Unlock the full potential of creating business rules on your data with advanced data filtering and monitoring capabilities. This update offers a wide array of options for filtering, summarizing, and scoping your data, allowing you to tailor your analysis to your specific needs. You can set up complex conditions to track when data values change, exceed certain thresholds, or when no new data has arrived within a specified timeframe. Ensure your communications are perfect before hitting send by seeing a preview of your Email and Teams messages. This will allow you to see a preview of your message exactly as it will appear to the recipient. Review your content, check formatting, and make any necessary adjustments to ensure clarity. With this feature, you can confidently have Data Activator send messages on your behalf knowing they look just the way you intended. Set up rules that trigger automatically with every new event that comes in on your stream of data. Whether you need to send notifications or initiate workflows, this feature ensures that your processes are always up-to-date and responsive. We renamed our feature to help create clarity about what it is and what it does and simplify the way you discover Data Activator and create actionable rules. If you are used to seeing Reflex, please note that it is now called Activator. The items you create to set up rules and actions are, therefore, activators. You can find Activator tile in Fabric Real-Time Intelligence section. We hope that, however small, these changes simplify the process of getting started with Data Activator. Activator billing is now enabled, and it is based on the following four meters: Compute resources The number of rules running, and the duration of time these rules have been up. Each active rule comes with a uniform ‘uptime’ cost. The number of events per second ingested. Evaluating the rules and, when the conditions are met, triggering the defined action. Storage The Fabric storage charges are based on the storage consumed for events and activation retention. The retention policy by default is set to 30 days. Should your account run out of capacity, we will show you an in-product banner and send email notification. You can always track and review your Activator capacity usage and, if needed, update it to fit your business needs. The formal billing for Activator usage will begin with our GA announcement on Nov 18. To learn more, find our documentation Activator. As always, we’d love to hear your feedback and look forward to hearing from you. Stay tuned for the detailed RTI Billing Blog post for more details. Real-Time Intelligence demos https://youtu.be/eyjVj-k8m1M?si=X3w7sqJV7sSsAtR8&t=3416 Data Factory Table and partition refreshes added to semantic model refresh One of the most popular features that we built in Fabric Data Factory came from our customer patterns that we observed being used in ADF and from our community. That is the semantic model refresh activity. After first releasing this pipeline activity, we heard your request to improve your ELT pipeline processing by including an option to refresh specific tables and partitions in your semantic models. We are super pleased to announce that we’ve now enabled this feature making the pipeline activity the most effective way to refresh your Fabric semantic models. Learn more about Semantic model refresh activity Import and export your Fabric Data Factory pipelines As a Data Factory pipeline developer, you will often want to export your pipeline definition to share it with other developers or to reuse it in other workspaces. We’ve now added the capability to export and import your Data Factory pipelines from your Fabric workspace. This powerful feature will enable even more collaborative capabilities and will be invaluable when you troubleshoot your pipelines with our support teams. New connectors available In the Data Factory, both data pipeline and dataflow gen 2 now natively support the Fabric SQL Database connector as source and destination. Additionally, data pipeline expands its connectivity to include the ServiceNow connector (source) and MariaDB connector (source). Worth mentioning, Iceberg format is newly introduced in data pipeline first. As the first click-stop, you can now use data pipeline to write data as iceberg format via Azure Data Lake gen2 connector. Alongside these new connectors, there’re numerous feature enhancements to existing connectors. Highlights include continued improvements to the following connectors: The Snowflake connector with added support for the China domain in data pipeline and dataflow gen2. The Dataverse connector with enrichment on the authentication type support in data pipeline. Both PostgreSQL connector and Azure PostgreSQL connector with the capability to customize query timeout in data pipeline. Simplify data ingestion with Copy Job - CI/CD upsert & overwrite Copy Job simplifies data ingestion, providing a seamless experience from any source to any destination. Whether you need batch or incremental copying, Copy Job provides the flexibility to meet your data needs while keeping things simple and intuitive. Since the Public Preview launch at FabCon Europe in late September, we've been rapidly enhancing Copy Job with powerful new features. Here’s our latest update: Copy Job now supports CI/CD capabilities in Fabric, including Git integration for source control and ALM Deployment Pipelines. Check out the details in CI/CD for copy job in Data Factory - Microsoft Fabric | Microsoft Learn. Copy Job now also offers expanded writing options: Upsert functionality for SQL DB and SQL Server, and an Overwrite option for Fabric Lakehouse—bringing added flexibility and control to data movement. Check out the details in What is Copy job (preview) in Data Factory. New capabilities in Copilot for Data Factory to efficiently build and maintain your Data pipelines The new Data pipeline capabilities in Copilot for Data Factory are now available. The new capabilities are now in preview and serve as an AI assistant to help users to effortlessly create data integration solutions with data pipelines, to easily understand complex data pipelines and to efficiently troubleshoot data pipeline error messages. Create a new data pipeline and click on the Copilot button on the home tab to get started. Easily get started to build the pipeline with three starter options and clear guidance. Copilot for Data Factory can easily understand your intent and business requests to transform them into data integration solutions. You can either easily set up your data pipeline with pre-filled prompt step by step or you can efficiently create your data pipeline with a comprehensive prompt. Copilot for Data Factory also improves Data pipeline troubleshooting error messages experience. It provides clear explanations and actionable recommendations for you to identify and resolve the Data pipeline errors easily. Copilot for Data Factory can quickly summarize your pipeline for better understanding, which is extremely useful in collaborative scenarios involving complex data pipelines. You can get the complex pipeline summary either by clicking on ‘Summarize this pipeline’ option or sending ‘Summarize this pipeline’ prompt. Then you will get a very clear explanation of the complex pipeline developed by other team members. OneLake datahub is now the OneLake catalog in Modern Get Data experience We are pleased to announce that the OneLake datahub has been rebranded as the OneLake catalog in Modern Get Data. When you use Get data inside Pipeline, Copy job, Mirroring and Dataflow Gen2, you will find the OneLake datahub has been renamed to OneLake catalog. The OneLake catalog represents the next evolution of OneLake data hub, offering a cohesive platform. In the current Modern Get Data, the OneLake catalog will keep the same functionality as previous OneLake Datahub. In the future, we will expand the OneLake catalog functionality to allow data engineers, scientists, analysts, and decision-makers to seamlessly explore, organize, and oversee their data in one comprehensive and user-friendly location. Dataflows now support CI/CD (Preview) With Dataflows Gen2, you can now leverage the benefits of GIT integration and CI/CD support. By enabling GIT integration within your workspace, you can store your dataflow definitions into git and branch to other workspaces, collaborating on the same dataflow. This improves your end-to-end experience, especially when working across for dev, test, and prod workspaces. Get started today! We look forward to learning more from your feedback to improve the experience. Data Factory demos https://youtu.be/eyjVj-k8m1M?si=6TQuAgbOyT45idII&t=5141
Microsoft Fabric January 2025 update
We’ve got a lot of exciting updates this month. To name a few, NotebookUtils session management utilities, Enhancing COPY INTO operations with Granular Permissions in Data Warehouse, Application Lifecycle Management (ALM) and Fabric REST APIs. Keep reading to hear about everything we have in store for you this month. Microsoft Fabric Community Conference 2025 After 2 consecutive sold-out events, FabCon returns bigger than ever to Las Vegas from March 31 to April 2, 2025. 215+ sessions, 4 keynotes, and 20 workshops to take you from Power BI to Fabric to AI and help you make the most of Copilot and SQL in Fabric. Plus, one-on-one time with Microsoft experts and community legends, a FREE pre-day for partners, and the famous Power Hour. Prices go up on February 11, register ASAP and use MSCUST to get $150 off. First-ever Power BI DataViz World Championships Are you ready? The first-ever Power BI Data Visualization World Championships are coming to FabCon Vegas! Participate to learn and win a chance to compete live on-stage! Stay tuned to the World Championships blog for more details. Free training and discount certification vouchers for DP-700! The Fabric Data Engineer Certification is now generally available! The best way to prepare for Exam DP-700 is to join Microsoft Fabric experts for live and on-demand sessions. Sessions start this week. Register now and save your spot! Ready to take the exam now? Head over to the community to request your discount voucher for Exam DP-700. Contents Microsoft Fabric Community Conference 2025 Free training and discount certification vouchers for DP-700 First-ever Power BI DataViz World Championships Power BI Copilot and AI Unlock suggested questions from standard prompts in Copilot Reporting Explore this data: new entry point from a visual Storytelling in PowerPoint - New reset behavior Storytelling in PowerPoint - Supporting page up & down Save to OneDrive and SharePoint: updated file picker (Preview) Enhancement to Text slicer (preview) Enhancements to Treemap visual Modeling Semantic model version history (Preview) Edit your data model in the Power BI Service – updates (Preview) Live edit of semantic models in Direct Lake mode with Power BI Desktop – updates (Preview) TMDL scripting experience (Preview) Data connectivity New Snowflake connector implementation (Preview) Visualizations Drill Down Scatter PRO by ZoomCharts: The All-in-One Scatter Visual Lollipop Chart by Powerviz Other Now in Power BI Desktop - OneLake catalog Platform Folder support in Git OneLake OneLake Catalog - Semantic model table & column description Filtering workspaces in OneLake Catalog Data Engineering Python notebook (Preview) Notebook live versioning NotebookUtils session management utilities Native Execution Engine on Runtime 1.3: simplified enablement and transition from Runtime 1.2 Legacy timestamp support in Native Execution Engine on Runtime 1.3 Notebook and Spark Job definition execution with service principal Lineage Enhancement to Spark Notebook Data Warehouse COPY INTO column count check Enhancing COPY INTO operations with Granular Permissions in Data Warehouse Introducing default schema changes in Data Warehouse Enhanced performance metrics in Query Insights Previewing estimated Query Plan available via SHOWPLAN_XML Query Hints in Fabric Data Warehouse Simplifying search & introducing Filter in Object Explorer Open from SSMS & VS Code Git Status Bar to Fabric Warehouse Artifact Tooltip support for built-in functions Stay updated with IntelliSense JSON aggregates (Preview) Spatial analytic functions SQL analytics endpoint performance improvement Databases Tenant Level Private Link (Preview) Copilot for SQL database in Fabric Region Availability Real-time Intelligence Override late arrival tolerance in Activator Create new Activator items RTI ALM & APIs GA Data Factory Mirroring Copy Job Dataflow Gen2 Conclusion Power BI Copilot and AI Unlock suggested questions from standard prompts in Copilot When launching Copilot or using the prompt guide, you can select from standard prompts. In preview mode, a new prompt ‘Answer a question about the data’ will be available coming end of January. Selecting this prompt will unlock 3 suggested questions to help you explore your data. You can continue to select the prompt by scrolling up or from the prompt guide (book icon) to generate 3 more suggested questions if none of the first set are interesting to you. Authors can also personalize these suggested questions using Q&A setup in Desktop for a particular semantic model. With this feature, suggested questions will now show up in both Copilot and Q&A visual. You can continue to select the prompt by scrolling up or from the prompt guide (book icon) to generate 3 more suggested questions if none of the first set are interesting to you. Authors can also personalize these suggested questions using Q&A setup in Desktop for a particular semantic model. With this feature, suggested questions will now show up in both Copilot and Q&A visual. Reporting Explore this data: new entry point from a visual Exploring your data is easier than ever, now that we’ve added an Explore this data option to the visual options menu. This lightweight and focused experience allows users to launch Explore and easily tweak their visual (change chart type, add new data, filter, and more!) and see the underlying data, making it easy to get the answers they need without all the distractions and extra complexity of reports. Learn more about how to use Explore here! Simply select ‘Explore this data’ in the ‘More options’ menu and start exploring the visual. Storytelling in PowerPoint - New reset behavior When integrating a report into your presentation, it is important to ensure that it remains stable and unaltered. The add-in refreshes data from Power BI without modifying the report definition. However, since Power BI reports are dynamic, sometimes you may want PowerPoint to get the latest changes done in the report in Power BI service. Previously, you had to remove and re-embed the report to achieve this. Now, with the improved ‘Reset’ command, you can choose either to reset the add-in to its original state as initially added to the presentation or to reset and update it with the current view from Power BI. Storytelling in PowerPoint - Supporting page up & down You can now use the Page Down/Up keys on your keyboard to quickly navigate between slides when using the Power BI add-in. This is especially useful when the add-in captures the entire slide, and you want to advance the slide rather than trigger a Power BI event. Save to OneDrive and SharePoint: updated file picker (Preview) Updates have been made to the Power BI file picker to simplify navigation and file-saving processes. We have considered your feedback regarding the current file picker and have made significant improvements in this update to ensure the Power BI file picker aligns more closely with the Office experiences you are familiar with. New updates include: Improved experience when opening and saving files in One Drive and SharePoint. Easy access to reports in OneDrive and SharePoint. Navigating between folders in various workspaces. Adding new folders to existing workspaces. Pinning folders and files in the file picker. We acknowledge that these updates may disrupt your workflow; therefore, they are not enabled by default. To access the updates, open the desktop app, go to Options and settings (under the File Menu) > Options > Preview features, select the checkbox to enable ‘Show the new file saving and open experience’ then select ‘OK’ to accept the setting. By early next year, these settings will be on by default and will no longer require an opt-in. We hope you enjoy these new updates, and we’d love to hear any feedback you may have via our feedback forum. When submitting feedback, be sure to include ‘OneDrive and SharePoint’ and/or ‘Updated File Picker’ in the title. For more information, please check out our documentation on the Power BI Desktop and OneDrive + SharePoint integrations. Enhancement to Text slicer (preview) Following the November 2024 Text slicer release, this update enhances functionality and user experience by allowing multiple text selections. The Text slicer is currently in preview. To enable the Text slicer, go to Options and settings > Options > Preview features > Text slicer visual to make sure it is selected, then restart Power BI. This month’s enhancement adds a new Slicer settings control with an on/off toggle allowing the slicer to Accept multiple values. All other existing formatting options for the Input text, Apply button, and Input text box from our November update remain the same in the Format pane. After creating a Text slicer visual and adding a text field from the data model, users can filter the dataset based on user input. Simply select the slicer input box, type your text, and apply the filter either by selecting the apply icon, pressing enter, or selecting outside the visual. The slicer immediately filters and displays the results, and you can repeat these steps to add more text selections. When the Accept multiple values option is enabled, additional text can be added to the slicer by repeating these steps, thereby allowing multiple selections for filtering the dataset. Keep in mind that switching the toggle on or off will clear any previous text selections. Adding filtering with multiple values brings more control to data slicing, and we encourage users to explore this new feature and provide feedback. Future enhancements are still planned as we continue to improve Power BI's visualization capabilities with the Text slicer. The addition of filtering with multiple values offers enhanced control over data slicing, and users are invited to explore this feature and provide feedback. Further improvements are planned for Power BI's visualization capabilities with the Text slicer. Share your comments and suggestions in the comments section below and stay connected with us through our dedicated Core Visuals LinkedIn blog where we announce new features, updates, and engage with our community. Learn about our new Core Visuals Vision Board, where customers can now explore, vote, and comment on the Epic Ideas that will shape the future of Power BI Core Visuals. The Power BI community can instantly see what features are already completed, what is currently in development, and the new features and enhancements that are upcoming. Enhancements to Treemap visual This month's update includes significant enhancements to the Treemap visual, with three new tiling methods that improve layout options, plus new spacing controls to enhance the visual's appearance and usability. These features offer richer control and customization, resulting in more precise and aesthetically pleasing treemaps in Power BI. Treemap visuals are powerful tools for data visualization that allow users to represent hierarchical data through nested rectangles. Each branch of the hierarchy is represented by a rectangle, which is then tiled with smaller rectangles representing sub-branches. This structure allows for quick comparison of different category proportions. To generate two-level Treemap visuals, ensure that both the Category and Details fields are enabled. This allows you to visualize the hierarchical relationships between various categories and their subcategories in a clear and organized manner. Three new Tiling methods: Squarified: This method uses a squarified treemap algorithm to prevent elongated rectangles, creating a balanced layout. It arranges rectangles so their aspect ratios are close to squares, making size comparisons potentially easier. Binary: This method continuously divides the chart area into two sections while incrementally adding new rectangles/nodes creating a balanced and visually appealing treemap. Each hierarchy level further splits the space, resulting in an organized treemap that adapts to the dataset's structure. It may produce different visual characteristics compared to squarified algorithm depending on the dataset. Alternating (Columns, Rows): The Alternating method clearly distinguishes categories by first splitting them by columns and then within each column by rows. This method effectively organizes datasets with numerous hierarchical levels. This month's update also introduces new spacing options to enhance the readability and appearance of the Treemap visual: Space between all nodes: This setting introduces gaps between adjacent nodes at all hierarchy levels, reducing clutter and improving clarity. Space between groups: By adding extra space around each node group, this option visually separates different hierarchical groups, which helps to visually distinguish categories within the hierarchy. This update to our Treemap visual has brought improvements that reflect the commitment of the Core Visuals team to delivering the tools and features most requested by our users. Your feedback helps us refine and expand the capabilities of core visuals. Test these Treemap enhancements and share your thoughts in the comments section below or visit our Core Visuals LinkedIn blog, to leave comments, and find up-to-date news, developments, and announcements. Learn about our new Core Visuals Vision Board, where you users can vote on upcoming features and see what is in the pipeline. Together, we can continue to innovate and improve the tools that help our community to visualize data with Power BI. Modeling Semantic model version history (Preview) Announcing the public preview of semantic model version history coming this month. This feature aims to empower self-service users by providing confidence to recover from critical mistakes when editing semantic models on the web. In this preview, versions will be automatically captured in an Office-like history pane for your web-edited Premium semantic models. You can easily select and restore any of these previous versions of your semantic model. Additionally, you have the option to manually save versions to the version history for your semantic model. Stay tuned as we continue to roll out updates to this experience, including future support for semantic models in Pro workspaces. We highly value your feedback, so please share your thoughts using the feedback forum. For more details on this feature, including limitations, please refer to the documentation. Edit your data model in the Power BI Service – updates (Preview) The following improvements to the data model editing in the Service preview will be introduced this month: On by default preview for Premium workspaces With the release of semantic model version history, we will start enabling the workspace-level preview feature for editing data models in the service. The users can edit data models workspace setting will be turned on by default for Premium workspaces. If you prefer, you can still disable the workspace preview for your workspace, but we recommend keeping it enabled! Power BI administrators will still have the ability to enable or disable data model editing in the service for the entire organization or specific security groups through the admin portal. Viewing mode Now, when you open your semantic models on the web, it will default to Viewing mode. This allows you to easily view the model in a safe environment, preventing any accidental edits. When you're ready to make changes, simply toggle to Editing mode to make your modifications directly on the web. For more details on the subject, reference the documentation. Please continue to submit your feedback directly in the comments of this blog post or in the feedback forum. Live edit of semantic models in Direct Lake mode with Power BI Desktop – updates (Preview) On by default preview Live editing semantic models in Direct Lake mode with Power BI Desktop is now enabled by default, allowing you to use this feature immediately without needing to turn on the preview feature. If you prefer, you can still disable this feature by turning off the live edit of Power BI semantic models in Direct Lake mode preview in Options and Settings > Options > Preview features. More details on the feature, including requirements, considerations, and limitations can be found in the documentation. We highly value your feedback on this feature and encourage you to share it through our feedback form or the Power BI Community. TMDL scripting experience (Preview) TMDL view is a new view in Power BI Desktop that lets you script, modify, and apply changes to the semantic model being edited in Desktop with a modern code editor using Tabular Model Definition Language (TMDL), improving development efficiency, and providing complete visibility over the semantic model metadata. TMDL view offers an alternative experience to semantic modeling using code instead of a graphical user interface like Model view. Enhance development efficiency with a rich code editor that includes search-and-replace, keyboard shortcuts, multi-line edits, and more. Increase Reusability by easily script, share and reuse TMDL scripts among semantic model developers. For example, use a centralized SharePoint site to easily share reusable semantic model objects such as calendar tables or time intelligence calculation groups. Get more control and transparency, showing all semantic model objects and properties, and allowing changes to items not available in Desktop GUI, such as IsAvailableInMDX or DetailRowsDefinition. Script any semantic model object such as table, measure, column or perspective by selecting the objects from Data pane and dragging them into the code editor: TMDL view will script the selected objects as a TMDL script and just like TMDL in VS Code you get an enriched code experience with features such as semantic highlighting, error diagnostics and autocomplete. You may change any valid property or object within the semantic model. For instance, the example below demonstrates how to modify the displayFolder property and detail rows definition of multiple measures: When ready you can hit the Apply button to execute the TMDL script against the semantic model to get your changes applied: When successful, an instant notification will be displayed, and your modeling change will be applied to the semantic model. In the event of a failure, your modeling changes will not be applied to the semantic model, and you can view more information about the error by selecting on show details, which expands the Output pane with the error details. Get started today by turning on this public preview feature, go to File > Options and settings > Options > Preview features and check the box next to TMDL View. To learn more about TMDL View refer to our documentation. Data connectivity New Snowflake connector implementation (Preview) We continue to enhance the integration with Snowflake. This month, we are introducing a new implementation for Snowflake connector, currently available in preview. To access this feature, in Power BI Desktop, navigate to Options and settings (under the File Menu) > Options > Preview features, select the checkbox to enable the ‘Use new Snowflake connector implementation’ option. Once the option is on, all the newly created connections will automatically use the new connector implementation. Your existing connections remain unchanged. You can also test the new feature by editing the queries. Learn more about the Snowflake connector from this documentation article. If you're using On-prem Data Gateway to refresh your semantic model, make sure you have the latest version to use this feature. We highly value your feedback on this feature and encourage you to share the feedback with us. Visualizations Drill Down Scatter PRO by ZoomCharts: The All-in-One Scatter Visual The latest ZoomCharts visual, Drill Down Scatter PRO, is now available on AppSource! Just like all ZoomCharts visuals, Scatter PRO combines powerful data visualization features with an intuitive and user-friendly user experience. It is designed for fully interactive Power BI reports that deliver quick insights and foster a decision-centric culture. Scatter PRO makes data exploration seamless and enjoyable with user interactions like panning, zoom-in, and rectangular or lasso selection. You can also create a multi-level hierarchy, which will allow users to drill down by simply selecting on a data point marker. You can learn more in our blog post, but here are the main features of Scatter PRO: Drill Down: Create a multi-level category hierarchy and drill down with just a select. Customization: Configure marker colors, shapes, outlines, labels, threshold lines/areas, X & Y axes, and more. Data-Driven Formatting: Apply marker colors, shapes, and even images directly from data. Area Shading: Highlight areas that need attention with up to 8 shapes at custom coordinates. Dynamic Regression Line: Show a linear or polynomial regression line. It will automatically recalculate upon any changes in the chart. Get on AppSource Lollipop Chart by Powerviz The Powerviz Lollipop chart is a variation of bar chart that uses lines and dots to represent data points. It is perfect for highlighting specific trends to help stakeholders make informed decisions. Key Features: Chart Options: Switch easily between vertical/horizontal chart. Marker Style: Choose from Shapes, Charts, Icons, Images, or Upload custom image. Small Multiples: Split your visual into multiple smaller visuals. Error Bars: Add error bars to show data variability, improving analysis accuracy. Race Chart: Enhance the chart by adding animations to show data changes over time. Cut/Clip Axis: Trim/Adjust the axis to accommodate the outliers. Dynamic Deviation: Analyze the deviation between two bars in a glance. Preview Slider: Easily explore various sections of a chart in large datasets using a slider. Conditional Formatting: Easily find outliers by using rules for measures or categories based on rules. Other features included are Templates, Import/Export Themes, Data Colors, Ranking and more. Business Use Cases: Sales Analysis, Financial Reporting, Market Research. Try Lollipop Chart visual for FREE from AppSource Check out all features of the visual: Demo_file Step-by-step instructions: Documentation YouTube Video: Video_Link Learn more about visuals: https://powerviz.ai/ Follow Powerviz: https://lnkd.in/gN_9Sa6U Other Now in Power BI Desktop - OneLake catalog The OneLake catalog is now part of the Power BI Desktop experience, providing a consistent and seamless way to discover and explore data. This update ensures alignment with the broader Fabric ecosystem, offering users a unified and familiar experience across tools. Platform Folder support in Git Timeline update - Folder support is planned to roll out to all customers by mid-April. Thank you for your patience! This update ensures that the folder structure in your Fabric workspace is seamlessly mirrored in your connected Git branch, providing an organized and consistent experience across both platforms. New features: Folder Structure Mirroring: The entire folder hierarchy in Fabric is reflected in Git and vice versa, enabling a more intuitive and organized collaboration process. Nested Folders Are Synced: Fabric items located within nested folders will now be included in the sync, and their folder structure will be preserved. Item Updates as Commits: Changes to an item's folder (e.g., moving an item to another folder or reorganizing folders) will now appear as updates or commits in Fabric. Subfolder support is enabled by default as soon as the feature is live. This means any folder differences between Fabric and Git will automatically show up as updates or commits. Handling Folder Changes Safely If changes to the connected branch cannot be made directly due to branch policy or permissions, we recommend using the ‘Checkout Branch’ option. Guidelines for managing this: Checkout a New Branch: Use the checkout branch feature to create a branch with the updated state of your Fabric workspace. Commit Folder Changes: Any workspace folder changes can then be committed to this new branch. Merge Changes: Use your regular pull request (PR) and merge processes to integrate these updates back into the original branch. OneLake OneLake Catalog - Semantic model table & column description We are expanding the details view of Semantic Models, to also include table and column descriptions which were set in the data model editor in the service or in Power BI Desktop. The goal is to provide consumers with multiple trust signals regarding an artifact, thereby enabling them to make swift and well-informed decisions. This improvement provides additional name and type details for tables and columns in semantic models, helping data consumers identify relevant tables more efficiently and encouraging data producers to document organizational knowledge. We are planning to expand this ability to other data items in the future. Filtering workspaces in OneLake Catalog You now have a dedicated filter for workspace names, which will allow you to quickly locate the required workspace. This makes it easier for users with access to multiple workspaces, and makes finding the relevant workspace difficult if it isn't featured at the top of the list. Data Engineering Python notebook (Preview) Announcing the preview of the highly anticipated Python Notebook! This new feature is designed to enhance the experience of BI developers and data scientists working with smaller datasets using Python as their primary language. Key features: Native Python support: Enjoy the full power of Python with native features and libraries right out of the box, like ipywidget, magic commands. Version flexibility: Easily switch between different Python versions (initially supporting Python 3.11 and 3.10). Optimized resource utilization: Benefit from better resource utilization with a smaller 2vCore/16G memory compute, real-time resource utilization monitor is available. Lakehouse & resources natively available: Leveraging the Fabric Lakehouse capabilities seamlessly, with built-in Resource folder to store your modules, libs and files. Mix programming with T-SQL: You can interact with data warehouses and SQL endpoints on Python notebook, with the built-in notebookutils connector. Superior Python intellisense: Powerful Pylance are natively integrated to provide smoother coding experience. Popular libraries are pre-installed: Using duckdb, polars and other popular 3 rd party libraries on Python notebook conveniently. Fabric utilities like Semantic Link and NotebookUtils are also natively supported. Seamless integration with Fabric ecosystem: All the advantages of Fabric notebook like sharing, CI/CD, schedule run, data pipeline integration, OrgAPP integration, are available for Python experience. Getting Started: Access the Notebook: You can access the Python Notebook from the Notebook language dropdown menu. Comprehensive Guide: A detailed guide is available to help you get started. Please refer to the public document to find more details. Your feedback is crucial in shaping the future of our product. We look forward to your active participation and valuable insights. Thank you for being a part of this exciting journey with us! Notebook live versioning Announcing the launch of the Fabric notebook version history feature. This new feature is designed to significantly improve your experience in developing and managing notebooks by providing robust built-in version control capabilities. Highlights: Automatic checkpoints: These checkpoints are created automatically every 5 minutes, ensuring that your work is consistently saved and versioned. Manual checkpoints: You can also manually create checkpoints to record your development milestones, providing flexibility in how you manage your notebook versions. Track history of changes: Users can now view a list of previous notebook versions, see what changes were made, contributed by whom, and when. Compare different versions: Easily compare different versions of a notebook through a diff view to understanding the evolution of your work. Restore previous versions: If you make a mistake or want to explore a different approach, you can restore previous versions of your notebook or save a new copy of it. NotebookUtils session management utilities Introducing a new utility in NotebookUtils- session management utilities, including a list of APIs that can help you manage your session and interpreter status. notebookutils.session.stop(): Support stopping the interactive session via code, it's available for Scala and PySpark. notebookutils.session.restartPython(): Support restarting the Python interpreter in PySpark notebook. For more details, please refer to the documentation. Native Execution Engine on Runtime 1.3: simplified enablement and transition from Runtime 1.2 Introducing a new update that simplifies enabling the Native Execution Engine. Now, activating it is as easy as toggling a switch! You’ll find the new toggle button in the Acceleration tab within your environment settings. If you were previously using the Native Execution Engine, please navigate to the Acceleration tab and re-enable it using the new toggle. This updated UI control now takes precedence over any previous configurations in Spark settings, meaning prior setups will remain inactive until re-enabled with the toggle. Additionally, the Native Execution Engine now fully supports our latest runtime version, Runtime 1.3 (Apache Spark 3.5, Delta Lake 3.2). As a result, support for Native Execution Engine on Runtime 1.2 is ending. We recommend upgrading to Runtime 1.3 to maintain support, as native acceleration will soon be unavailable on Runtime 1.2. Legacy timestamp support in Native Execution Engine on Runtime 1.3 The latest Native Execution Engine on Fabric Runtime 1.3 introduces legacy timestamp handling, ensuring compatibility across Spark versions. This feature addresses timestamp issues caused by Spark 3.0’s shift to the Java 8 date/time API (Proleptic Gregorian calendar) from the previous hybrid Julian-Gregorian calendar. With the configuration spark.gluten.legacy.timestamp.rebase.enabled, the Native Execution Engine auto-adjusts for calendar differences in Parquet files and Delta Tables, handling dates seamlessly across Spark versions. Dates post-1970 are unaffected, ensuring consistency without extra steps. To activate this feature, add the following to your Spark session: SET spark.gluten.legacy.timestamp.rebase.enabled = true; Notebook and Spark Job definition execution with service principal The service principal support for Fabric API was announced back in September. Today, we unblock another key scenario, enabling run the Notebook/Spark Job Definition execution under SP (service principal) for Data Engineering experience. Using the Fabric Job Scheduler API, users can trigger the execution of either a Notebook or Spark Job Definition (SJD) and monitor its execution status. By utilizing the same API with a service principal's access token, the Spark Job associated with the notebook/SJD will run within the security context of that service principal. This update enhances SP support for the Data Engineering experience, expanding its capabilities beyond current CRUD operations to include comprehensive job execution coverage. To make sure the SP does have the privilege to run the job, you need to add that SP as Admin/Contributor/Member into the workspace which hosts the Notebook/SJD. If the notebook/SJD has some code related with Data Science scenario such as Model/Experiment, the SP triggered execution could fail, this is something we are working on to unblock them later. Lineage Enhancement to Spark Notebook A new improvement to the lineage related to Spark Notebooks has been introduced to enhance data exploration effectiveness. You can now view all Lakehouses connected to your notebook, including pinned and additional Lakehouses. This update helps you: Perform Impact Analysis: Easily assess how changes affect data workflows by identifying which Lakehouses are being used. Document Data Pathways: Streamline collaboration and audits with clear visibility into data relationships. Experience smarter data management today with this lineage enhancement to your Spark Notebooks! Data Warehouse COPY INTO column count check The COPY statement offers flexible, high-throughput data ingestion from an external Azure storage account into Fabric Data Warehouse tables. When there is a column count mismatch between rows in source files and the target table, COPY INTO has the following behavior: If a row within the source files has less columns than the target table, COPY INTO inserts columns with missing values as NULL. If there’s no corresponding value for a non-nullable column in the source data, then COPY INTO fails. If a row within the source files has more columns than the target table, any excess columns from source files are ignored in the target table. We’re introducing a new option for COPY INTO that allows you to control the behavior of your data ingestion jobs by checking if the count of columns in the source data matches the count of columns on your target table. The following syntax should be used for the column count check option: COPY INTO FactSale FROM '<external_location>' WITH ( FILE_TYPE = 'CSV', [ , MATCH_COLUMN_COUNT = { 'ON' | 'OFF’ } ] ) MATCH_COLUMN_COUNT checks the column count on each row of each source file for a match against the target table specified in the COPY INTO statement. This option is available only for CSV file type sources now, with support for Parquet coming soon. The default behavior of COPY INTO remains unchanged and is equivalent to using MATCH_COLUMN_COUNT = ‘OFF’. Learn more about COPY INTO and this new option, refer to the documentation. Enhancing COPY INTO operations with Granular Permissions in Data Warehouse One of the challenges our customers shared is that executing COPY INTO command requires users to have at least the Contributor role at the workspace level, granting broad permissions that may exceed what is necessary for specific tasks. We are excited to announce that now a user with minimum ‘read’ permissions on the control plane will be allowed to execute write operations at the Data Warehouse level. The benefits of this change include reducing the need for broad workspace roles, and it also works seamlessly even when storage account is protected behind a firewall. Learn more about COPY INTO in our documentation and to know more about this new option, check out this blog post COPY INTO support for secure storage with granular permissions. Introducing default schema changes in Data Warehouse We are happy to announce the ability to change the default schema in Fabric Data Warehouse. With this improvement, which has been highly requested by our customers, we strive to make database management and enhanced security more straightforward. This is done using the ALTER USER statement, ensuring that every user has a predefined schema context when they connect to the database. ALTER USER [username] WITH DEFAULT_SCHEMA = [schema_name]; By allowing administrators to assign default schemas to users, we ensure that users operate within their designated schemas, reducing the risk of unauthorized access and simplifying permissions management. For more information, check our documentation: What is data warehousing in Microsoft Fabric? ALTER USER (Transact-SQL) Create a Database Schema Enhanced performance metrics in Query Insights New features have been introduced to provide deeper insights into query performance. With the introduction of Data Scanned Analysis, you can now determine if large data scans are contributing to slower query execution. This feature allows you to compare similar queries, pinpoint fluctuations caused by changes in data scanned, and even identify when cache was utilized. Additionally, we've introduced allocated CPU time as a key performance metric. This enables you to understand the resources consumed by your queries and workloads. High CPU time often correlates with higher costs, making it easier to identify and address resource-intensive queries. These enhancements empower you to optimize performance and manage costs effectively. These columns are available in queryinsights.exec_requests_history: Column name Data type Description allocated_cpu_time_ms Bigint Shows the total time of CPU(s) that was allocated for a query’s execution. data_scanned_remote_storage_mb Bigint Shows how much data was scanned/read from remote storage (One Lake). data_scanned_memory_mb Bigint Shows how much data was scanned from local memory. Data scanned from disk and memory together indicates how much data was read from cache. data_scanned_disk_mb Bigint Shows how much data was scanned/read from local disk. Data scanned from disk and memory together indicates how much data was read from cache. For more information, check out our documentation: Query insights queryinsights.exec_requests_history (Transact-SQL) Previewing estimated Query Plan available via SHOWPLAN_XML The Preview for SHOWPLAN_XML in Microsoft Fabric Data Warehouse is now available. This capability allows users to generate and view the estimated query execution plan in XML format, a tool for analyzing and optimizing SQL queries. Whether you're troubleshooting performance bottlenecks or refining query strategies during development, SHOWPLAN_XML offers a granular, detailed view of how the database engine plans to execute your queries. By providing insights into operations like joins, data movements, etc. it helps pinpoint inefficiencies and identify opportunities to enhance performance. How can you use SHOWPLAN_XML? Enabling SHOWPLAN_XML - To enable SHOWPLAN_XML, execute the following SQL command: SET SHOWPLAN_XML ON;This command instructs Fabric DW to return execution plans in XML format for all subsequent queries. Running Queries - After enabling SHOWPLAN_XML, run the queries that you wish to analyze.The execution plan for this query will be returned in XML format. Capturing the Output - Capture the SHOWPLAN_XML output by saving the result set to a file or copying it to an XML viewer. Ensure that the entire XML content is preserved for accurate analysis. When running SHOWPLAN_XML in Fabric UI, copy results and save them as a .sqlplan file. Open this file in SSMS to view the graphical plan. If you run in SSMS, use the SET SHOWPLAN_XML syntax as explained above. You can also use the plan Display Estimated Plan button to see the graph. Turn OFF SHOWPLAN_XML - Run SET SHOWPLAN_XML OFF to receive results instead of the execution plan when you run queries. For more details, check out our documentation: SET SHOWPLAN_XML (Transact-SQL) Query Hints in Fabric Data Warehouse Along with SHOWPLAN_XML, we are announcing support for some query hints. Query hints in Fabric SQL are optional keywords that can be added to SQL statements to provide additional information or instructions to the query optimizer. These hints can improve the performance, scalability, or consistency of queries by overriding the default behavior of the query optimizer. To use a query hint, the OPTION clause is added at the end of the query, followed by the name of the query hint and its optional parameters in parentheses. For instance, if you want to instruct the query optimizer to use a hash-based algorithm for the GROUP BY operation, you can use the HASH GROUP query hint. SELECT band_id, SUM(ticket_cost) FROM gigs GROUP BY band_id OPTION (HASH GROUP) Fabric SQL supports a variety of query hints, including HASH GROUP, ORDER GROUP, MERGE UNION, HASH UNION, CONCAT UNION, FORCE ORDER, LOOP JOIN, HASH JOIN and REPLICATE. Each of these hints serves a specific purpose, such as improving the performance of GROUP BY operations, UNION operations, or join operations. However, query hints should be used with caution and tested thoroughly, as they can have negative effects on the performance, scalability, or consistency of queries if used incorrectly or unnecessarily. It is essential to monitor and evaluate the impact of query hints on queries and adjust them as needed. For more details, refer to our documentation: Join hints (Transact-SQL) Query hints (Transact-SQL) Simplifying search & introducing Filter in Object Explorer The search and filter features in Fabric Data Warehouse empower users to efficiently navigate and manage their data. The new search feature is designed for ease of discovery and intuitive use, allowing users to locate objects in the object explorer by entering keywords. The search function quickly highlights matching objects and highlights the results within the object explorer for the user. The filtering feature in Object Explorer is an essential tool for managing large data warehouses and simplifying navigation within your warehouse environment. When dealing with numerous objects, such as schemas, tables, or stored procedures, finding specific items can be challenging. The filtering capability allows you to streamline this process effectively. The filtering options allow for precise object selection based on various criteria such as object type, created date, last updated, enabling you to focus on the most relevant information for your exploration in object explorer. By leveraging this combination, users can significantly reduce the time spent searching or filtering data, allowing for more focus on troubleshooting, generating scripts for development and documenting objects in Object Explorer. Open from SSMS & VS Code Developers now can easily access their Fabric Warehouse through their preferred client tools. With a renewed focus on integrating with widely used developer tools, this enhancement prioritizes flexibility and convenience, enabling seamless connections with SQL Server Management Studio and Visual Studio Code. This means you can dive right into your data analysis and management without any hassle, using the tools you know and love. Developers have the option to open Fabric Warehouse in SQL Server Management Studio (SSMS) or Visual Studio Code, either from a workspace or within the warehouse itself. You can open the warehouse in VS Code or by downloading VS Code. Note that Visual Studio code will install ms-sql for you and pre-populate Server and Database name in the connection to get started. You can also open or download SSMS to begin using your preferred tool. Git Status Bar to Fabric Warehouse Artifact The Git artifact status bar offers a comparable experience to the status bar in the workspace. When accessing the DW item page, you can view the details of the connection between the workspace and the Git repository, such as: The name of the branch to which the workspace is connected The time of the last sync event between the workspace and the repository A hyperlink to the most recent commit on the branch. The Git status bar is useful for following scenarios It offers a user-friendly interface for developers, like that of Visual Studio Code and other applications that display the status of the connection to the remote repository at the artifact level. More features, such as the ability to commit directly from the artifact page, will be added soon. Tooltip support for built-in functions Fabric Web editor provides robust tooltip support for built-in functions, enhancing the development experience by offering quick access to function details. When you hover over a built-in function in your query, web editor displays an interactive tooltip. This tooltip includes the function’s name, possible parameters, and a brief description. For example, if you hover over the COUNT() function, the tooltip will show its syntax and a short description of what the function does. This feature is particularly useful for quickly referencing function parameters and understanding their usage without leaving the query window. It helps streamline the coding process and reduces the need to manually look up function details. Stay updated with IntelliSense Our goal is to facilitate Fabric developers in writing queries by enabling new T-SQL statements in Fabric Warehouse. Recently, we have integrated IntelliSense support for the following newly released features. FOR JSON - Announcing improved JSON support in Fabric DW COPY INTO – Column count check: COPY INTO (Transact-SQL) - Azure Synapse Analytics and Microsoft Fabric JSON aggregates (Preview) As part of our ongoing enhancements to JSON functionalities in Fabric DW, we are excited to announce the preview of two new JSON aggregate functions: JSON_OBJECTAGG JSON_ARRAYAGG These aggregate functions simplify the process of concatenating columns within a GROUP BY operation and formatting them as JSON text. Previously, achieving this required the use of the generic STRING_AGG() aggregate combined with manually concatenated and escaped column values to produce a valid JSON string. Now, with JSON_OBJECTAGG and JSON_ARRAYAGG, this process is streamlined and more efficient. These functions are already in public preview in Azure SQL Database and Azure SQL Managed Instance, and Fabric DW is now joining this preview. They will become generally available across all SQL flavors simultaneously. Spatial analytic functions Spatial analytics functions are now fully supported in Fabric DW and SQL endpoints. Spatial functions enable you to perform complex calculations on the geographical and geometrical shapes, such as determining the distance between points, checking whether a point is within a polygon, or whether the shapes intersect. Previously, these functions were not fully supported. Comprehensive support commenced in December 2024. Fabric DW supports both geography objects and functions for simple 2D geometries, as well as geography objects for more realistic shapes represented on the Earth's surface. This includes all spatial reference systems available in SQL Server and Azure SQL databases. While Fabric DW does not support storing spatial types directly, you can represent your spatial objects as float columns representing the (latitude, longitude) pairs or store complex shapes in VARBINARY columns in Well-Known Binary (WKB) format. You can then use spatial functions to convert them to spatial objects and apply spatial operations. For example, the following query finds the number of trips starting near the Empire State Building in New York (40.748817, -73.985428): Be aware that the spatial functions are among the most complex calculations you can perform in your DW, which might impact performance. To improve the query performance, ensure you are physically storing a geoindex column that can approximately determine the location of an object or shape (e.g., using a bounding box, quadkey, H3 index, geohash, or something similar) and prefilter data based on this index column instead of applying a direct spatial filter. Since spatial filters and joins are the most resource-consuming operations, they might impact performance if rely only on them without additional indexing columns. Once you reduce your data set using geo index columns, you can perform complex spatial analytics using geography and geometry methods. SQL analytics endpoint performance improvement We released an update to the SQL analytics endpoint that improves query performance and data freshness. Previously you might have encountered slow SELECT statements and stale data in your tables. With this update, metadata changes are synced more efficiently to the SQL analytics endpoint, resulting in faster SELECT query execution and data updates. This improvement ensures a more responsive and reliable experience for our users. We'll continue to enhance the SQL analytics endpoint based on your feedback, so make sure to comment or vote on Ideas. Databases Tenant Level Private Link (Preview) We are excited to announce the preview of Tenant Level Private Link for SQL database in Fabric! This new feature enhances the security and privacy of your data by allowing you to connect to your SQL database through a private endpoint within your virtual network. With Private Link, you can now ensure that your data traffic remains within the Microsoft network, reducing exposure to the public internet and minimizing potential security risks. This integration simplifies network architecture and provides seamless and secure connection experience for your SQL database in Fabric. To enable Private Link in Fabric, start by creating a private endpoint within your virtual network (VNet) to securely connect to the Fabric service using a private IP address as outlined here Set up and use private links for secure access to Fabric Next, enable the Private Link toggle in the Fabric Admin Portal for your tenant to allow VNet requests to access Fabric resources. Additionally, you can choose to completely disallow any connections other than via Private Link by enabling the Block Public Internet Access toggle. For more information check out SQL database Overview (Preview). Copilot for SQL database in Fabric Region Availability Copilot for SQL database is now available in all supported regions listed in Fabric region availability! Today, we are offering three key features: Inline Code Completion for faster, smarter query writing. ‘Explain the Query’ & ‘Fix Query Error’ Quick Actions to simplify complex tasks. Sidecar Q&A Chat for answers and deeper understanding. Before your business can start using Copilot capabilities in Microsoft Fabric, please make sure to enable Copilot in the tenant settings. For more information, refer to these resources: Overview of Copilot for SQL database (Preview) Introducing Copilot for SQL database in Microsoft Fabric Introducing Copilot for SQL database in Microsoft Fabric | Data Exposed Real-time Intelligence Override late arrival tolerance in Activator Late arrival tolerance refers to how long Activator waits for an event to arrive and be acknowledged and processed. This setting ensures that late events and events that arrive out of order have an opportunity to be included in the rule evaluation. The consideration is to tradeoff on getting more ‘accurate’ rule evaluations by waiting longer for late data points to arrive or run your rule on potentially incomplete data, so the rule is activated sooner. The default setting is 2 min, but now you can set the late arrival tolerance to a longer or shorter period. Please note that this setting will not be shown for rules that are built on Power BI or KQL data. Create new Activator items As of November 2024, Real-time Intelligence and Activator are now Generally Available (GA)! You may need to work with your admin to ensure that you have all the capabilities available to you in Activator GA. If your tenant is using the preview version of Data Activator but does not have Fabric enabled, you will no longer be able to create new Activator items. To keep using Activator and create new Activator items, enable Fabric for your tenant. To enable Fabric, go to the admin portal and make sure that users are allowed to create Fabric items. Please note that if you have delegated settings to other admins, you should also allow capacity admins to enable/disable. RTI ALM & APIs GA Application Lifecycle Management (ALM) and Fabric REST APIs are now available for all RTI items: Eventstream, Eventhouse, KQL Database, Realtime dashboard, Query set and Data Activator. ALM includes both deployment pipelines and Git integration, both allow you to manage change within your workspaces. This enables multiple scenarios, from simply ‘recording’ changes in Git to have an audit trail, to deploying your development workspace, with all its dependencies to a staging and production workspace, to introducing changes via feature branch connected to feature workspace. REST APIs enables more control over changes by allowing you to programmatically create / read / update / delete (CRUD) your artifacts. Data Factory Mirroring Mirroring now supports replicating source schemas Mirroring in Fabric now supports replicating source schemas. When data is mirrored from various types of sources, your source schema hierarchy is preserved in the mirrored database. This ensures that your data remains consistently organized across different services, allowing you to consume it using the same logic in SQL analytics endpoint, Spark Notebooks, semantic models, and other references to the data. The existing mirrored databases remain unchanged to maintain backward compatibility and avoid affecting downstream workload. If you want to reorganize your tables with schemas, please recreate the mirrored database. Please refer to schema support with Mirroring in Microsoft Fabric. Delta column mapping support for Mirroring is now available Mirroring in Fabric now supports Delta column mapping. Column mapping is a feature of Delta tables that allows users to include spaces and special characters such as ',;{}()\n\t=.' in the table's column names. With this new capability in mirroring, columns containing spaces or special characters in names can now be replicated from your source databases to the mirrored databases. For tables with special characters in column names that are already under replication, you can update the mirrored database settings by removing and re-adding them to include those columns. To learn more, refer to Delta column mapping support with Mirroring in Microsoft Fabric. Mirroring now supports CI/CD (Preview) Mirroring in Fabric now supports CI/CD capabilities, enhancing the efficiency and reliability of your development workflows. Users can integrate Git for source control and utilize ALM Deployment Pipelines, streamlining the deployment process and ensuring seamless updates to mirrored databases. To learn more about these new capabilities, refer to CI/CD for mirrored databases in Fabric (Preview). Integrating SAP data into Open Mirroring dab (dab – We are your company for SAP data analytics) is the first partner to announce support for Open Mirroring from our SAP ecosystem. With over 20 years of experience in SAP analytics, dab offers a variety of analytic solutions covering multiple lines of business including accounting and procurement. Dab Nexus now integrates with Open Mirroring to synchronize data from various SAP sources including SAP S/4HANA (on-premises and Private Cloud Edition), SAP ECC, CRM, SRM, SCM and EWM. For more information on Open Mirroring with dab Nexus, refer to: Microsoft Fabric Open Mirroring: Efficient and innovative. To learn more about Open Mirroring, please refer to the documentation. Copy Job Simplify data ingestion with Copy Job: more connectors and better usability Copy Job simplifies data ingestion, providing a seamless experience from any source to any destination. Whether you need batch or incremental copying, Copy Job provides the flexibility to meet your data needs while keeping things simple and intuitive. Since the Public Preview launch at FabCon Europe in late September, we’ve quickly enhanced Copy Job with new features. We’re excited to announce that Copy Job now supports more connectors, including Snowflake and Azure SQL Managed Instance, for easier integration with your data sources. More connectors are coming soon! We’re committed to making Copy Job as simple and intuitive as possible, and your feedback is key to achieving that goal. You can now easily configure the update method and schedule before creating a copy job, giving you greater control and flexibility right from the start. Check out the details in What is Copy job (preview) in Data Factory. Dataflow Gen2 CI/CD support for Dataflows in Fabric We are delighted to share that CI/CD and GIT integration support for Dataflow Gen2 is now available in preview! You can opt into enabling these capabilities when creating a new Dataflow Gen2. With this new set of features, you will be able to seamlessly integrate your Dataflow Gen2 artifacts with your existing CI/CD pipelines and version control of your workspace in Fabric. This integration allows for better collaboration, versioning, and automation of your deployment process across dev, test, and production environments. Learn more about these new capabilities: Dataflow Gen2 CI/CD and GIT source control integration are now in preview! Single-line ribbon in Power Query editor The default experience for Dataflows Gen2 in Fabric now uses the ribbon in its single line mode. This brings consistency against other experiences that you will find within Microsoft Fabric. The full ribbon mode will remain available. You can access the complete ribbon experience by selecting the expand button. osoft_Fabric_January_2025_update Conclusion We hope that you enjoy the update! Be sure to join the conversation in the Fabric Community As always, keep voting on Ideas to help us determine what to build next. We are looking forward to hearing from you!Fabric February 2025 Feature Summary
Welcome to the Fabric 2025 update! There are a lot of exciting features for you this month! Here are some highlights: In Power BI, Explore from Copilot visual answers which lets you do easy ad-hoc exploration. In Data Warehouse, Browse files with OPENROWSET (Preview) and Copilot for Data Warehouse Chat (Preview). For Data Science, AI Skill is now conversational. These are just some of the great features this month, keep reading to learn about all of what’s happened in Fabric this month. Save your spot at Microsoft Fabric Conference 2025 Workshops are selling out, rooms are selling out, flights are selling out. Don’t wait! Register now for the biggest-ever Fabric learning event – March 31 to April 2, Las Vegas. Need help convincing your boss? Use this letter. We’re also thrilled to announce exclusive pricing for Fabric users! Use code FABINSIDER to save $400 and reserve your spot. Limited tickets available. Valid for new registrations only. Cannot be combined with any other offers. Limited-time 50% discount on Exam DP-700 Get ready to fast-track your career in data and AI by earning your Microsoft Certified: Fabric Data Engineer Associate certification (Exam DP-700). There's no faster way to climb the ladder than with industry-recognized Microsoft certifications. Don't wait to claim your discount—this offer ends on March 31, 2025. LIVE NOW! The first-ever Power BI DataViz World Championships! Data enthusiasts from around the world are battling it out and flexing their Power BI skills like true data wizards! The top four winners will earn their spot in the Grand Finale at the MGM in Las Vegas—LIVE at the Fabric Community Conference! Think you have what it takes? There are two more chances for you to show off your skills, connect with an amazing community, and win epic prizes! Don’t miss out on this legendary event! Contents General Upgrade Power BI Desktop to 64-bit version Upgrade Power BI Desktop to February 2025 version Copilot and AI Copilot in the Power BI mobile apps: now available on iPads and Android tablets (Preview) Updated section – ‘How Copilot arrived at this’ Explore from Copilot visual answers Reporting Conditional formatting support for visual calculations (Preview) Publish to Web Support for the Azure Maps visual (Coming soon) Modeling Live edit of semantic models in Direct Lake mode with Power BI Desktop – updates (Preview) Data connectivity Performance improvement in Snowflake connector (Preview) Use the official ODBC driver to connect to Vertica database (Preview) Visualizations Drill Down Donut PRO by ZoomCharts Heatmap by Powerviz Maximize data efficiency with accoMASTERDATA writenback for Power BI Other The OneLake catalog is now available in Microsoft Teams Monitoring Hub enhancement for Semantic models Fabric Platform Fabric API update: control cross-region deployments in Fabric Deployment pipelines Permission tab in the OneLake catalog Govern your individual data estate in OneLake catalog (Preview) Fabric Capacities Management APIs OneLake Announcing support for storage accounts that are behind a firewall for Mirrored Azure Databricks catalog items (Preview) Data Engineering Managed private endpoint and private link support for Native Execution Engine in Spark New Capacity settings for better compute governance on your Data Engineering Workloads T-SQL Notebook enhancements Full CI/CD support for T-SQL notebook including Git and Deployment pipeline New Lakehouse samples Monitoring for High Concurrency Notebook Snapshots enhancements Fabric Spark Resource Analysis (Generally available) Data Science Announcing AI functions for LLM-powered data enrichment and transformation (Preview) Support for multiple data sources in AI Skill AI Skill is now conversational Semantic model support with AI Skill KQL databases support in AI Skill Reason over multiple steps with AI Skill Data Warehouse Collate clause support in Warehouse (Generally available) Copilot for Data Warehouse Chat (Preview) Browse files with OPENROWSET (Preview) Use the BULK INSERT to load your data (Preview) Nested common table expression (Generally available) Real-Time Intelligence Configure latency of Eventhouse OneLake availability Real-Time Dashboard & Power BI templates for Eventhouse Monitoring New Real-Time Dashboard customization features: more control, better usability, and improved performance New and improved Data Source Tree in KQL Queryset Data exploration is now available for raw KQL table data Databases Performance Dashboard in SQL database in Microsoft Fabric Data Factory Mirroring in Fabric now supports workspace monitoring (Preview) New regions available for Mirroring in Fabric Open mirroring for SAP sources – Simplement and SNP https://youtu.be/yxQzr5qzeJE?si=9QOIfyFU1Ox1FJiz Power BI General Upgrade Power BI Desktop to 64-bit version The 32-bit version of Power BI Desktop will no longer be supported after 6/30/2025. Upgrade to the 64-bit version Power BI Desktop to keep receiving updates and support. Upgrade Power BI Desktop to February 2025 version To continue using Report view Copilot chat pane, you will need to upgrade to February 2025 version. Report view Copilot chat pane might not work as expected for versions before February 2025 after 4/30/2025. Copilot and AI Copilot in the Power BI mobile apps: now available on iPads and Android tablets (Preview) Copilot support is now available in the Power BI Mobile apps on iPads and Android tablets, extending the functionality of AI to more mobile devices. Introduced a few months ago for phones, Copilot in the Power BI Mobile apps empowered users to quickly analyze data, gain insights, and make informed decisions on the go. Now, the same powerful capabilities have come to iPads and Android tablets, making it easier than ever for you to explore your data anytime, anywhere. Getting started with Copilot on your mobile app is easy, simply tap the Copilot button located in the report header (for reports that meet Copilot requirements in Power BI). From there, you can choose whether to receive a summary or uncover insights. Copilot will deliver a response based on your request, which you can then copy, share, or continue interacting with. Suggestions at the bottom of the screen help you refine your request or create new ones, making it simple to explore your data further. Updated section – ‘How Copilot arrived at this’ We’ve heard an overwhelming amount of feedback that users would like more transparency into how Copilot generates visual answers. This month, we’ve revamped our previous ‘show reasoning’ section to a new detailed section we’re calling ‘How Copilot arrived at this’. When receiving a visual answer from Copilot, expand this new section to dig into what data (fields/measures) and filters were used to generate the answer. You can also click on the field to see more information like the aggregation or home table. This is especially helpful when there could be multiple fields with the same name, like having multiple Dates in your data, verifying Copilot picked the correct one. As a reminder, Copilot does not currently incorporate existing filters from the report when generating a visual answer. This list only contains filters that were used at the time of generating the answer. This is a step towards increasing transparency in how Copilot understands and generates responses. If you have any additional suggestions, please share your thoughts in the comments. https://youtu.be/2uQ2MTy9a1k?si=2QeUlWylEPih3WBm Explore from Copilot visual answers Sometimes in Copilot, you may want to do a bit more with the visual answer, like drill down, filter it further, maybe even swap it to a different visual type. Until now, the only way to take action on the Copilot generated visual was to add it to the page. This of course was limited to ‘edit mode’ only. But we have great news! This month we’ve replaced the previous ‘expand view’ action with our fully interactive Explore feature. Even better, you can use the feature in both read and edit modes of a report. With Explore, quick actions are now possible from Copilot. Simply open the Explore experience to filter, sort, swap fields, or change visual types easily for ad-hoc exploration. You can choose to save the new explore visual to a report or an exploration to come back to. Note: Saving the new visual to the original report is not yet available. Explore does have certain limitations, some users may still see the ‘expand view’ button if requirements to use Explore are not met. https://youtu.be/BS0RxFhuXl0?si=6gL64j4HxytozTiw Reporting Conditional formatting support for visual calculations (Preview) Another month, more updates to visual calculations. This month we are introducing another customer request: support for conditional formatting! You can now use visual calculations to set up conditional formatting rules for columns and measures on your visuals. Also, you can now apply condition formatting rules to visual calculations. Once you open the conditional formatting dialog you will notice there is a new section that shows the data that’s on your visual. This section includes any column, measure and visual calculations on your visual and will include any hidden fields as well. For example, here I have a simple visual that shows sales by quarter using a Total Sales measure and the Fiscal Quarter column. I have added a visual calculation that for each quarter calculates the difference with the first quarter: VSFirst = [Total Sales] - FIRST( [Total Sales] ) I have hidden the VSFirst calculation and am going to use it to set a conditional background color for the fiscal quarter. In the conditional formatting dialog, we see the data in this visual section: All your model data is still accessible, it is in the 'All data' section. Next, select VSFirst visual calculation and set up the rule as usual: And the conditional formatting is applied: As mentioned, you can also format the visual calculations themselves, as done here with the same visual calculation that was made visible to show the result: ower_BI_Desktop_showing_the_conditional_formatting_dialog_for_the Learn more about visual calculations in our documentation and please let us know what you think! https://youtu.be/E7s_nVHgNVk?si=5TNAgXxpECfhdpn0 Publish to Web Support for the Azure Maps visual (Coming soon) We’re excited to announce that Azure Maps support for Publish to Web reports is rolling out. With this update you don’t need to take any extra steps to take advantage of your Azure Map visuals in Publish to Web reports. Just get your embedded code as you normally would using the Publish to Web (public) option under the Embed Report section of the File menu. From there, you can use the embedded report as you normally would, now with your maps showing. This update is in the process of rolling out, so you should see the change within the next couple of weeks. https://youtu.be/6BLmRHyeZMM?si=PWEu87I5ziq_m-wu Modeling Live edit of semantic models in Direct Lake mode with Power BI Desktop – updates (Preview) Improved modeling performance We have significantly enhanced the modeling performance for live editing of Direct Lake semantic models using Power BI Desktop. Upgrading to the latest version will result in at least a 50% improvement in each modeling change. More details on the feature, including requirements, considerations, and limitations can be found in the documentation. We highly value your feedback on this feature and encourage you to share it through our feedback form or the Power BI Community. Data connectivity Performance improvement in Snowflake connector (Preview) We sincerely appreciate your trial and feedback on the new Snowflake connector released as preview in January. This month, we have enhanced this connector with performance improvements and bug fixes to provide a smoother experience. We invite you to upgrade to the latest version to evaluate these enhancements. Learn more about the Snowflake connector from the Snowflake connector documentation. Use the official ODBC driver to connect to Vertica database (Preview) This month, we are introducing a new option for getting data from the Vertica database using the official Vertica ODBC driver, currently available in preview. This feature allows you to take advantage of the native client tool provided by the data source. More details about this feature, including the driver installation and configuration instructions, can be found in the documentation. Visualizations Drill Down Donut PRO by ZoomCharts Donut PRO is more than just a beautiful and customizable donut chart visual. It is also a way to instantly make your reports more interactive and user-friendly. Simply click directly on a slice to drill down a level and reveal more details. Donut PRO will also seamlessly cross-filter with other visuals, allowing users to quickly filter data and instantly uncover valuable insights that empower confident business decisions. Main features: Interactivity: Donut PRO provides equally great experience for mouse and touch input - click/tap to drill down, hold to select, triple click/tap to invert selection. Animated navigation: Each user interaction has smooth animation feedback that aids with navigation and shows exactly what changed in the chart. Multi-level drill down: Create up to nine levels of hierarchy and drill down with just a click. Automatic ‘others’ slice: Declutter the chart by grouping smaller slices into one larger slice. Simply click to reveal its contents. Cross-chart filtering: Enhance data storytelling by dynamically filtering data across multiple visuals. Full customization: Finetune your chart with more than 80 settings, including donut appearance, slice colors, detail labels, legend, and tooltip customization. Get on AppSource Heatmap by Powerviz The Powerviz Heatmap visualizes data density and value distribution using color gradients, transforming complex datasets into an intuitive matrix format. It is ideal for identifying patterns, correlations, and outliers, aiding data-driven decision-making. Key features: Shapes: Choose from Default, Fixed, or Diverging built-in shapes. Data colors: Offers 30+ palettes, color-blind mode, and custom color options. Data labels: Select from different styles available or add custom labels with formatting. Grid lines: Improve data readability with visual grid lines. Reference lines: Highlight key points using X-axis and Y-axis lines. Totals: Display row and column totals with advanced bar customization. Null values: Customize null values (Blank /NA / Null/ ”0”) and their styling. Small multiples: Split visuals into smaller ones using selected fields. Conditional formatting: Spot outliers with Value/ Percentage/ Ranking based rules. Ranking: Filter Top/Bottom N rows or columns. Other features included Axis Settings, Grid View, Sorting, Show Condition and more. Business use cases: Sales Analysis, Performance Evaluation, Risk Assessment. Try Heatmap visual for FREE from App Source Check out all features of the visual: Demo Step-by-step instructions: Documentation YouTube video: Video Link Learn more about visuals: https://powerviz.ai/visuals Follow Powerviz: https://lnkd.in/gN_9Sa6U Maximize data efficiency with accoMASTERDATA writenback for Power BI Unlock the power of seamless masterdata management with accoMASTERDATA for Power BI. Streamline your data processes and enhance your writeback capabilities for smarter business decisions. Key features: Effortless writeback: Easily configure writeback to SQL and FABRIC databases. Track changes in a dedicated log for full transparency. Full control: Create, update, and delete rows in Masterdata tables to keep your data dynamic and accurate. Custom validation: Use regular expressions for custom data validation and ensure integrity. Dropdown list validation: Restrict input to predefined options for consistent data entry. Conditional formatting & rules: Ensure quality data with powerful validation and formatting tools. Enhanced copy-paste: Copy and paste from Excel directly into Power BI while maintaining validation rules. Customizable themes: Create visually stunning reports with flexible theme options. Tags & column grouping: Organize data with multi-selected tags and improve navigation with collapsible column sections. Integrating accoMASTERDATA with Power BI will supercharge your data management, improve reporting, and streamline your operations. Start transforming your data handling today! Visit the accoTOOL website for more. Other The OneLake catalog is now available in Microsoft Teams The OneLake catalog is now integrated into the Power BI app experience in Microsoft Teams, enabling seamless data discovery and exploration within the Microsoft Office ecosystem. This integration empowers business users and professionals to interact with their data - explore it, take actions, and more - all without leaving Teams. Monitoring Hub enhancement for Semantic models In Fabric Monitoring Hub, you can centrally monitor Microsoft Fabric activities. It displays refresh activities for all semantic models, each showing one line with the status of the last refresh. Previously, clicking on an activity name in the Monitoring Hub directed you to the semantic model detail page. This month, we’re introducing the Semantic model refresh detail page. This new page shows comprehensive details of a selected refresh activity, including capacity, gateway, start and end times, error details, and multiple refresh attempts. For each refresh attempt, you can view the execution metrics by clicking on the 'Show' link in the 'Execution details' column. This information can assist with troubleshooting or optimizing the semantic model refresh. Previously, this data was accessible through Log Analytics or Fabric Workspace Monitoring. It's also possible to link refresh details from external applications. The semantic model refresh detail page can be accessed from other locations by constructing a URL with the workspace, semantic model, and refresh ID: https://app.powerbi.com/groups/{workspaceId}/datasets/{semanticModelId}/refreshdetails/{refreshId} For instance, this Fabric Notebook uses semantic link sempy and Power BI API Get Refresh History to create a refresh detail URL for each run of a semantic model: import sempy import sempy.fabric as fabric import pandas as pd workspaceId = "[Your Workspace Id]" semanticModelId = "[Your semantic model Id]" client = fabric.FabricRestClient() response = client.get(f"/v1.0/myorg/groups/{workspaceId}/datasets/{semanticModelId}/refreshes") refreshHistory = pd.json_normalize(response.json()['value']) refreshHistory["refreshLink"] = refreshHistory.apply(lambda x:f"https://msit.powerbi.com/groups/{workspaceId}/datasets/{semanticModelId}/refreshdetails/{x['requestId']}", axis=1) displayHTML(refreshHistory[["requestId", "refreshLink"]].to_html(render_links=True, escape=False)) More details on the feature can be found in the data refresh documentation. Fabric Platform Fabric API update: control cross-region deployments in Fabric Deployment pipelines We are excited to announce a significant update to the Fabric Deployment pipelines APIs, which now includes a new feature for managing cross-region deployments. This change is designed to enhance security and compliance while providing developers with greater control over their deployment processes. What's new? The latest update introduces a new flag in the fabric deployment request API that allows developers to specify whether cross-region deployments are permitted. This flag, known as allowCrossRegionDeployment, must be explicitly set by the developer to enable cross-region deployments in fabric. By default, this flag is set to false for security reasons. Why the change? Cross-region deployments can pose security and compliance risks, especially in highly regulated environments. By requiring developers to explicitly state whether cross-region deployments are allowed, we aim to mitigate these risks and ensure that deployments adhere to organizational policies and regulatory requirements. How does it work? When creating or updating a deployment pipeline, developers can include the allowCrossRegionDeployment flag in their API request. If this flag is set to false (the default value), any attempt to deploy across regions will be blocked. If set to true, the deployment will proceed, provided all other conditions are met. Here is an example of how to use the new flag in an API request: { "options":{ "allowCrossRegionDeployment": true } } What is the impact? The introduction of the allowCrossRegionDeployment flag in the Fabric Deployment pipelines APIs brings several changes that impact existing deployment pipelines. Here are the key points to consider: Default behavior: By default, the allowCrossRegionDeployment flag is set to false for security reasons. This means that unless explicitly set to true, cross-region deployments will be blocked. Explicit permission required: Developers must now explicitly set the allowCrossRegionDeployment flag to true in their deployment requests if they wish to enable cross-region deployments. This change ensures that cross-region deployments are intentional and comply with organizational policies. Impact on existing pipelines: Existing deployment pipelines that do not include the allowCrossRegionDeployment flag will default to blocking cross-region deployments. Developers will need to update their pipelines to include this flag if cross-region deployments are required. Error handling: If a deployment request attempts to perform a cross-region deployment without setting the allowCrossRegionDeployment flag to true, the deployment will fail, and an error message will be returned. This message will inform the user that the deployment failed because cross-region deployment is not allowed. Security and compliance: This update enhances security and compliance by ensuring that cross-region deployments are explicitly authorized. It helps organizations adhere to regional data residency requirements and other regulatory guidelines. Overall, the introduction of the allowCrossRegionDeployment flag provides greater control over cross-region deployments, ensuring that they are secure and compliant with organizational policies. Developers will need to update their existing deployment pipelines to include this flag if cross-region deployments are necessary. Permission tab in the OneLake catalog For data owners, security and compliance are top priorities. To support this, we’re expanding the OneLake catalog's detailed view of all item types to include permission management. Users with the necessary privileges will be able to view and manage access to items directly within the catalog. This feature will provide a more efficient and streamlined way to manage permissions without leaving the item's context. Govern your individual data estate in OneLake catalog (Preview) We are excited to introduce as preview a centralized data governance experience in the OneLake catalog. In this tab, data owners can view aggregated insights on the items they created, consider improving their governance by taking recommended actions, and access more information along with all available tools in Fabric. This functionality facilitates data creators in ensuring their items are secure and compliant with organizational policies. https://youtu.be/5rtSNAvvyVM?si=TUmPgRInCARyB4YH Fabric Capacities Management APIs In the ever-evolving landscape of cloud computing, the introduction of new application programming interfaces (APIs) can significantly enhance the capabilities and efficiency of developers. We are excited to announce the availability of a new set of ARM APIs designed for comprehensive management of Fabric capacities. These APIs are tailored to streamline operations, enable precise control, and provide unprecedented flexibility in managing Fabric resources. Overview of the new ARM APIs The new APIs for Fabric capacities management encompass a wide range of functionalities. Below, we provide an overview of each API and its primary operations: Check Name Availability: Implements local CheckNameAvailability operations to verify if a particular name is available for use. This API is crucial for ensuring that your chosen names for Fabric capacities are unique and compliant with naming conventions. Create Or Update: This API allows for the creation or updating of a FabricCapacity. It provides developers with the flexibility to either establish a new Fabric capacity or modify an existing one based on their evolving needs. Delete: The Delete API facilitates the removal of a FabricCapacity. This operation is essential for maintaining an optimal and efficient resource environment by eliminating obsolete or unnecessary capacities. Get: This API retrieves details about a specific FabricCapacity. It allows developers to access comprehensive information about their Fabric capacities, ensuring transparency and informed decision-making. List By Resource Group: Lists all FabricCapacity resources within a specified resource group. This functionality is particularly useful for organizing and managing resources in a structured manner. List By Subscription: This API lists FabricCapacity resources by subscription ID, offering a bird's-eye view of all capacities under a particular subscription. It aids in efficient subscription-level resource management. List Skus: Lists eligible SKUs for the Microsoft Fabric resource provider. This API enables developers to explore and select from various SKUs that best meet their requirements. List Skus For Capacity: Similar to the previous API, but specifically lists eligible SKUs for a particular Microsoft Fabric resource. This ensures that developers can tailor their capacities with the most appropriate SKUs. Resume: This API resumes the operation of a specified Fabric capacity instance. It is vital for restoring the functionality of capacities that have been temporarily suspended. Suspend: Conversely, the Suspend API allows for the temporary suspension of a Fabric capacity instance. This can be useful for maintenance or other operational considerations. Update: The Update API provides the means to modify an existing FabricCapacity, ensuring that capacities can be adjusted and optimized as needed. Benefits of the New APIs The introduction of these new ARM APIs offers several significant benefits to developers and organizations alike: Enhanced Control and Flexibility These APIs offer fine-grained control over Fabric capacities, allowing developers to create, update, suspend, and resume capacities as needed. This flexibility ensures that resources can be dynamically managed to meet changing demands and operational requirements. Improved Resource Management With APIs such as List By Resource Group and List By Subscription, developers can easily organize and manage their resources. This structured approach to resource management aids in maintaining an efficient and optimized cloud environment. Streamlined Operations The ability to quickly check name availability, list eligible SKUs, and retrieve detailed information about Fabric capacities simplifies the operational workflow. Developers can now perform these tasks with ease, reducing the time and effort required for resource management. Seamless Integration These APIs are designed to integrate seamlessly with existing ARM tools and processes. This ensures that developers can leverage the new functionalities without the need for significant changes to their existing workflows. More information can be found in the Fabric Capacities documentation and Fabric REST API documentation OneLake Announcing support for storage accounts that are behind a firewall for Mirrored Azure Databricks catalog items (Preview) A significant number of enterprise customers have their ADLS storage accounts behind a firewall, which houses their Unity Catalog data. Previously, Mirrored Azure Databricks catalog items were unable to access storage accounts behind a firewall, preventing these customers from accessing their catalog data via Fabric. We are excited to announce the preview of support for storage accounts behind a firewall for Mirrored Azure Databricks catalog items. This new feature enables enterprise customers to securely access their Unity Catalog data, even when it is stored in ADLS accounts protected by a firewall. Navigate to the Network Security tab once you have picked a catalog that you want to access from Fabric. 2. Select an existing ADLS connection if you have one configured or create a new one. Authorization types supported are Workspace Identity (recommended), Organizational Account, and Service principal. 3. Provide access rights to the ADLS account based on the authentication type you picked in the previous step. 4. Firewall enables your ADLS storage account using Trusted Workspace Access. Data Engineering Managed private endpoint and private link support for Native Execution Engine in Spark Organizations heavily rely on cloud data platforms for processing data at scale and enterprise operating in healthcare, financial and other domains where data security becomes critical requirement enforcing strict data security standards on all layers. One of the primary approaches has been through blocking public access from untrusted networks using Private Links for inbound access restrictions. Now when it comes to outbound access enterprise also stores these sensitive data in data sources behind a firewall and would want their analytics systems to connect using a secure channel like managed private endpoints to block public access on these data sources. We are excited to announce the support for Private Links and Managed Private Endpoints for Native Execution Engine which is Fabric for your data engineering workloads. Native Execution Engine is the vectorized engine optimizes the performance and efficiency of your Spark queries by running them directly on your lakehouse infrastructure. The engine's seamless integration means it requires no code modifications and avoids vendor lock-in. It supports Apache Spark APIs and is compatible with Runtime 1.3 (Apache Spark 3.5) and works with both Parquet and Delta formats. Users can enable Native Execution Engine from the Acceleration tab in the Environment item. With this support users can now run their jobs using Native Execution Engine in their tenants where they have enabled Private Links and run their Spark Jobs using Native Execution Engine when they are connecting to data sources using Managed Private Endpoints. Learn more about Native Execution Engine from our documentation: Native execution engine for Fabric Spark. New Capacity settings for better compute governance on your Data Engineering Workloads We have added new controls as part of the Data Engineering/Science Capacity settings including the option to Disable Starter Pool for workspaces to help capacity admins have better compute restrictions on workspaces attached to their Fabric Capacity. Use the following steps to manage the Data Engineering/Science settings for Microsoft Fabric capacity: 1. Select the Settings option to open the setting pane for your Fabric account. 2. Select Admin portal under Governance and insights section. 3. Choose the Capacity settings option to expand the menu and select Fabric capacity tab. Here you should see the capacities that you have created in your tenant. Choose the capacity that you want to configure. 4. In the Spark Settings, disable ‘Customized Workspace Pools’. This option enables Capacity administrators to prevent Workspace administrators from creating and managing Spark compute on their Workspaces. 5. Now by selecting the option to Disable Starter Pool for the workspace, capacity admins can also disable Starter Pools for the workspaces, allowing workspace users to utilize Capacity Pools created based on their workload requirements for better controls over resource governance. T-SQL Notebook enhancements We’ve made some updates to incremental updates the T-SQL Notebook, including: More T-SQL code snippets Open the full data set within Excel Traceability of Primary warehouse CI/CD support Inside the Notebook Code snippet library, a new TSQL category is added. You can find a list of T-SQL code templates based on different use cases. Data ingestion: Copy Into, create table as select, Insert into. Business continuity: Create clone table, create clone table at a point in time. Design and develop: Create schema, create table, create view, create parameterized stored procedure, create stored procedure with output parameter, create statistics, create inline table-valued function, drop schema, drop table, drop view, drop store procedure, drop function. Monitoring: Active queries, historical queries, session history, long running queries, most frequent queries. Open in Excel feature is also available in T-SQL notebook now. To get the full dataset from the submitted query, you can click the ‘Open in Excel’ button in the result view: Learn more about setting up authentication inside Excel. In T-SQL notebooks, users can add multiple data warehouse/SQL endpoints and set one as the primary warehouse. The primary warehouse name appears in the code cell status bar after execution, with traceability of its usage enabled, even if the primary warehouse is changed later. https://youtu.be/e31JvJ-VW6A?si=32U0DW2p9Pxelmm- Full CI/CD support for T-SQL notebook including Git and Deployment pipeline With Git integration, you can commit your T-SQL code and the bindings between Notebook and warehouse into Git repo for history checking. Once you export the repo source into a different workspace, a new T-SQL notebook and warehouse item will be created. With deployment pipeline integration, you can deploy the T-SQL notebook across different stages/workspaces. New Lakehouse samples We’ve listened to your feedback and are excited to release two new Lakehouse samples, designed to support deeper analysis and flexibility for both schema-enabled and non-schema Lakehouses. New samples Retail data model (Wide World Importers): a dataset from a fictional novelty goods distributor, great for retail and supply chain analytics. NYC Taxi: taxi trip data, including pick-up/drop-off details, distances, and fares—ideal for pattern and trend exploration. How to try these, you can get started in two ways: From the Workload page: navigate to Data Engineering > Explore a sample, and a schema-enabled lakehouse is automatically created. 2. From the Lakehouse experience: start in an empty lakehouse, and the samples will adapt to your schema settings. Larger datasets provide a more realistic foundation for analysis, enabling you to test complex queries, simulate real-world scenarios, and uncover richer insights. Start exploring today and unlock the full potential of your Lakehouse! Monitoring for High Concurrency We are excited to introduce several enhancements for monitoring Notebook runs within high concurrency Spark sessions. These improvements allow for better tracking and troubleshooting of Notebook executions, whether they run directly in high concurrency Spark sessions or share the same Spark sessions from a pipeline. Key enhancements: Log segmentation for improved troubleshooting. Troubleshooting through logs is a critical task for data engineers. We now support log segmentation for each Notebook running in a high concurrency Spark session. This feature allows you to filter driver logs at the Notebook level, making it easier to identify failures or performance issues specific to an individual Notebook run. 2. Auto-mapping Spark Jobs, stages, and tasks to Notebooks. Spark jobs, stages, and tasks within a Spark session are now automatically mapped to their respective Notebooks. You can easily navigate Spark jobs and stages associated with a Notebook and access the corresponding code snippet when needed. Notebook Snapshots enhancements Notebook Snapshots feature has been a widely used feature, enabling users to view the original notebook code at the time of execution, examine outputs based on different parameters, and drill down into each notebook cell to gain deeper insights into execution status and cell-level outputs. To better address customer needs, we have introduced several enhancements to support additional Notebook run scenarios. Support for Notebook runs triggered by NotebookUtils Users can now view snapshots for Notebook runs triggered by NotebookUtils methods, such as notebookutils.notebook.run() or notebookutils.notebook.runMultiple(), once the reference Notebook runs are complete. The snapshot tree explorer now provides a hierarchical view, making it easier to understand parent-child relationships and track which parent notebooks have triggered child notebooks via NotebookUtils. Snapshots for running Notebooks Users can now access snapshots of Notebook runs from a Pipeline or direct schedule while they are still in progress. This enhancement allows you to monitor execution status in real-time, view progress, and check output or cell-level errors as they occur. Enhanced visibility into parallel Notebook runs in High-Concurrency Spark Sessions An expandable and collapsible tree-structured explorer provides a snapshot of all parallel Notebook runs within a high-concurrency Spark session. You can view all Notebooks associated with the same Spark session and click on each Notebook to access its corresponding snapshot, including the original code, input parameters, and cell outputs. Fabric Spark Resource Analysis (Generally available) We are thrilled to announce the general availability of the Fabric Spark Resource Utilization Analysis feature, designed to provide in-depth insights into your Apache Spark applications within Microsoft Fabric. This feature helps identify potential bottlenecks by analyzing the distribution of running and idled cores and enables you to better understand executor allocation to optimize resource usage and improve application performance. Key features: Real-time executor cores Monitoring - Access the Resources tab to view a comprehensive graph displaying four distinct metrics: Running: Actual number of cores utilized by the Spark application for executing jobs and tasks. Idled: Cores that are available but currently unused during the application's runtime. Allocated: Total cores allocated for the Spark application's operation. Maximum Instances: The upper limit of cores that can be allocated to the Spark application. 2. Interactive resource utilization graph: Interactively explore these metrics to gain a clear understanding of resource allocation and utilization. Hover over the running executor cores graph to view summaries of cores and corresponding executor information. Click on specific points in the graph to access detailed information about executors and jobs at that moment, facilitating precise performance analysis. This feature is supported in Spark runtime version 3.4 and above. For a comprehensive guide on utilizing this new capability, please refer to our official documentation: Monitor Apache Spark Applications Resource Utilization We encourage you to explore the Fabric Spark Resource Utilization Analysis feature to optimize your Spark applications and enhance performance monitoring within Microsoft Fabric. Data Science Announcing AI functions for LLM-powered data enrichment and transformation (Preview) We’re thrilled to announce the preview of AI functions in Fabric will be available by the end of February 2025. AI functions provide a simplified API for common LLM-powered enrichments on text-based data in notebooks—whether you’re looking to translate customer reviews from one language into another or to analyze the sentiment of detailed call logs. Now, in just a single line of code, you can harness the power of Fabric’s built-in LLM endpoint for summarization, translation, classification, grammatical correction, custom text generation, and more. AI functions are designed for ease, with no need for complex config or GPU infrastructure management. But, if you choose, you also have the option to bring in your own AzureOpenAI LLM for more customized solutions. AI functions are initially available in pandas and Spark but stay tuned for the ability to invoke them with SQL—and much more. https://youtu.be/UEv7wvmo020?si=DcC3v1GWoObsXDNl Support for multiple data sources in AI Skill The AI skill in Fabric now supports multiple data sources, giving users even more flexibility when exploring their data. Users can now add up to five data sources, select the relevant tables, and ask natural language questions—all in one seamless experience. The AI skill intelligently determines the best data source to answer each question, ensuring more accurate and relevant responses. Additionally, users can provide instructions to fine-tune results and guide the AI skill on which sources to prioritize. This enhancement makes it easier than ever to gain insights from diverse datasets, all within Fabric. https://youtu.be/e31JvJ-VW6A?si=32U0DW2p9Pxelmm- AI Skill is now conversational The AI skill in Fabric is now fully conversational, making interactions more seamless and intuitive. Your conversations are preserved between sessions, allowing you to pick up right where you left off without losing context. Additionally, the AI skill now remembers your previous questions and answers, making it easier to ask for follow-ups or clarifications without having to re-explain your query. Whether you're analyzing data, refining insights, or iterating on queries, this update makes engaging with the AI skill feel more like a natural conversation, streamlining your workflow and enhancing productivity. Semantic model support with AI Skill The AI Skill now supports Semantic models, a new data source that enables AI Creators to integrate structured business data for natural language querying. Semantic Models provide a curated data layer that defines business logic, relationships, and aggregations, making it easier for users to analyze and explore data without needing deep technical expertise. AI Creators can add Semantic models and selectively scope which tables are available for querying. Once configured, both AI Creators and consumers can ask natural language questions, which the AI Skill translates into DAX (Data Analysis Expressions) queries using LLMs. Users can trace the entire query process in the chat canvas, viewing how their input is interpreted, which transformations occur, and the exact DAX queries executed. KQL databases support in AI Skill Users can now link their KQL databases as a data source in AI Skill, enhancing capabilities for real-time streaming and data analysis. AI Creators can configure their AI Skill with an Eventhouse Data Source by utilizing AI Instructions to provide the AI Skill agent with relevant context, providing Few Shots sample KQL queries, and selecting relevant tables for the creators use case. After configuring the AI Skill, consumers can submit natural language questions about the data set and have the AI Skill generate a curated response presenting its interpretation of the prompt and the Kusto query used to generate the response. This powerful combination leverages the real-time streaming and analysis capabilities of Kusto with the customizability of the AI Skill to empower users to gain valuable insights from their data without technical expertise. Reason over multiple steps with AI Skill The AI skill in Fabric is now even more intelligent, breaking down complex questions into multiple steps to find the best answer. Instead of just responding to a question in one go, the AI skill can now figure out when it needs to pull information from different sources and take the right steps to get a complete answer. For example, if your question requires checking both your Lakehouse and a KQL database, the AI skill will automatically query each one. You’ll also be able to see exactly how it got to the final answer—every step, every query, and every result—all clearly displayed in the AI skill chat experience. This update makes the AI skill more transparent, reliable, and smarter in helping you get the answers you need. https://youtu.be/VpLnX-_CU1g?si=kNcO_WXX9tqJidRV Data Warehouse Collate clause support in Warehouse (Generally available) All Fabric warehouses by default are configured with case-sensitive (CS) collation Latin1_General_100_BIN2_UTF8. You can also create warehouses with case-insensitive (CI) collation Latin1_General_100_CI_AS_KS_WS_SC_UTF8 via REST API. In a SELECT statement, the COLLATE clause is unrestricted and can be applied with unsupported collations. You will now be able to use the COLLATE clause in the following T-SQL statements to define the collation for your VARCHAR or CHAR fields, enhancing your database management capabilities. It’s important to note that only CS collation Latin1_General_100_BIN2_UTF8 and CI collation Latin1_General_100_CI_AS_KS_WS_SC_UTF8 are supported for these statements: CREATE TABLE ALTER TABLE ADD Nullable COLUMN CTAS (Create Table as Select) SELECT INTO Sample Syntax: CREATE TABLE TableName ( ColumnName VARCHAR(length) COLLATE CollationName ); This feature allows you to specify collations on individual columns, offering more granular control than artifact level collation. It ensures precise text data handling by consistently applying collation settings across various operations. It enhances data integrity, minimizes potential collation conflicts, and simplifies the management and querying of text data in mixed-case environments. This release also includes support for the DATABASE_DEFAULT collation, giving you even more flexibility in managing your data. Upgrade your database management with these new capabilities and experience the difference! Learn more about Collate Clause in the collate documentation. Copilot for Data Warehouse Chat (Preview) We are excited to release the chat experience within Copilot for Data Warehouse! You will now see a ‘Copilot’ button in the ribbon which will allow you to interact with your own personalized AI assistant for all data warehousing tasks. In addition to the Copilot code completions and quick actions, chat now enables you to talk to Copilot directly in natural language. You can ask it anything from creating SQL queries from natural language to asking questions about your warehouse, like: ‘How can I monitor query performance’? ‘How should I secure my tables’? ‘Create a relationship between table [x] and table [y]’. The possibilities are endless! For more information on how to use chat within Copilot for Data Warehouse, learn more: How to: Use the Copilot chat pane for Fabric Data Warehouse Browse files with OPENROWSET (Preview) We are enabling you to read the content of parquet and csv files directly from the Fabric DW and SQL endpoints using the OPENROWSET function. The OPENROWSET function enables you to implement new scenarios in Fabric DW: Ad-hoc file browsing and exploration: the OPENROWSET function enables you to read the content of external files without ingesting them in DW. Improving ingestion (ETL/ELT) scenarios: the OPENROWSET function enables you to explore the file schema before the load and inspect rejected rows files in case of errors. With the OPENROWSET function, you can easily read the content of a parquet or csv file and return a set of rows representing the content of the file. You just need to provide the URL of a file that you can read: SELECT TOP 10 * FROM OPENROWSET( BULK 'https://pandemicdatalake.blob.core.windows.net/public/curated/covid-19/bing_covid-19_data/latest/bing_covid-19_data.parquet' ) AS r The OPENROWSET function enables you to use the ‘schema inference’ to automatically identify the columns and their types, or to explicitly specify the schema with the types that match your source schema. You can use the * and /** wildcards in the path to reference multiple files, access the parts of URL using filepath() and filename() function. The overview of supported and unsupported functionalities is shown in the table: Supported Not supported File formats Parquet, CSV Delta, CosmosDB Authentication EntraID passthrough, public storage SAS/SAK, SPN, Managed Identity Storage Azure Blob storage, ADLS OneLake The OPENROWSET function is currently in preview, and more enhancements will be added in the future. Use the BULK INSERT to load your data (Preview) We are happy to announce that we are starting a preview of BULK INSERT statement in Fabric Data Warehouse. Fabric Data Warehouse enables you to use the well-known Transact-SQL BULK INSERT statement for loading data from external files. With the BULK INSERT statement, you can easily load the CSV files from ADLS and specify the options such as row or field terminators: BULK INSERT ecdc_cases FROM 'https://pandemicdatalake.blob.core.windows.net/public/curated/covid-19/ecdc_cases/latest/ecdc_cases.csv' WITH (FIRSTROW = 2, FIELDTERMINATOR = ',', ROWTERMINATOR = '\n'); With the BULK INSERT statement, you can use the same code that you are using in the existing SQL Server warehouses for loading data. Note that Fabric DW provides COPY INTO statement for loading data, and these two statements are almost identical – you can load the same file formats, they have the same performance, etc. In most cases, you will use the COPY INTO statement to load data in the warehouse. But if your warehouse is already in SQL Server and you are planning to migrate it to Fabric DW, there are scenarios where you cannot change the code that uses BULK INSERT and rewrite it to COPY INTO. If you have a large codebase, or external tools or components that use BULK INSERT and you cannot modify their code. To enable easier migration of your warehouses from SQL Server and Azure SQL to Fabric DW, we enabled the same BULK INSERT statement that you are using in the existing warehouses so you can reuse your code with no or minimal changes. BULK INSERT statement provides some options that are not available in the COPY INTO statement such support for loading non-Unicode file sources or FORMATFILE for defining the column mapping schema. If you are using these options in your warehouse solutions, you can use the same code and the file without any changes. Nested common table expression (Generally available) Common Table Expressions (CTEs) can simplify complex queries by deconstructing ordinarily complex queries into reusable blocks enhancing query’s readability and making troubleshooting easier. With this release, Microsoft Fabric Warehouse now supports three types of CTEs in general availability. Standard CTE: This type doesn't reference or define another CTE in its definition. Sequential CTE: This CTE's definition can reference an existing CTE but can't define another CTE. Nested CTE: This CTE’s definition includes defining another CTE. Real-Time Intelligence Configure latency of Eventhouse OneLake availability Eventhouse offers a robust mechanism that intelligently batches incoming data streams into one or more Parquet files, structured for analysis. Batching data streams is important when dealing with trickling data. Writing many small Parquet files into the lake can be inefficient resulting in higher costs and poor performance. Eventhouse's adaptive mechanism can delay writing operations if there isn't enough data to create optimal Parquet files. This ensures Parquet files are optimal in size and adhere to Delta Lake best practices. The Eventhouse adaptive mechanism ensures that the Parquet files are primed for analysis and balances the need for prompt data availability with cost and performance considerations. By default, the write operation can take up to 3 hours or until files of sufficient size (typically 200-256 MB) are created. For scenarios where you desire a lower latency, you can now adjust the delay to a value of between 5 minutes and 3 hours. For example, use the following command to set the delay to 5 minutes: .alter-merge table <TableName> policy mirroring dataformat=parquet with (IsEnabled=true, TargetLatencyInMinutes=5); Please consider this change with caution as adjusting the delay to a shorter period may result in a suboptimal delta table with many small files, which can lead to inefficient query performance. The resultant table in OneLake is read-only and cannot be optimized after creation. To learn more, refer to the Eventhouse OneLake Availability documentation. Real-Time Dashboard & Power BI templates for Eventhouse Monitoring Following the release of Fabric Workspace Monitoring we are happy to announce the release of Real-Time Dashboard and Power BI Templates designed to easily manage your Fabric Eventhouses. The templates were designed to provide an out-of-the-box graphical management experience. With these templates you can easily monitor your query performance and data ingestions. To leverage these reporting templates, only two steps are required: Enable Workspace monitoring - this can be done by turning on a toggle in the Fabric Workspace settings. Once enabled, a Monitoring KQL database that captures performance and ingestion metrics and events is automatically created. Download the templates - go to github and download the templates. Review the Readme file to get a detailed description of how to import the templates into the Fabric experience. You can see a demo in the fabric-toolbox github repo. New Real-Time Dashboard customization features: more control, better usability, and improved performance We’re introducing new customization features that give real-time dashboard authors and viewers more control over visuals, layout, and performance. From managing data series visibility to improving navigation and map behavior, these updates help create a clearer, more efficient dashboard experience. Legend number configuration – Limit the number of data series displayed on load, reducing visual clutter and improving performance. Additional series remain accessible via the legend. Adjustable panel width – Resize the pages pane in both edit and view modes for better navigation, especially when working with long page names. Map centering configuration – Control whether a map visualization maintains its zoom level on refresh or resets based on new data. 4. Crosshair tooltip number configuration – Set a limit on the number of data points displayed in a line or time chart’s tooltip for improved readability. New and improved Data Source Tree in KQL Queryset The enhanced Data Source Tree now shows not only the data source connected to the current query tab but also all other data sources previously connected in your Queryset. This enhancement makes it easier than ever to switch between sources and work with multiple datasets without needing to open new tabs. Multiple data source types are supported, including: Fabric sources (Eventhouse/KQL database). Azure sources (Azure Data Explorer, Application Insights, Log Analytics clusters). Key features: Switch seamlessly: switch between data sources connected to the current Queryset tab without opening a new tab. Create cross-cluster queries: browse through schemas from different clusters and double-click on item names for faster editing and query creation. Search for Data Sources: At the top of the Data Source Explorer pane, you can now use the search bar to quickly find any data source you need. Perform actions on Data Sources items: Contextual action menu is available for data sources items. Data exploration is now available for raw KQL table data We’ve made it even easier to explore your data - no code required – with the Explore Data feature, which is now available in additional places across the UI. What’s new? In addition to the entry point in from every Real-Time Dashboard tile, you can now explore raw KQL table data directly. Real-Time Hub KQL Queryset KQL Database These new entry points make it easier than ever to dive into your data, whether you're starting from a dashboard or working directly with raw tables. Try it out today and uncover new insights effortlessly! Databases Performance Dashboard in SQL database in Microsoft Fabric The Performance Dashboard in SQL database in Microsoft Fabric offers comprehensive diagnostic tools to identify and address performance issues, providing detailed metrics to detect bottlenecks and optimize database performance. It includes features like CPU consumption analysis, user connections monitoring, requests per second tracking, and automatic indexing, making it an essential tool for database administrators and developers. For a detailed guide on troubleshooting and optimizing your SQL database performance, refer to the Speed up your SQL databases with the Performance Dashboard blog. Data Factory Mirroring in Fabric now supports workspace monitoring (Preview) Workspace monitoring is an observability feature in Fabric that enables Fabric developers and admins to access detailed logs and performance metrics for their workspaces. We are thrilled to announce that we have added support for mirrored database operation logs in workspace monitoring to provide a more comprehensive monitoring experience for mirroring. You can leverage these logs to monitor execution and performance of your mirrored database, including data replication, table changes, mirroring status, failures, and replication latency for mirrored databases and tables. To get started, enable monitoring in your workspace. The mirrored database execution logs will automatically be ingested into the MirroredDatabaseTableExecution table in the monitoring KQL database. You can then access a monitoring experience that allows you to: Derive insights on-demand: Query the granular operation logs directly using KQL in the monitoring database to get instant access to the information you need. Build customized monitoring dashboards: Create Real-Time Dashboards or Power BI reports against the data in the monitoring database, tailored to your needs. Set up alerting: Set up alerts based on the logs and metrics you’re tracking. To learn more about this feature, refer to Mirrored database operation logs. New regions available for Mirroring in Fabric To meet the increasing customer demand, Mirroring in Fabric has expanded its support to include three additional regions: Central US, Poland Central, and Italy North. For detailed information about the Fabric regions that support mirroring, please refer to the What is Mirroring in Fabric? documentation. Open mirroring for SAP sources – Simplement and SNP With Simplement and SNP, two more partners have joined our open mirroring partner ecosystem: Open mirroring (Preview) Partner Ecosystem - Microsoft Fabric | Microsoft Learn. Simplement Roundhouse is an SAP-certified solution for data extraction from various SAP source systems including SAP ECC, SAP CRM, SAP CFIN, SAP S/4HANA. It supports near real-time data integration into Microsoft Fabric via open mirroring. For more information, refer to: Simplement integration into open mirroring in Microsoft Fabric. SNP Glue offers robust integration capabilities with SAP systems, including SAP ECC, SAP S/4HANA, and SAP BW. Based on an SAP-certified ABAP Add-On, it facilitates near real-time data ingestion through open mirroring. For more information, refer to: SNP Group. Closing We hope that you enjoy the update! Be sure to join the conversation in the Fabric Community and as always, keep voting on Fabric Ideas to help us determine what to build next. We are looking forward to hearing from you!
Enhancing SQL database in Fabric: share your feedback and shape the future
SQL database in Fabric is currently in preview, and we are actively rolling out improvements as we progress towards general availability (GA). Your insights play a crucial role in this journey. Whether it’s challenges you’ve encountered, features you adore, or any suggestions for enhancement, we want to hear it all—the good, the bad, and the ugly! In this blog post, we will discuss the various forums available for you to provide feedback or get support. We'll also explore best practices for communicating your needs, helping us prioritize our roadmap effectively. Your feedback carries significant weight and directly influences our planning and development efforts. Join us in shaping the future of SQL Database in Fabric by sharing your valuable thoughts and experiences. - Provide contextual feedback in the Query Editor In the ribbon of the SQL database query editor, you'll find a feedback button that allows users to submit product feedback, report issues or blockers, and request new features. This button is conveniently located in various feature inside the query editor, such as the performance dashboard, security management UI, and copilot, among others. This setup enables users to provide detailed, context-specific feedback pertinent to their current tasks and helps product managers efficiently locate and prioritize this feedback on the backend. - Best practices Please ensure you do not provide any private or sensitive information. However, to most effectively help us improve the product, provide as much detail as possible – - Scenario #1 - Reporting an issue: If you encounter a blocker, include: Specific error messages (if any). Steps to reproduce the issue. Any solutions you've already tried. - Example of effective issue reporting: "Copilot sidecar chat was working fine at the beginning, but when I asked how to create a new report, I receive the error message ‘Unable to process request.’ I tried submitting a few more prompts, but I keep on getting the same error message. I’ve tried opening a new chat session or refreshing the browsers, but the issue persists." Ineffective issue reporting: "Feature not working and shows a system error." Why? This lacks details about when and where the issue occurs, making it harder to diagnose. Even better: Open a support case via Fabric and Power BI Support or share your issue in the community forum. This ensures you receive professional assistance promptly and helps our engineering team to gather the necessary information to reproduce and resolve the issue as quickly as possible. - Scenario #2 - Requesting a new feature If you have a feature request, describe: A detailed description of how you’d like the feature to work. Your specific needs or business scenarios where the feature would be beneficial. - Example of effective feature request: "Would it be possible to bulk delete queries? This would be incredibly helpful because currently, with the query auto-save capability, we need to right click a saved query and delete it one by one, which is time-consuming. A feature that allows us to multi-select multiple queries and delete them in bulk would significantly improve efficiency." Ineffective feature request: “Add a feature to help me figure out how to make query performance better” Why? This request is too vague and lacks specific details about the information required to diagnose performance issues and implement optimizations, making it difficult to develop a targeted feature solution. Even better: Submit your idea so others can see and upvote it, helping the team prioritize development based on user demand. - Scenario #3 - Provide general feedback When sharing general feedback or sentiment, it’s important to explain what you like or don't like in detail—and why. This helps the team understand your perspective and make informed decisions about improvements. - Example of effective feedback: "I really appreciate the intuitive interface of the query editor. The color coding and Copilot auto-complete features significantly enhance my productivity by reducing errors and speeding up the query writing process. However, I find the loading time for larger datasets to be quite slow, which affects my workflow." Ineffective feedback: "The editor interface is good, but too slow" Why? This feedback lacks specific details about which aspects of the interface are appreciated and which parts are slow, making it difficult to identify and address the issues effectively. - Wrapping up & next steps To summarize, there are multiple ways you can submit your feedback or get help: Use the editor feedback button directly within the platform. Submit your idea on the designated idea submission page so others can see and upvote it. Open a new support case for immediate assistance. Join discussions or ask questions on the SQL database section of the Microsoft Fabric Community Forum. To ensure your feature requests are effective and actionable, always provide detailed descriptions and specific use cases. This will help the development team understand the context and prioritize accordingly. If you haven’t already started exploring SQL database in Fabric: Begin by signing up for a 60-day Fabric free trial. Enable SQL database in your tenant settings. Learn more via MS Learn: Documentation Learning Path Data Exposed Fabric Databases playlist
AI Ready Apps: build RAG Data pipeline from Azure Blob Storage to SQL Database in Microsoft Fabric within minutes
Microsoft Fabric is a unified, secure, and user-friendly data platform equipped with features necessary for developing enterprise-grade applications with minimal or no coding required. Last year, the platform was enhanced by introducing SQL Database in Fabric, facilitating AI application development within Microsoft Fabric. In a previous blog post, we discussed how to build a chatbot using SQL Database in Fabric. This blog post will provide step-by-step instructions for creating a Retrieval Augmented Generation (RAG) pipeline to prepare your data for AI integration using SQL Database in Microsoft Fabric. For this blog post, we will consider a use case involving Contoso Recruiting Agency, which aims to make more sense of the stored candidates' resumes by creating a searchable database of candidates to match specific skill requirements. RAG pipeline activity workflow Each recruiter uploads the resume (in PDF format) to their designated folder in Azure Blob storage. Whenever a file is uploaded, the RAG pipeline is triggered. The pipeline processes the file by extracting the text content, chunking the text, redacting any Personally Identifiable Information (PII), generating embeddings, and finally storing the embeddings in a SQL database within Fabric as a vector store. These stored embeddings can subsequently be utilized to develop AI applications such as smart search, recommendation engines, chatbots, and other tools. Prerequisite This blog post requires users to bring their own key (BYOK) for AI services, which also means creating these services outside of the Microsoft Fabric platform. Download Git and clone the rag-pipeline repository. Azure subscription: Create a free account. Microsoft Fabric subscription: Create a free trial account. Azure OpenAI access: Apply for access in the desired Azure subscription as needed. Azure OpenAI resource: Deploy an embedding model (e.g. text-emebdding-3-small). Azure AI multi-service resource, specifically, we will be using Document Intelligence and Azure Language Services from this resource. Azure Portal: Create a storage account and assign storage blob data contributor role. Optionally, download Azure Storage Explorer to manage the storage account from your desktop. Optionally, download Visual Studio Code for free and install Azure Functions Core Tools if you plan to edit User Data Functions using Visual Studio Code. Dataset Considering the file formats supported by the Document Intelligence Service we utilize the PDF files from Resume Dataset from Kaggle. Steps Create a workspace named “IntelligentApp”. Create a Lakehouse named “blob_filestorage”. Create SQL Database in Fabric named “datamart”. Navigate to the workspace “IntelligentApp”, click Import - Notebook - From this computer and then import the notebook “cp_azblob_lakehouse.ipynb” from the cloned repository’s notebook folder. Import a Notebook 5. Attach the Lakehouse "blob_filestorage" to "cp_azblob_lakehouse" Notebook. Open the notebook, on the explorer click 'Add sources'. Select, 'Existing data sources'. Select, “blob_filestorage” from OneLake catalog and then click 'Connect'. Attach a Lakehouse to the Notebook 6. Create User Data Function, Navigate to the “IntelligentApp” workspace and click "+New item" Search for “Function” and Select “User data function(s)” Provide the name "file_processor". Click "New function". Add Lakehouse and SQL Database as managed connection(s). On the Functions Explorer screen click "Manage Connections" and then select "+Add Data connection". From the “OneLake catalog” select datamart (SQL database) and then click “Connect”. Repeat the previous step to add blob_filestorage (Lakehouse) as a managed connection. Click "Library management", add the following dependencies (Click “+ Add from PyPI” to add the dependencies). The dependencies are also listed in “/functions/requirements.txt” file of the cloned repository. Ensure you are using fabric-user-data-functions version 0.2.28rc0 or higher. Add User Data Function dependencies In the function editor, replace existing content with the contents of "function\function_app.py" from the cloned repository. Click “Publish” (on the top right) to deploy the function. After the functions are deployed, click “Refresh”. 7. Create a Data pipeline by navigating to the workspace and then clicking on “+ New Item” Search and select “Data pipeline”. Provide the name “blob_ingest_pipeline”. 8. Create a data pipeline storage trigger by clicking “Add trigger (preview)” button and provide the following configuration: Source: Select “Azure Blob Storage events” Storage account: “Connect to existing Azure Blob Storage account” Subscription: Select your Azure subscription Azure Blob Storage account: Select the blob storage account under your subscription. Eventstream name: blob_ingest_stream Create Event stream Click “Next”, to configure the event type and source Event Type(s): Select only the Microsoft.Storage.BlobCreated event. This will ensure that an event is generated each time a new blob object is uploaded. Click "Next" to review the configuration. Then, click "Connect" to connect to the blob storage. A successful connection will be indicated by the status "Successful". Finally, click "Save". On the Set alert screen, under Save location, configure the following settings. Select, “Create a new item”. New item name: blob_activator. Click “Create” to create and save the alert. Create Fabric Activator Now that we have setup the stream, it's time to define the “blob_ingest_pipeline”. Pipeline definition Pipeline can be defined in two ways as outlined below, Import template Templates offer a quick way to begin building data pipelines. Importing a template brings in all required activities for orchestrating a pipeline. To import a template: Navigate to the Home menu of the data pipeline. Click, “Use a template” From the “Pipeline templates” page click “Import template” Import the file “template/ AI-Develop RAG pipeline using SQL database in Fabric.zip” from the cloned repository. Import Data pipeline from template The imported data pipeline is preloaded with all necessary activities, variables, and connectors required for end-to-end orchestration. Important!!! If you are using a template, there is no need to manually add a variable or an activity. Instead, you can proceed directly to configuring values for the variables and each activity parameter in the pipeline, as detailed in the Blank Canvas section. Blank Canvas 1. Establish pipeline variables: Click on the pipeline canvas, select the “Variables” menu, and then click “+ New” to add and configure values for the following variables: Configure pipeline variables Name Type Value Comment fileName String @pipeline()?.TriggerEvent?.FileName container String @pipeline()?.TriggerEvent?.FolderPath source String @pipeline()?.TriggerEvent?.Source cognitiveServiceEndpoint String <<YOUR-MULTI-SERVICE-ACCOUNT-ENDPOINT>> Replace<< YOUR-MULTI-SERVICE-ACCOUNT-ENDPOINT>> with the cognitive Service Endpoint. For example, https://myservices.cognitiveservices.azure.com/ apiKey String <<YOUR-MULTI-SERVICE-ACCOUNT-APIKEY>> Replace <<YOUR-MULTI-SERVICE-ACCOUNT-APIKEY with the apikey of your multi-service account openAIEndpoint String <<YOUR-OPENAI-SERVICE-ENDPOINT >> Replace <<YOUR-OPENAI-SERVICE-ENDPOINT>> with the endpoint of your Azure OpenAI Account. For example, https://myopenaiservice.openai.azure.com/ openAIKey String <<YOUR-OPENAI-APIKEY>> Replace <<YOUR-OPENAI-APIKE>> with the apikey of your Azure OpenAI Account embeddingModel String text-embedding-3-small recepientEmailAddress String <<to-email-address>> recipients email address senderEmailAddress String <<from-email-address>> sender's email address Get multi-services account endpoint 2. Add a "Notebook" activity. The notebook associated with this activity utilizes NotebookUtils to manage file system. During the execution of the notebook, a folder corresponding to the container name will be created if it does not exist. Subsequently, the file will be copied from Azure Blob Storage to the Lakehouse folder. Configure this activity as outlined below: General tab, Name: azureblob_to_lakehouse Settings tab, Notebook: cp_azblob_lakehouse Base parameters Click “+New” to add the following parameters, Name: fileName Type: String Value: @variables('fileName') Name: container Type: String Value: @variables('container') Name: source Type: String Value: @variables('source') Use On Success connector of the activity to link to the subsequent function (Extract Text) activity. 3. Add a “Functions activity”. The function “extract_text” associated with this activity uses Azure AI Document Intelligence service to extract the "text" content from the file copied into the Lakehouse by the previous activity. Configure this activity as outlined below: General tab, Name: Extract Text Settings tab, Type: Fabric user data functions Connection: Sign-in (if not already) using your workspace credentials. Workspace: IntelligentApp (default selected) User data functions: file_processor Function: extract_text Parameters, Name: filePath Type: str Value: @activity('azureblob_to_lakehouse').output.result.exitValue Name: cognitiveServicesEndpoint Type: str Value: @variables('cognitiveServiceEndpoint') Name: apiKey Type: str Value: @variables('apiKey') Use On Completion connector of the activity to link to the subsequent If Condition (Text Extraction Results) activity. 4. Add an “If Conditions” activity to verify the success of the text extraction in the previous step. If the extraction was unsuccessful, an email would be sent to the configured recipient, and the pipeline would be terminated. Configure this activity as outlined below, General tab, Name: Text Extraction Results Activities tab, Expression: @empty(activity('Extract Text').error) Case: False, edit the “false” condition using the edit (pencil) icon, and add the following activities, “Office 365 Outlook” activity: To send alert emails. General Tab, Name: Text Extraction Failure Email Alert Settings Tab, Signed in as: Sign- in (if not already) using the credentials as those of your workspace. To: @variables('recepientEmailAddress') Subject: Text Extraction Error Body: <p>@{replace(string(activity('Extract Text').error.message), '\','')}</p> Advanced, From: @variables('senderEmailAddress') Importance: High Use On Success connector of the activity to link to the subsequent Fail activity. “Fail” activity: To terminate the pipeline General tab, Name: Text Extraction Process Failure Settings tab, Fail message: @{replace(string(activity('Extract Text').error), '\','')} Error code: @{activity('Extract Text').statuscode} Return to the main canvas by clicking the pipeline name (blob_ingest_pipeline) and use the On Success connector of the “If Condition” activity to link to the subsequent Function (generate chunks) activity. 5. Add a “Function” activity. The function “chunk_text” associated with this activity uses the tiktoken tokenizer to “generate chunks” for the text extracted by the previous activity. Configure this activity as outlined below: General tab, Name: Generate Chunks Settings tab, Type: Fabric user data functions Connection: If not already, sign in using the credentials that have complete access to your workspace. Workspace: IntelligentApp (default selected) User data functions: file_processor Function: chunk_text Parameters, Name: text Type: str Value: @activity('Extract Text').output.output Name: maxToken Type: int Value: 500 Name: encoding Type: str Value: cl100k_base Use On Success connector of the activity to link to the subsequent Function (Redact PII Data) activity. 6. Add a “Function” activity. The function “redact_text” associated with this activity uses the Azure AI Language service to “Redact PII Data” for the chunks generated by the preceding activity. The chunking of text is done prior to redaction to comply with the service limits requirements for the PII detection feature. Configure this activity as outlined below: General tab, Name: Redact PII Data Settings tab, Type: Fabric user data functions Connection: Sign-in (if not already) using the credentials that have complete access to your workspace. Workspace: IntelligentApp (default selected) User data functions: file_processor Function: redact_text Parameters, Name: text Type: list Value: @activity('Generate Chunks').output.output Name: cognitiveServicesEndpoint Type: str Value: @variables('cognitiveServiceEndpoint') Name: apiKey Type: str Value: @variables('apiKey') Configure a User Defined Function Use On Completion connector of the activity to link to the subsequent If Condition (PII Redaction Results) activity. 7. Add an “If Conditions” activity to verify the success of the PII redaction in the previous step. If the redaction was unsuccessful, an email would be sent to the configured recipient, and the pipeline would be terminated. Configure this activity as outlined below, General tab, Name: PII Reaction Results Activities tab, Expression: @empty(activity('Redact PII Data').error) Case: False, edit the “false” condition using the pencil icon, and add the following activities: “Office 365 Outlook” activity: To send alert emails. General tab, Name: Redaction Failure Email Alert Settings tab, Signed in as: Sign-in (if not already) using the credentials as those of your workspace. To: @variables('recepientEmailAddress') Subject: Text Redaction Error Body: <p>@{replace(string(activity('Redact PII Data').error.message), '\','')} </p> Advanced, From: @variables('senderEmailAddress') Importance: High Use On Success connector of the activity to link to the subsequent Fail activity. “Fail” activity: To terminate the pipeline General tab, Name: Text Redaction Process Failure Settings tab, Fail message: @{replace(string(activity('Redact PII Data').error), '\','')} Error code: @{activity('Redact PII Data').statuscode} Return to the main canvas and use the On Success connector of the “If Condition” activity to link to the subsequent Function (Generate Embeddings) activity. 8. Add a “Function” activity. The function “generate_embeddings” associated with this activity uses Azure Open AI Service embedding model to convert the redacted chunks into embeddings. Configure this activity as outlined below: General tab, Name: Generate Embeddings Settings tab, Type: Fabric user data functions Connection: Sign-in (if not already) using the credentials that have complete access to your workspace. Workspace: IntelligentApp (default selected) User data functions: file_processor Function: generate_embeddings Parameters, Name: text Type: list Value: @activity('Redact PII Data').output.output Name: openAIServiceEndpoint Type: str Value: @variables('openAIEndpoint') Name: embeddingModel Type: str Value: @variables('embeddingModel') Name: openAIKey Type: str Value: @variables('openAIKey') Name: fileName Type: str Value: @variables('fileName') Use On Completion connector of the activity to link to the subsequent If Condition (Generate Embeddings Results) activity. 9. Add an “If Conditions” activity to verify the success of the Generate Embeddings in the previous step. If the embeddings generation were unsuccessful, an email would be sent to the configured recipient, and the pipeline would be terminated. Configure this activity as outlined below: General tab, Name: Generate Embeddings Results Activities tab, Expression: @empty(activity('Generate Embeddings').error) Case: False, edit the “false” condition using the pencil icon, and add the following activities: “Office 365 Outlook” activity: To send alert emails. General tab, Name: Generate Embeddings Failure Email Alert Settings tab, Signed in as: Sign-in (if not already) using the credentials as those of your workspace. To: @variables('recepientEmailAddress') Subject: Generate Embeddings Error Body: <p>@{replace(string(activity('Generate Embeddings').error.message)</p> Advanced, From: @variables('senderEmailAddress') Importance: High Use On Success connector of the activity to link to the subsequent Fail activity. “Fail” activity: To terminate the pipeline General tab, Name: Generate Embeddings Processing Failure Settings tab, Fail message: @{replace(string(activity('Generate Embeddings').error), '\','')} Error code: @{activity('Generate Embeddings').statuscode} Return to the main canvas and use On Success connector of the “If Condition” activity to link to the subsequent Function (Create Database Objects) activity. 10. Add a “Function” activity. The function “create_table” associated with this activity executes a SQL command to create a documents table within the previously created “datamart”, SQL database. Configure this activity as outlined below: General tab, Name: Create Database Objects Settings tab, Type: Fabric user data functions Connection: Sign-in (if not already) using the credentials that have complete access to your workspace. Workspace: IntelligentApp (default selected) User data functions: file_processor Function: create_table Use On Success connector of the activity to link to the subsequent Function (Save Data) activity. 11. Add a “Function” activity. The function “insert_data” associated with this activity executes a SQL command to bulk insert rows in the documents table created in the previous activity. Configure this activity as outlined below: General tab, Name: Save Data Settings tab, Type: Fabric user data functions Connection: Sign-in (if not already) using the credentials that have complete access to your workspace. Workspace: IntelligentApp (default selected) User data functions: file_processor Function: insert_data Parameters, Name: data Type: list Value: @activity('Generate Embeddings').output.output Troubleshooting When adding a Python library from PyPI to User Data Functions, you might notice an error, such as a wiggly line under the library name (e.g., "azure-ai-textanalytics"), like a spelling mistake. Users should ensure the library name is spelled correctly and then ignore the error by tabbing out to the Version dropdown and selecting the correct version, this transient error should resolve itself. The imported pipeline reportedly doesn't seem to preload with the parameter values. For each activity in the pipeline, ensure that the parameter values are provided and correct. Refer to the Blank Canvas section for the required parameters and their values. Execute pipeline (pipeline in action) It’s now time to put everything we have done so far into perspective and see the pipeline in action. Upload a PDF file, Use the Azure Storage Explorer or alternatively Azure Portal and create a Blob container named “resume”. Upload a PDF file from the Kaggle dataset. Create a container and upload a PDF file Review pipeline execution From the pipeline’s “Run” menu, select “View run history” and select the recent pipeline run. In the details view, check to see if the status is “Succeeded”. In case of a Failure, try to “Rerun” the pipeline using the rerun option. Pipeline activity execution Review Lakehouse A folder with the same name as that of the container (resume) is created. The PDF file is copied from Azure Blob Storage to the Lakehouse files. PDF file copied to Lakehouse Review database The document table should be automatically created by the pipeline. Redacted chunk data and the embeddings stored in the documents table. Save data in SQL Database in Fabric Conclusion We hope this blog post has provided useful insights for preparing your data for AI applications. Your feedback is important, and we look forward to seeing how you use this information to develop innovative AI solutions. Would also like to express our sincere gratitude to Aaron Saidi for the invaluable contribution towards reviewing this blog post. Thank you, Aaron, for your support. Submit your feedback on Fabric Ideas and join the conversation on the Fabric Community.
