Microsoft Fabric July 2024 Update
Welcome to the July 2024 update. Here are a few, select highlights of the many we have for Fabric. Creating and managing Git branches & connected workspaces with Git integration just got with the latest enhancements to Fabric Git integration. You now have the capability to perform restore-in-place of a warehouse in Microsoft Fabric through the Microsoft Fabric Warehouse Editor. Real-time Dashboards now support ultra-low refresh rates of just 1 and 10 seconds, ensuring you always have the freshest and most accurate data at your fingertips. There is much more to explore, please continue to read on. European Fabric Community Conference Join us at Europe’s first Fabric Community Conference, the ultimate Power BI, Fabric, SQL & AI learning event in Stockholm, Sweden from September 24 -27, 2024. With 120 sessions, daily keynotes, 10 pre-conference workshops, an expo hall with community lounge, and “ask the expert” area, the conference offers a rich learning experience you don’t want to miss. This is a unique opportunity to meet the Microsoft teams building these products, customers betting their business on them, and partners at the forefront of deployment and adoption. Register today using code MSCUST for an exclusive discount on top of early bird pricing! Get certified on Fabric! We’d like to thank the thousands of you who completed the Fabric AI and 30 Days to Learn It Skills Challenge and earned a discount voucher for Exam DP-600 which leads to the Fabric Analytics Engineer Associate certification. If you earned a discount voucher, you can find redemption instructions in your email. We recommend that you schedule your exam promptly, before your discount voucher expires. If you need a little more help with exam prep, visit the Fabric Career Hub which has expert-led training, exam crams, practice tests and more. Fabric Influencers Spotlight Check out our latest initiative, the Fabric Influencers Spotlight. Each month, we'll be highlighting some of the great blog, videos presentations and other contributions submitted by members of Microsoft MVP & Fabric Super User communities that cover the Fabric Platform, Data Engineering & Data Science in Fabric, Data Warehousing, Power BI, Real-Time Intelligence, Data Integration, Fabric Administration & Governance, Databases and Learning. Governance, Databases, and Learning. Attention Power BI users! If you are accessing Power BI on a web browser version older than Chrome 94, Edge 94, Safari 16.4, Firefox 93, or equivalent, you need upgrade your web browser to a newer version by August 31, 2024. Using an outdated browser version after this date, may prevent you from accessing features in Power BI. Contents Reporting Customize your reference layers in the Azure Maps visual Modeling Announcing general availability ofenhanced row-level security editor in Power BI Desktop DAX query view is available in live connect Add or update multiple measures in DAX query view Data connectivity Certified connector updates Service Storytelling in PowerPoint – New export to PowerPoint dialog Developers New update for field parameter feature for custom visuals Power BI enhanced report format (PBIR) update Visualizations New visuals in AppSource Linear Gauge by Powerviz Drill Down Map PRO by ZoomCharts PowerGantt Chart by Nova Silva Advanced Geospatial Analytics Made Simple with Icon Map Pro for Power BI Paginated Reports Bind to Gateway API support for paginated reports Parameters, Header/Footer and much more in the web authoring experience for paginated reports (Preview) Power BI Report Server Power BI Report Server key in Fabric Capacities Core General Availability ofFabric Private Links General Availability ofManaged Private Endpoint General Availability ofTrusted workspace access CI/CD – GitHub integration for source control New Branching Capabilities in Fabric Git Integration Data Warehouse Announcing the General availability of time travel and 30 days of data retention in Fabric Warehouse Seamless Data Recovery through Warehouse Restoration within Fabric Query Editor Alter Table, Add Nullable columns Data Engineering Capacity Pools for Spark in Microsoft Fabric for Data Engineering and Data Science Environment Resources folder MSSparkUtils new API Lakehouse Schemas (Public Preview): A New Way to Organize Your Tables Real-time Intelligence Real-Time Dashboards 1s and 10s refresh rate KQL Database - .update command GA Data Factory Dataflow Gen2 Global view in Manage connections Support for editing Navigation steps On-Premises support for Fast Copy Certified connector updates Data pipeline Edit JSON code for Data pipelines New connectors Snowflake storage integration Use existing connections from the OneLake Datahub integration Check out or Monthly Update video here: Reporting Customize your reference layers in the Azure Maps visual Recently, we've brought you a lot of improvements to the Azure Maps visual, including reference layer support for a variety of new data formats. This month, we're excited to announce several more improvements to reference layers: CSV support, new customization options, and dynamic URL sources! First, the Azure Maps visual now supports CSV files as data sources for reference layers! Just as you can already use GeoJSON, Shapefiles, WKT, and KML files, you can now upload a CSV file instead in the reference layer section of the formatting pane. You can also now format reference layer shapes from within the formatting pane. Previously, Azure Maps required you to define the color and width of points, lines, and polygons from within your reference layer files. Otherwise, these shapes would be drawn on your maps with the default colors and formatting. This requirement brought additional complexity to working with your reference layers in Power BI, since the files needed more than just the data you intended to visualize. Now, we've added these standard formatting settings to each type of object in your reference layers in the formatting pane, so you can customize them directly from within Power BI! Lastly, for those of you who need your reference layers to change with time or other data-bound conditions, you can now provide a dynamic URL using conditional formatting! This allows you to set custom logic to determine the reference layer URL the Azure Maps visual will use. For example, you can load in different reference layers based on the categories selected by a slicer, like to visualize performance of different product lines over the same geography. We're still hard at work bringing you more improvements to the Azure Maps visual -- let us know what you think about these new capabilities and keep an eye out for further updates in the future! Check out a Reporting demo here: https://youtu.be/WDpWMsOuqCM?si=JFS9W93Z7AABBGBW&t=95 Modeling Announcing general availability of enhanced row-level security editor in Power BI Desktop We are excited to announce the general availability of the enhanced row-level security editor in Power BI Desktop! With this editor, you can quickly and easily create row-level security roles and filters. Simply choose ‘Manage roles’ from the ribbon to open the editor. By default, this will open an easy-to-use drop-down interface for creating and editing security roles all without having to write any DAX! If you prefer using DAX or need it for your filter definitions, you can switch to use the DAX editor to define your role. This is a rich editor complete with autocomplete for formulas (intellisense). It also allows you to easily verify the validity of your DAX expressions by selecting the check button and revert any changes by selecting the X button. At any point you can also switch back to the default editor by selecting ‘Switch to default editor’. All changes made in either editor persist when switching interfaces when possible, giving you maximum flexibility as you create your row-level security roles. Learn more information about Power BI row-level security including limitations in our documentation. Check out Modeling demos here: https://youtu.be/WDpWMsOuqCM?si=WsmWSowVUPUlD78J&t=368 DAX query view is available in live connect We heard your request and have added the ability to use DAX query view while live connected to a published semantic model. With this release you can write DAX queries with DAX query view when live connected to a published semantic model in Power BI Desktop! This includes live connecting to the amazing Direct Lake semantic models created in Microsoft Fabric. Live connects to your published Direct Lake, import, DirectQuery, or composite semantic model in Desktop and use the DAX query view to quickly view data without having to create any visuals. Use quick queries to have a DAX query generated for you from any table, column, or measure, and Copilot can help you as you write your DAX queries. This can be helpful to further your analysis beyond just report authoring even when you are using a published semantic model managed by someone else. Try it out today and learn more about DAX query view. Add or update multiple measures in DAX query view Another popular ask for DAX query view is also now available – add or update the model with multiple measure changes. In a DAX query, you can use the DEFINE syntax to add a measure. These DAX query scoped measures are helpful for authoring DAX formulas and trying them out with different groups by columns before adding them to the model. In DAX query view, we made it easy to then add these measures to the model by clicking the text between the lines above the DEFINE MEASURE. DAX queries also allow you to define many measures at once, so it can be tedious to click each one. Now this task is very easy with the option to update the model with a single click for all measures! This can be handy to quickly format all your DAX formulas at once. In DAX query view, right-click in the Data pane and choose Quick queries > Define all measures in this model Click the Format query button in the ribbon. Click Update model with changes button. And you are done! Try it out today and let us know what you think by using the Share feedback button. Data connectivity Certified connector updates We’re pleased to announce the new and updated connectors in this release: [New] Windsor [Update] SingleStore [Update] Smartsheet [New] BuildingConnected Are you interested in creating your own connector and publishing it for your customers? Learn more about the Power Query SDK and the Connector Certification program. Service Storytelling in PowerPoint – New export to PowerPoint dialog To reduce complexity and drive more clarity we merged the two options to export to PowerPoint into a single dialog. You can now choose between embedding live data using Power BI add-in for PowerPoint and exporting the report as images from the same dialog. Check out a Service demo here: Developers New update for field parameter feature for custom visuals sourceFieldParameters is a new property in DataViewMetadataColumn that identifies whether a query field results from a field parameter resolution. If a single field can originate from multiple field parameters, this property will list all the related field parameters. This new update is available with API v5.10.0. Power BI enhanced report format (PBIR) update The following previously announced limitations of the PBIR format have been resolved: Can't be exported to PPTX or PDF. Can't be included in Subscriptions Mobile layouts aren't applied. Can't be utilized in Power BI Embedded. For further information regarding PBIR, please refer to the documentation. Visualizations New visuals in AppSource Icon Map Pro StackedTrends Visual Smart Grid-Map Multilevel Matrix Xerppa Sankey Diagram Waffle Chart Maker Waterfall Chart SPC_Visual Linear Gauge by Powerviz Powerviz Linear Gauge is an advanced visual that is used to display the progress against set targets on a linear scale, with an axis displaying a range of values or percentages. The Linear Gauge quickly conveys the status or progress of a task or value being measured. Key Features: Gauge Styles: Four different gauges including Linear, Bar in Bar, Cylinder, Thermometer, and customization option. Templates: Select from pre-made templates or customize your own. Scale: Select an absolute or percentage scale, with a customizable min-max range. Targets: Set a custom target or apply a target using a value field. Data Colors: 30+ color palettes available. Band: 30+ color palettes and customization options. Labels: Improve readability with labels. Small Multiples: Divide visuals based on fields. Ranking: Filter Top/Bottom N shows remaining as “Others”. Other features included are fill pattern, annotation, grid view, show condition, and more. Business Use Cases: Sales Performance Tracking, Project Milestone Monitoring, Financial KPI Analysis, 🔗 Try Word Cloud Visual for FREE from AppSource 📊 Check out all features of the visual: Demo file 📃 Step-by-step instructions: Documentation 💡 YouTube Video: Video Link 📍 Learn more about visuals: https://powerviz.ai/ ✅ Follow Powerviz: https://lnkd.in/gN_9Sa6U Drill Down Map PRO by ZoomCharts When visualizing data with geographic coordinates, what better way to do it than literally placing it on a map? That’s why map charts are a growingly popular visualization type in Power BI reports, and the Drill Down Map PRO custom visual by ZoomCharts expands on the capabilities of map charts. Node Clustering: Multiple nearby nodes can create clusters and even display the values as pie charts. Simply zoom in to drill down. Base Layer Customization: Choose between AzureMaps or any custom tileserver, use your own images as the base layer, or disable it entirely. Custom Shape Layers: Enable up to 10 individually customizable shape layers. Use preset shapes or import your own KML/GeoJSON files. Conditional Formatting: Automatically apply color fill to each area by comparing their values against other shapes or by using each shape’s own reference value. And More: Paginated tooltip s, custom tooltip fields, auras, node images, lasso tool. Drill Down Map PRO works incredibly well with other visuals by dynamically cross-filtering data, enabling you to build even more insightful and user-friendly Power BI reports. 🌐Get Drill Down Map PRO on AppSource Product Page | Documentation | LinkedIn | Community PowerGantt Chart by Nova Silva We are excited to receive ongoing feedback from you and want to express our gratitude for your valuable contributions. Your insights help us make significant enhancements to our visuals. In the latest releases of the PowerGantt Chart, we have added many new features based on your feedback. These include the ability to show incomplete tasks and display progress as a separate column. We've also included links in the additional columns and enhanced their formatting options. You can now change the milestone shapes and wrap text in columns for better readability. New: Progress column URL links Milestone shapes Milestone labels Dynamic zoom slider period Vertical grid lines Expand & Collapse all Additionally, we've added the option to preset the zoom slider, expand and collapse all hierarchy elements, and add milestone labels. To further improve your experience, we've enabled the display and formatting of vertical grid lines. These updates are designed to provide you with more flexibility and control over your project visuals. We appreciate your continued support and look forward to receiving more of your valuable feedback. Together, we can keep enhancing PowerGantt Chart to meet your needs. Try the PowerGantt Chart for FREE now on your own project data by downloading it from the AppSource. Questions or remarks? Visit us at: https://visuals.novasilva.com/. Advanced Geospatial Analytics Made Simple with Icon Map Pro for Power BI Icon Map Pro, the new professional version of Icon Map, has been built from the ground up with an extensive set of new features and a simplified interface. This tool offers a robust solution for visualising and analysing geospatial data within Power BI. It is designed for data analysts, GIS specialists, and business intelligence professionals, addressing the need for seamless integration of geographic insights into BI dashboards. Users can effortlessly transform complex geospatial data into actionable visuals, enhancing decision-making and strategic planning with the intuitive low/no-code Power BI interface. Visit the supporting website iconmappro.com for: Extensive documentation Supporting videos Professional help and support Try Icon Map Pro for free from Microsoft AppSource Paginated Reports Bind to Gateway API support for paginated reports You can now bind your paginated reports to gateways with a REST API. This will allow paginated reports to connect to on-prem gateways without requiring users to go to the UI in the Power BI service and bind the report to the specified gateway. Learn more about the Bind to Gateway API for paginated reports. Parameters, Header/Footer and much more in the web authoring experience for paginated reports (Preview) We have introduced a new experience to web author paginated reports! It’s not just an update to the look and feel, but we’ve also introduced a host of new capabilities. You can now define parameters, headers, footers, page numbers in your web authored reports. The update will be rolling out in the coming weeks. Please check again in a couple of weeks, if you don’t see it. Once you select the fields, they will appear in the “Editor” along with a “Preview” of the report with sample data. You can move the table in the “Editor” and the preview will reflect the change as well. You can choose to add a header, footer, textbox or image. To add a footer, choose “Insert” and select “Footer”. Add a Text box, Image, page number and/or execution time. You can choose to display the footer and header on the first and last pages as well. Exit the footer by clicking out, once you are in the body of the report, you can “Create parameter”. By creating a parameter, you can create a report that requires the viewer of the report to enter one or more values to view the report. When you “Create parameter”, you can see the parameter at the top of the “Preview” portion of the screen. You can show/hide the parameter by clicking on the “Parameters” on the preview ribbon. When you save the report, it is saved with the parameter defined and the viewer of the report must specify the parameters to view the report. You can now share the report with others! This is a preview feature and will not be available on Sovereign clouds until we are generally available. Check out a Paginated Reports demo here: Power BI Report Server Power BI Report Server key in Fabric Capacities Power BI Report Server is now included with F64+ Reserved Instance purchases. It continues to be available with SQL Server Enterprise core licenses with software assurance. You can get the PBIRS key in the “Fabric Capacity” tab under “Capacity Settings” in the admin portal. Learn more about ways to get the Power BI report server key. Check out a Power BI Report Server demo here: Core General Availability of Fabric Private Links Private links for Fabric tenant secures inbound access to Fabric from select virtual networks (VNets) and allow you to block access from the internet. Tw With this feature enabled, you can secure connectivity to Fabric Onelake and experiences like Data Warehouse, Data Engineering, Data Science and Data Factory. In addition to the above, we also announced private link support for Eventhouses. To learn more about Azure Private Link Support for Microsoft Fabric. Check out the blog post to learn more - About private Links for secure access to Fabric – Microsoft Fabric | Microsoft Learn General Availability of Managed Private Endpoint Managed private endpoints provide secure connectivity from Fabric to data sources that are behind a firewall or not accessible from the public internet. Managed Private Endpoints currently enable Fabric Data Engineering items to access data sources securely without exposing them to the public network or requiring complex network configurations. Managed private endpoints are supported for various data sources, such as Azure Storage, Azure SQL Database, and many others- the most recent addition being Azure Event Hub and Azure IOT Hub. We will also expand to more workloads as we go. To learn more about Managed Private Endpoints and supported data sources see Overview of managed private endpoints for Microsoft Fabric – Microsoft Fabric | Microsoft Learn General Availability of Trusted workspace access Trusted workspace access allows seamless and secure access to firewall enabled Azure storage accounts. It is designed to help you securely and easily access data stored in Storage accounts from Fabric workspaces, without compromising on performance or functionality. This feature extends the power and flexibility of OneLake shortcuts to work with data in protected storage accounts in place without compromising on security. You can also use this capability with Data pipelines and the COPY INTO feature of Fabric warehouses to ingest data securely and easily into Fabric workspaces. Trusted workspace access is based on the concept of workspace identity, which is a unique identity that can be associated with workspaces that are in Fabric capacities. When you create a workspace identity, Fabric creates a service principal in Microsoft Entra ID to represent the identity. A workspace identity enables OneLake shortcuts, data pipelines, and warehouse Copy INTO command to access Storage accounts that have resource instance rules configured. Resource instance rules are a way to grant access to specific resources based on the workspace identity or managed identity. You can create resource instance rules by deploying an ARM template with the resource instance rule details. To get started with this feature and to learn about limitations, see Trusted workspace access in Microsoft Fabric – Microsoft Fabric | Microsoft Learn CI/CD – GitHub integration for source control Organizations who use GitHub or GitHub Enterprise as their source control tool, can now seamlessly integrate with Fabric by connecting Fabric workspaces to GitHub and backup their work. This integration will enjoy the same functionalities that the integration with Azure DevOps offers in Fabric today. Learn more about the new release. New Branching Capabilities in Fabric Git Integration Creating and managing Git branches & connected workspaces with Git integration just got with the latest enhancements to Fabric Git integration. The updated source control pane in every Git-connected workspace now includes these features: Branch Out to a New Workspace – Effortlessly create a new isolated environment, allowing developers to work independently without impacting the team’s shared workspace. This makes it quick and easy to get started on new tasks. Explore related workspaces – Easily access any workspace connected to the same Git repository and folder, enabling seamless work across different workspaces and branches directly from your current workspace. These new capabilities come with a redesigned source control pane, which now organizes all features into a couple of streamlined tabs. For more details and to explore these options, read the full update here. Check out Core demos here: Data Warehouse Announcing the General availability of time travel and 30 days of data retention in Fabric Warehouse In the rapidly evolving world of generative artificial intelligence, historical data plays a significant role in influencing the decision-making process and shaping organizational strategies. Data retention within data warehouse refers to the practice of preserving and managing previous iterations of the data encompassing any inserts, updates or deletes made to the warehouse for a specified period. As a quick response to the valuable customer feedback, we are thrilled to extend the data retention period to 30 calendar days from 7 calendar days and announce the General availability of Time travel at the T-SQL Statement level. Extending the data retention period to 30 calendar days opens new avenues for exploration, by leveraging Fabric warehouse features - Time travel at T-SQL statement level, Time travel with table clone, and restore in place. In the data-centric world, data retention is pivotal in modern data management enabling organizations to utilize historical data effectively for analysis, reporting, compliance, and strategic decision-making. By the utilization of data retention, businesses can extract valuable insights from the past to drive success both now and in the future. In today's rapidly evolving digital landscape, adaptability is crucial. Our decision to extend the data retention period from 7 to 30 calendar days underscores our commitment to agility and customer-centricity. Come embrace the data retention of 30 calendar days to obtain deeper insights and stay ahead of the curve. Seamless Data Recovery through Warehouse Restoration within Fabric Query Editor We are thrilled to announce the capability to perform restore-in-place of a warehouse in Microsoft Fabric through the Fabric Warehouse Editor. In today's rapidly evolving data management landscape, maintaining the resilience and continuity of your data infrastructure is essential. Unplanned system failures and scheduled maintenance alike demand the ability to restore data warehouses swiftly and seamlessly. This capability is no longer just a feature – it's a critical necessity in modern analytics environments. A quick and reliable data warehouse recovery solution is indispensable, not only to protect against data corruption but also to ensure business continuity. From the moment the warehouse item is created, system restore points are created at least every 8 hours. You can create any number of user-defined restore points aligned with your specific business or organizational recovery strategy. Both system-created and user-defined restore points come with a retention period of 30 calendar days. You can view all restore points in warehouse settings. To create user-defined restore points, go to Warehouse Settings -> Restore points and Add a restore point by providing Name and Description. Then click on context menu action to restore the warehouse item back to that point. To learn more about Restore in-place of a warehouse, please refer to documentation and announcement Alter Table, Add Nullable columns We are excited to announce the General Availability of Alter Table – Add Nullable columns! In the ever-evolving data landscape of data organizations, schemas are shifting and changing to keep up with the influx of new data. Whether your schema modifications are few and far between, or a regular occurrence that constantly needs to adapt to changing requirements, we have you covered. Our goal is to ensure that customers have everything they need for a seamless warehousing experience, and we continue to strive towards ensuring our TSQL surface area meets the needs of our customers. Here are some example syntaxes to get started with ALTER TABLE functionality. Note that today we only support the ability to add nullable columns to the Warehouse table. ALTER TABLE [Contoso].[dbo].[Customers] ADD [Customer_County] varchar(30) Learn more here. Check out our Data Warehouse demo here: Data Engineering Capacity Pools for Spark in Microsoft Fabric for Data Engineering and Data Science We’re excited to introduce the public preview of Capacity Pools, designed specifically for Fabric Data Engineering and Data Science workloads. With this new feature, capacity administrators can create custom pools using the capacity spark settings option. Here’s how: Navigate to the Capacity Settings Page: Go to the Admin Poral -> Capacity Settings Page. Navigate to the “Data Engineering/Science Settings”. Create New Pools: Within the Spark Settings, you’ll find the option to create new pools. These pools can be sized based on the Max Burst Cores limit for Spark your Fabric Capacity SKU. Availability Across Workspaces: Once configured and saved, these pools become available for all workspaces connected to the Fabric capacity. Workspace administrators will see the newly created Capacity pool listed among their available options. Data Engineers or Data Scientists using Environments will also see the newly created Capacity pool listed among their available compute options. With these options Capacity admins gain additional controls to manage compute governance for Spark compute in Microsoft Fabric. You can create pools for specific workspaces and even disable workspace-level customization. Environment Resources folder We are thrilled to announce the launch of our new Environment Resources Folder, a shared repository designed to streamline collaboration across multiple notebooks! In the existing notebook the resources folder provides a per notebook resources to store small sized dataset, sample files, code modules and configuration files etc. It provides convenient browsing, folder structure management, upload/download, drag and drop and simple file interactions. In the recent release, the resources functionality is extended to the Environment, you can find the Resources tab inside the environment and have the full operations to manage the resource files here, and these files can be shared across multiple notebooks once the notebook is attached to the current environment, you can leverage this enhancement to better collaborate with the team on large project with shared resources. In the Notebook page, you can easily find a second root folder under Resources inherited from the attached environment, and you operate on the files/folders just same with the Built-in resources folder, you can easily drag & drop different files into the code cell, and we’ll generate the executable code snippet to help you consume the file easily. The Environment resource path will be automatically mounted to the notebook cluster, you can use the relative path /env to access the environment resources. For more detailed usage please find the public document here: How to use notebooks - Microsoft Fabric | Microsoft Learn MSSparkUtils new API We are excited to announce that mssparkutils.runtime.context is available to all now! With the new API you can get the context information of the current live session, including the notebook name, default lakehouse, workspace info, if it's a pipeline run, etc. And the context information is very useful when you want to have more advanced implementation logic inside notebook. For more information, see Microsoft Spark utilities documentation. Lakehouse Schemas (Public Preview): A New Way to Organize Your Tables We are excited to announce the release of lakehouse schemas. It is a new feature that allows you to group and manage your tables in logical namespaces. With lakehouse schemas, you can organize your tables into categories, such as business domains, projects, or teams. You can reference multiple tables at once using schema shortcuts, instead of shortcutting each table individually. Apply security policies to schemas for more granular access control, such as granting or revoking permissions to a group of tables at once. You can also cross-reference and join tables from different workspaces using Spark SQL, without having to copy or move data. Lakehouse schemas is a new feature that enables logical grouping and management of tables in namespaces. With lakehouse schemas, users can organize tables by categories, use schema shortcuts, apply security policies, and cross-reference tables from different workspaces. The feature is available for public preview. For more detailed information find the public documentation here: Lakehouse schemas (Preview) - Microsoft Fabric | Microsoft Learn Check out our Data Engineering demo here: Real-time Intelligence Real-Time Dashboards 1s and 10s refresh rate Real-time Dashboards now support ultra-low refresh rates of just 1 and 10 seconds, ensuring you always have the freshest and most accurate data at your fingertips. With the auto-refresh feature, your dashboards update automatically, eliminating the need to manually reload the page or click a refresh button. Learn more about enabling Real-Time Dashboards auto refresh and setting the refresh rate Create a Real-Time Dashboard (preview) - Microsoft Fabric | Microsoft Learn KQL Database - .update command GA Eventhouse and KQL databases are optimized for append ingestion. In recent years, we've introduced the .delete command, allowing you to selectively delete records. In February, we introduced the .update command in public preview. This command allows you to update records by deleting existing records and appending new ones in a single transaction. The .update command is now Generally Available (GA)! We encourage you to go through the many examples of the online documentation page to familiarize yourself with the syntax. Data Factory Dataflow Gen2 Global view in Manage connections Manage connections is the feature that allows you to see and edit all the connections in your Dataflow and data sources that don’t have any linked connection so you can create a new one. Starting today, we’re adding a new Global view to the Manage Connections that will also allow you to see all the available connections in your Fabric environment so you can modify them or delete them without ever having to leave the Dataflow experience. Previously, if you wanted to delete a connection you would’ve had to go into the Manage Connections and Gateways portal to do so, but now you have an alternative way without ever having to leave the Dataflow experience. You can learn more about the Manage Connections feature from the official documentation article. Support for editing Navigation steps Whenever you connect to a data source that showcases a Navigation dialog you end up with a query that has a Source step and either one or a series of Navigation steps. For example, when you use the Lakehouse connector you can navigate to the workspaces that you have access to, then the Lakehouse within them and the objects found within your Lakehouse so you can navigate to them. Another example is when you connect to an OData service like Northwind that displays a list of tables available to you. What if you wanted to connect to a different object within your Lakehouse or perhaps a completely different Lakehouse? Introducing the new experience to edit navigation steps within DataflowYou can find it inside of the Applied steps section of the Query settings pane by double clicking a Navigation step or clicking on the gear icon. It is similar to the way that a navigation step behaves in Power Query Desktop (in Excel and Power BI Desktop), but the main difference is that it is available for all Navigation steps, and it only modifies the Navigation steps. It doesn’t result in deletion of subsequent steps like it happens in Power Query Desktop (in Excel and Power BI Desktop). We encourage you to give this new feature a try and share your feedback in our Data Factory community forum. On-Premises support for Fast Copy You can now boost your performance with Fast Copy in Dataflow gen2 when loading data from on-premises data stores including SQL Server. The way you configure Gateway in Dataflow gen2 to get access to on-premises data stores, it is the same for Fast Copy in Dataflow gen2. By doing so, you can ingest terabytes of data from on premise store with the easy experience of dataflows, but with the scalable back end of the pipeline Copy Activity. Learn more about Fast Copy in Dataflow Gen2. Certified connector updates Check out the new and updated connectors in this release: [New] Windsor [Update] SingleStore [Update] Smartsheet [New] BuildingConnected [New] OneStream [Update] Exact Online Premiumv [Update] Funnel [Update] Wolters Kluwer CCH Tagetik [Update] Delta Sharing Data pipeline Edit JSON code for Data pipelines You can now edit the JSON behind your Data Factory pipelines in Fabric Fabric Data Factory is the low-code data integration service in Microsoft Fabric that includes dataflows, data pipelines, and many other data integrations and ETL capabilities. When designing your low-code pipeline workflows, directly editing the JSON code behind your visual pipeline canvas can increase your flexibility and improve your market time. Long-time ADF and Synapse developers have also asked for this capability in Fabric, and we are super excited to now announce the edit JSON capability has been added to Fabric Data Factory data pipelines! New connectors We are excited to announce the release of two powerful new connectors in Fabric Data Factory data pipeline: Azure MySQL Database Connector and Azure Cosmos DB for MongoDB Connector. These connectors are designed to significantly enhance your data integration and management capabilities. The Azure MySQL Database Connector enables seamless, secure, and efficient integration with MySQL databases hosted on Azure, simplifying data workflows and ensuring robust performance. Meanwhile, the Azure Cosmos DB for MongoDB Connector offers unparalleled ease of connection to Azure Cosmos DB for MongoDB instances, providing high compatibility and performance for your NoSQL data needs. Elevate your data integration experience with these new connectors in Fabric Data Factory and unlock the full potential of your data ecosystems. Learn more here. Snowflake storage integration We are thrilled to announce a powerful new feature in the Fabric Data Factory Snowflake Connector to support Snowflake storage integration. This enhancement allows users to seamlessly connect and integrate Snowflake’s storage integration, enabling more efficient and secure data movement. With Snowflake storage integration, you can streamline data workflows, optimize performance across all staging scenarios without the need to bring external storage to stage your dataset as you do today. This update not only simplifies your data integration processes but also significantly expands your ability to move data from or to Snowflake with richer authentication methods through staged approach and thus improves the security. Learn more here. Use existing connections from the OneLake Datahub integration We are excited to share the new capability of OneLake Datahub at the homepage of Modern Get Data in Data Pipeline. Now you can select any existing connections from OneLake Datahub, not just your recent and favorite ones. This makes it easier to access your data sources from the homepage of modern get data in data pipeline. Learn more here.Microsoft Fabric June 2024 Update
Welcome to the June 2024 update. Here are a few, select highlights of the many we have for Fabric. Fabric Spark connector for Fabric Data Warehouse in Spark runtime is now available. Query activity, a one-stop view of your running and completed SQL queries for workspace admins is being announced. You can now run queries in a KQL Queryset and pin the results directly to your Real-Time Dashboard, whether it's new or existing. There is much more to explore, please continue to read on. European Fabric Community Conference Join us at Europe's first Fabric Community Conference, the ultimate Power BI, Fabric, SQL & AI learning event in Stockholm, Sweden from September 24 -27, 2024. With 120 sessions, daily keynotes, 10 pre-conference workshops, an expo hall with community lounge, and "ask the expert" area, the conference offers a rich learning experience you don’t want to miss. This is a unique opportunity to meet the Microsoft teams building these products, customers betting their business on them, and partners at the forefront of deployment and adoption. Register today using code MSCUST for an exclusive discount on top of early bird pricing! Get certified on Fabric! We’d like to thank the thousands of you who completed the Fabric AI and 30 Days to Learn It Skills Challenge and earned a discount voucher for Exam DP-600 which leads to the Fabric Analytics Engineer Associate certification. If you earned a discount voucher, you can find redemption instructions in your email. We recommend that you schedule your exam promptly, before your discount voucher expires. If you need a little more help with exam prep, visit the Fabric Career Hub which has expert-led training, exam crams, practice tests and more. Attention Power BI users! If you are accessing Power BI on a web browser version older than Chrome 94, Edge 94, Safari 16.4, Firefox 93, or equivalent, you need upgrade your web browser to a newer version by August 31, 2024. Using an outdated browser version after this date, may prevent you from accessing features in Power BI. Contents https://youtu.be/sdnPeFR6jmQ?si=b5DyCyucCrQ-1Ovu Reporting Visual calculations update (preview) Support for scatter plot and maps You've asked, and we’ve delivered: scatter plots and maps are now supported with visual calculations! This means you can now add visual calculations to these visual types. Please note that the play axis is not yet supported. Highlights in visual matrix If you look closely at the previous screenshots, you might notice we have added a highlight to the visual matrix to indicate which visual calculation you are editing. We plan to rely on the visual matrix to make working with visual calculations even easier in the future. This highlight is just the first step of our plans. For more information about visual calculations read our documentation. Please try the preview today and let us know what you think. Power BI Home in Desktop is now generally available The new Power BI Home in Desktop is now generally available! Introduced last February, this update brings new functionalities that make it easier for users to create reports. With data sources directly accessible from the home screen and enhanced discoverability features like the recommendation section and the quick access list, getting started is simpler than ever. Check out the documentation to learn more about all the new features and start using them today. Modeling Download large semantic models You can now download your large semantic models to Power BI Desktop as a .pbix file! Previously you could only edit reports connected to these models in live connected report mode. Now you can open and edit your reports and the large model itself as a .pbix file all within Desktop! This unlocks the full end to end Desktop authoring experience you know and love for your large semantic models. Learn more information including limitations in our documentation. New INFO functions and DAX query view improvements New INFO functions are available June 2024. All the INFO functions are DAX function versions of existing Analysis Services Dynamic Management Views (DMVs), so that documentation is provided here for additional information. DAX function DMV Columns documentation INFO.CALCDEPENDENCY DISCOVER_CALC_DEPENDENCY 3.1.4.2.2.1.3.62.1 Columns INFO.CATALOGS DBSCHEMA_CATALOGS 3.1.4.2.2.1.3.1.1 Columns INFO.PROPERTIES MDSCHEMA_PROPERTIES 3.1.4.2.2.1.3.10.1 Columns All INFO functions can now take optional input parameters. As DMVs these optional input parameters are called restrictions. For all the INFO functions this includes their columns, but some have additional restrictions. INFO.CALCDEPENCY can specify QUERY as one if it’s restrictions, or optional input parameters, allowing you to analyze the items in the semantic model needed for that DAX query. This is helpful when analyzing what is being used by the semantic model in a DAX query from a visual in a report. For example, these are valid uses of INFO functions with and without the input parameters: Learn more: INFO functions blog Deep dive into DAX query view blog Additionally, a bug with the copy functionality of the results grid has been fixed. Previously, if the results included a blank cell the copy did not work. Now copy works with blank cells, and we included a right-click copy as well as the option to choose if you want to copy the entire table or just the selected cells from the copy button. Copy > Entire table will include the headers. Copy > Selected cells by selecting all the cells, including if you click the table icon in the top left of the results, will not include the headers. Mobile Show Visuals as Tables (Preview) We strive to make the Power BI mobile app as accessible as possible for everyone. That’s why, in this month’s update, we’ve added the Show Visuals as Tables view mode to all reports. Use this feature to instantly view all the visuals (excluding slicers, cards, and non-data-driven visuals) in your report as table visuals. This feature makes it possible for users who rely on accessibility screen readers to read the underlying data presented in each visual. It also benefits users who prefer to see their data in tabular form, which is more like viewing it in Excel. To turn on the Show Visuals as Tables view mode, tap the new button that has been added to the options menu (...) in the report header. The view mode is applied to all pages in the report. To go back to the regular view, tap Show original visuals in the same menu. Developers Power BI enhanced report format (PBIR) We are very excited to announce the Power BI enhanced report format (PBIR) for Power BI Project files (PBIP). This marks a significant milestone in achieving the primary goal of Power BI Desktop developer mode: provide source control friendly file formats that unblock co-development and enhance development efficiency. Power BI Projects (PBIP) now support saving the report and semantic model into a folder using source-control friendly formats: PBIR for the report and TMDL for the semantic model. The PBIR file format greatly simplifies the tracking of changes and resolution of merge conflicts by using properly formatted JSON and organizing each visual, page, bookmark, etc., in separate individual files within a folder structure. You can also greatly enhance your report development efficiency, either by simply copy & paste visuals/pages/bookmarks/… files between reports or apply manual/programmatic batch changes to the PBIR files. Unlike PBIR-Legacy (report.json), PBIR is a publicly documented format and allows modifications from non-Power BI applications. Each file has a public JSON schema, which documents each property and lets code editors like Visual Studio Code perform syntax validation while editing. On open, Power BI Desktop will validate the changed PBIR files to guarantee successful loading. How to enable it? PBIR is currently in preview, and you can only create or convert existing Power BI project files to PBIR using Power BI Desktop. You must first enable the feature in Power BI Desktop preview features: go to File > Options and settings > Options > Preview features and check the box next to “Store reports using enhanced metadata format (PBIR)”. During preview, Fabric Git Integration and Fabric REST Apis will continue to use PBIR-Legacy (report.json) when exporting the report definitions. However, if the report is imported into Fabric using PBIR format, then both features will start exporting the report definition using PBIR format. At General Availability, PBIR will become the default report format. Limitations Initially, the PBIR format will have some service restrictions, such as: Unable to publish the report in Power BI App. Unable to use subscriptions. Unable to download PBIX. These restrictions will be removed in the following months. Find out more about PBIR and the limitations of Public Preview in our documentation. Visualizations New visuals in AppSource Aimplan Status Visual Water Cup Matrix Planner vuurmans_custom_polar_area_chart hi-chart Reporting Studio Word Cloud by Powerviz Powerviz's Word Cloud is a visual representation of text, with word size indicating frequency or importance in the given content. It offers a quick overview of key themes and is commonly used in presentations and data analysis to highlight patterns and key terms. Key Features: Word Options: Customize text styles and appearance. Direction: Control word orientation with various styles. Colors: Choose from 30+ color schemes. Shapes: Create unique word clouds with icons and images. Ranking: Filter out Top/Bottom N Words. Exclude: Easily remove unwanted words, symbols from the text to create a clean and focused word cloud. Conditional Formatting: Easily spot words with dynamic rules. Lasso/Reverse Lasso: Select and deselect multiple words together. Business Use Cases: Marketing: Analyze feedback, SEO keywords, and sentiment. Education: Improve writing skills and engage students with word clouds. Market Research: Quickly analyze survey responses and opinions. Presentations: Capture attention and summarize information visually. Data Analysis: Explore textual data for insights and trends. 🔗 Try Word Cloud Visual for FREE from AppSource 📊 Check out all features of the visual: Demo file 📃 Step-by-step instructions: Documentation 💡 YouTube Video: Video Link 📍 Learn more about visuals: https://powerviz.ai/ ✅ Follow Powerviz: https://lnkd.in/gN_9Sa6U Drill Down Timeline PRO by ZoomCharts With Drill Down Timeline PRO, you can create slick and intuitive timeline charts with up to 25 series. Users can easily drill down by simply clicking data directly on the chart – for example, the initial view can show aggregated monthly totals, and the user can click on a specific month to quickly access daily values. When paired with other visuals that support cross-filtering, Timeline PRO can become a part of a dynamic and engaging report that provides quick and all-encompassing insights to the user. For example, when you select a specific time range on Timeline PRO, other visuals will display data that’s relevant only to that time period. Conversely, other visuals or slicers can dynamically filter data on your timeline chart as you explore the report. Main Features: On-chart drill down Up to 25 series (columns, lines, and areas) ‘Legend’ field support Custom date/time hierarchy (from milliseconds to decades) DAX measure support Customization – defaults or individual series settings Up to 4 static or dynamic threshold lines/areas Conditional formatting Touch support 🌐Get Drill Down Timeline PRO on AppSource Product Page | Documentation | Follow ZoomCharts on LinkedIn | Community Attribute Control Chart by Nova Silva Maintaining quality standards is a non-negotiable aspect of every successful operation. Enter the new Attribute Control Chart for Power BI, a robust mechanism for monitoring and controlling categorical data variations. The Attribute Control Chart offers a comprehensive suite of six attribute charts to cater to diverse analytical needs: c chart: Monitors number of defects per sample (sample size is constant). u chart: Tracks the number of defects per unit (sample size varies). Laney u' chart: A modified version of the u chart to adjust for overdispersion or under dispersion in your data. p chart: Observes the proportion of defective items per sample (sample size varies). Laney p' chart: A modified version of the p chart to adjust for overdispersion or under dispersion in your data. np chart: Displays the count of defective items per sample (sample size is constant). Power BI users can delve deeper into their data, facilitating timely decision-making and continuous improvement. From manufacturing to healthcare, the Attribute Control Chart equips organizations across industries with the insights needed to uphold quality standards and drive operational excellence. Ready to experience the Attribute Control Chart? Download it from AppSource today and explore its capabilities with your own data – try for free! Download Button by JTA The "Download Button by JTA" is a custom visual designed specifically for Power BI users, empowering them to export large datasets effortlessly. Key Features: Enhanced Data Export: Download up to 300,000 rows and 15 columns of data in CSV format Customizable Interface: Tailor the visual interface to match the reporting style Constraints: The custom visual leverages the browser’s cache memory. It works by fetching data in chunks of 30k rows, storing each chunk in the browser's cache until the entire dataset is compiled for download. This incremental approach requires the use of browser memory to ensure seamless processing. Unfortunately, this means that the visual does not operate as intended in Power BI Desktop due to technical constraints and security restrictions within that environment and you must use the Power BI Service. For the very same reason, there's a possibility that it may encounter limitations or restrictions in certain browsers. Each browser has its own set of security measures and permissions that can affect how our visual operates. If you liked what you saw, you can try it for yourself and find more information here. Also, if you want to download it, you can find the visual package on the AppSource. You can watch the video on YouTube! New Updates for accoPLANNING (Release 69) accoPLANNING for Power BI enhances your business capabilities with advanced planning, forecasting, budgeting, project management, and analysis solutions, all while enabling writeback capabilities. Exciting New Features in This Release: Advanced Options and Save State: Customize column and row colors, adjust text formats, and create custom calculated rows and columns. Improved Settings Interface: We have completely revamped the formatting areas and relocated some settings to make them more intuitive. Writeback Table Scripts: Easily create writeback tables in your database for accoPLANNING visuals with this new feature. Dropdown List: Limit input to predefined values from a dropdown list - perfect for data validation. New Aggregation Options: We've added Median, Product, Min, and Max, among other useful aggregations for ad hoc scenarios. Enhanced Copy-Paste Functionality: Improved copy-paste capabilities between Excel and the visual. Themes: Quickly create visually appealing reports with new theme options. Self-Hosted API: Now available with enhanced data security features, including additional encryption. Fabric Ready: the visual now supports writeback directly to Fabric Data Warehouse. By integrating accoPLANNING with Power BI, you can streamline your planning and reporting processes. For more details, visit our website. Watch on YouTube! Paginated Reports Subfolder support in Power BI Report Builder You can now open reports that are in folders and subfolders from Power BI Report Builder. These reports may reside in folders and/ subfolders in workspaces that the user has access to. You can also publish paginated reports to folders/subfolders that you have access to in the Power BI service. Learn more about creating folders in workspaces. Service Master data - Public Preview We previously introduced endorsement for Fabric items. Certified and promoted endorsements encouraged the use of standardized and trustworthy data. Now, we’re taking it a step further with the introduction of Master Data. This new endorsement stage empowers IT and data teams to define and establish the organization’s single source of truth. By defining Master Data, your organization can benefit from creating a repository of all critical organizational data, making it available to users with a variety of skills to discover and build upon. Learn more about Endorsement and how to enable Master Data in your organization. Master Data endorsement is available to any item type that stores data (e.g. Lakehouse, Warehouse, Semantic model). Note: Master Data feature rollout will be completed by the end of the first week of July. OneLake External data sharing public API In addition to the public preview of external data sharing announced last month, we are proud to announce the public preview of APIs for external data sharing. Users can now scale their data sharing use cases by automating the creation of shares with the public API. Click here to access the API documentation. Synapse Data Warehouse Query activity: A one-stop view to monitor your running and completed SQL queries We’re excited to announce Query activity, a one-stop view of your running and completed SQL queries for workspace admins! We know monitoring SQL queries is essential for maintaining the performance and efficiency of the warehouse to ensure that queries aren’t taking longer than expected to execute and are completed successfully. This feature enhances your ability to monitor queries from your Data warehouse or Lakehouse SQL Analytics Endpoint without having to run any SQL code! There are two ways you can launch the Query activity experience: Click on More Options (…) next to the warehouse you want to monitor within the workspace view and select Query activity. Once you are within the query editor, click on Query activity in the ribbon. On the Query runs page, you can filter your queries from a specific status, submitter, or submit time and cancel any long-running queries that you would like to stop the execution of. On the Query insights page, you can see a list of long running queries and frequently run queries to help determine any trends within your warehouse’s queries. To learn more about Query activity, please refer to the documentation at: Monitor your running and completed queries using Query activity and blog. Discover Insights, Patterns, and Trends with Copilot for Data Warehouse We announced Copilot for Data Warehouse at //Build last month. Sign up here to enable Copilot for Data Warehouse in your organization today! Following up on this announcement, we are excited to announce the ability to brainstorm questions with Copilot for Data Warehouse! Developers and analysts can now leverage Copilot to help discover insights, data points, and trends that are personalized to your warehouse’s schema and metadata. Click the Ask a question button or simply ask Copilot to “suggest questions to ask” to get started: Mirroring Azure SQL Database We are happy to share that we have added more SQL’s Dynamic Definition Language (DDL) support for tables being mirrored. You can now perform Alter Table, Alter column, and change from one datatype to another supported datatype. Data Engineering Spark Connector for Fabric Data Warehouse (DW) We are thrilled to announce the availability of Fabric Spark connector for Fabric Data Warehouse (DW) in Spark runtime. This connector will empower a Spark developer or a data scientist to access and work on data from Fabric DW and SQL analytics endpoint of the lakehouse (either from within the same workspace or from across workspaces) with a simplified Spark API. It hides the underlying complexity involved, automatically discovers the SQL endpoint based on the context of the specified warehouse/lakehouse and literally works with just one line of code. This connector has been designed to keep security in mind and takes a minimalistic approach. Hence it requires minimal permission to work with Fabric SQL engines and honors security models (like Object Level Security (OLS)/ Row Level Security (RLS)/ Column Level Security (CLS)) defined at the SQL engine level while accessing a table or view. The connector will be shipped as a default library within Fabric Runtime and no separate installation will be necessary to make use of it. To learn more about Spark Connector for Fabric Data Warehouse (DW), please refer to the documentation at: Spark connector for Fabric Data Warehouse Notebook column resizing on display () table view The built-in rich dataframe table view on Notebook now supports column resizing! You can use the display(df) command to explore your data and try out the new column resizing experience. Feel free to share more suggestions (like this one) through the feedback channel, we would love to further improve the experience. Notebook pane switcher We recently introduced pane switcher on Notebook, which is a new interactive component to manage your opened panes! With the pane switcher, you can open and use multiple panes at the same time without worrying about them taking up extra Notebook canvas space, it can help you focus on programming while leveraging the rich functions Fabric notebook provided, such like Copilot, browse code snippet, find & replace, comment, etc. You can also easily navigate among each panes embedded in the pane switcher and close them one by one. Real-time Intelligence Effortlessly Pin Query Results to Your Dashboard: Streamline Insights and Collaboration Introducing the ability to run queries in a KQL Queryset and pin the results directly to your Real-Time Dashboard, whether it's new or existing. This functionality streamlines the insights analysis process, allowing you to quickly transform raw data into actionable insights. By pinning query results, you can effortlessly build and share comprehensive dashboards with your team, enhancing collaboration and decision-making. Learn More Unleashing the Power of Graph Semantics in Eventhouse We are thrilled to announce the general availability of the Graph Semantics in Eventhouse. This groundbreaking feature allows users to model their data as graphs and perform advanced graph queries and analytics using the Kusto Query Language (KQL). With support for various graph types, such as directed, undirected, weighted, and labelled graphs, the Graph Semantics extension provides a comprehensive set of graph operators and functions to help users gain deeper insights into complex datasets. Learn more Data Factory Dataflow Gen2 Certified connector updates We’re pleased to announce the new and updated connectors in this release: [Update] BitSight Security Ratings [Update] Supermetrics [Update] Wolters Kluwer CCH Tagetik [Update] LEAP [Update] MongoDB Atlas SQL [Update] Denodo [Update] Zendesk Data [New] Intersystems Health Insight Data pipeline Use Data pipeline to read and write to Lakehouse table with schema information (Public Preview) With the release of the Lakehouse schemas feature, available for public preview on June 28 th , data pipeline will now support reading the schema info from Lakehouse tables and will support writing data into tables under specified schemas. New connectors are available in data pipeline More connectors are now available for data pipelines: Azure AI Search Azure Cosmos DB API for MongoDB Azure Files Dynamics AX Google BigQuery MySQL Oracle For more information on data pipeline connectors, please read Data pipeline connectors in Microsoft Fabric. Move Data Across Workspace via Data pipeline Modern Get Data Experience We are thrilled to introduce the native capability that allows you to easily move data among Lakehouses, warehouses, etc. across different workspaces. You can intuitively select Fabric Lakehouse, Warehouse, etc from another workspace under the “Explorer” on the left side of the “OneLake data hub” in Pipeline Modern Get Data. It also lets you filter by artifact types. Besides selecting existing Lakehouse and Warehouse across various workspace, you can also easily create a new one in a different workspace. Support to create a new Warehouse as destination in Data pipeline You can now create a new Warehouse as a destination in Data Pipeline, instead of only selecting an existing one. This feature allows you to load your source data into Warehouse more easily and intuitively. You can easily create a new Warehouse as your destination in the “Choose data destination” step of Data Pipeline Copy assistant. You can find this option in the “New Fabric name” section of the homepage, or in the “New Fabric name” page. You can also easily create a new Warehouse in the “New Fabric item” section when you edit the destination of Pipeline Copy activity.Fabric September 2024 Monthly Update
Welcome to the September 2024 Update! Announcements We have a lot of exciting announcements to share with you for FabCon Europe! We've brought Copilot to Dataflows Gen2 and a richer Copilot experience when building and consuming Power BI Reports. With Real-Time Intelligence we have redesigned and enhanced user experience in the Real-Time hub. We announced the general availability of Fabric Git integration. You can sync Fabric workspaces with Git repositories, leverage version control, and collaborate seamlessly using Azure DevOps or GitHub. We now have an enhanced and redesigned left navigation experience in Real-time Intelligence with the new Real-Time hub user experience. In AI, we have released Copilot in Fabric experience for Dataflows Gen2 into general availability; allowing everyone to design dataflows with the help of an AI-powered expert. We also released Copilot in Fabric experience for Data Warehouse into preview. This AI assistant experience can help developers generate T-SQL queries for data analysis, explain and add in-line code comments for existing T-SQL queries, fix broken T-SQL code, and answer questions about general data warehousing tasks and operations. To learn more, read about all these announcements, and more in Arun's blog post Building an AI-powered data platform. Contents Power BI Core Announcing the availability of Trusted workspace access and Managed private endpoints in any Fabric capacity. Multitenant organization (MTO) (public preview) Announcing Git integration (generally available) A new design for Deployment pipeline (preview) Homepage improvements OneLake Access Databricks Unity Catalog tables from Fabric (public preview) Google Cloud Storage shortcuts and S3 Compatible shortcuts (generally available) REST APIs for OneLake shortcuts (generally available) OneLake SAS (public preview) Data Warehouse Copilot for Data Warehouse (public preview) Delta column mapping in the SQL analytics endpoint (public preview) Enabling SQL analytics endpoint on schema enabled Lakehouse’s (public preview) New editor improvements for Data Warehouse and SQL analytics endpoint Database Migration Experience (private preview) TSQL Notebook (public preview) Nested Common Table Expression (public preview) Data Engineering High Concurrency mode for Notebooks in Pipelines (public preview) Workspace Level Setting to Reserve Maximum Cores for Jobs in Fabric Data Engineering (public preview) Session Expiry Control in Workspace Settings for Notebook Interactive Runs (public preview) Spark Connector for Fabric DW – New Features T-SQL Notebook (public preview) Fabric Spark Diagnostic Emitter: Collect Logs and Metrics (public preview) Environment Artifact integration with Synapse VS Code extension Notebook debug within vscode.dev (public preview) Adding Python support in Fabric User Data Functions Invoke Fabric User Data Functions in Notebook Functions Hub is now available in Fabric User Data Functions Support for spaces in Lakehouse Delta table names Fabric Runtime 1.3 GA Native Execution Engine on Runtime 1.3 (public preview) Acceleration tab and UI enablement for the Native Execution Engine Fabric Spark Runtimes Release Notes Enable/Disable Functionality in API for GraphQL Public REST API of Livy Endpoint Data Science Announcing Public Preview: Share Feature for Fabric AI Skill Data Wrangler now supports Spark DataFrames and PySpark code generation (generally available) Announcing new usability improvements for Data Wrangler File editor in Notebook Real-time Intelligence Creating a Real time Dashboard by Copilot Adding a real-time dashboard to an org app Introducing A New Real-Time Hub User Experience New Streaming Sources Now Available via Eventstream Connectors Introducing Eventhouse as a new Destination in Eventstream Eventstream’s Integration with Managed Private Endpoint Introducing the new look and feel of KQL Database Set alerts on KQL Querysets with Data Activator triggers Data Exploration made easier with Top values feature Real-Time Dashboard lower than ever refresh rate Multivariate anomaly detection Real-Time Intelligence Copilot conversational mode New Rule Creation Experience We’ve made it easier to alert your teammates in Power BI Data Factory Dataflow Gen2 Copilot in Dataflow Gen2 (generally available) Incremental refresh for Dataflow Gen2 Certified connector updates Data pipeline Fabric Pipeline Integration in On-premises Data Gateway (generally available) Invoke remote pipeline in Data pipeline Spark Job environment parameters Mirroring Mirroring Azure SQL Database New Azure Data Factory Item Copy Job (public preview) Monthly Update Video Power BI You can now choose from a variety of themes Power BI Desktop, including the most requested Dark Mode! You can personalize your data visualization experience to match your preferences and working environment. In addition, we’ve now consolidated similar options in the menu bar and streamlined the button text for better readability and responsive screen sizing. The Copilot chat pane will now automatically provide text-based answers and summaries across all pages in a report. Previously, users had to specifically request cross-page summaries or click a "base summary on the entire report" button. With this update, cross-page summaries and answers are now the default setting, streamlining the exploration process. Power BI has a transformative new feature designed to redefine how organizations manage and consume metrics and features visuals and Copilot insights Called Metrics Hub. Metrics Hub is an innovative metric layer within Fabric, aimed at helping organizations define, discover, and reuse trusted metrics with ease. This feature allows trusted creators within an organization to develop standardized metrics that incorporate essential business logic, ensuring consistency across the organization. To learn more about these, and all the other new features this month, read the Power BI September 2024 Feature Summary. Core Announcing the availability of Trusted workspace access and Managed private endpoints in any Fabric capacity. We’d like to share an update for the Fabric network security features that were announced in general availability earlier this year. Trusted workspace access, and Managed Private endpoints enable you to secure and optimize your data access and connectivity with Fabric and protect your business-critical data from unauthorized or unwanted access. However, these features were available only in F64 or higher capacities. Based on your feedback, we are now making these features available in all F capacities. You can now use these features with any F capacity that suits your business needs. We are also making Managed Private endpoints available in Trial capacities as part of this release. Here’s a quick recap of what these features do and how they can help you: Trusted workspace access allows seamless and secure access to firewall enabled Azure storage accounts. It is designed to help you securely and easily access data stored in Storage accounts from Fabric workspaces, without compromising on performance or functionality. This feature extends the power and flexibility of OneLake shortcuts to work with data in protected storage accounts in place without compromising on security. You can also use this capability with Data pipelines and the COPY INTO feature of Fabric warehouses to ingest data securely and easily into Fabric workspaces. To get started with this feature and to learn about limitations, see Trusted workspace access in Microsoft Fabric – Microsoft Fabric | Microsoft Learn. This feature can be used in any F capacity. Managed private endpoints provide secure connectivity from Fabric to data sources that are behind a firewall or not accessible from the public internet. Managed Private Endpoints enable Fabric Data Engineering items to access data sources securely without exposing them to the public network or requiring complex network configurations. Managed private endpoints are supported for various data sources, such as Azure Storage, Azure SQL Databases, and many others – the most recent addition being Azure Event Hub and Azure IOT Hub. To learn more about Managed Private Endpoints and supported data sources see Overview of managed private endpoints for Microsoft Fabric – Microsoft Fabric | Microsoft Learn. This feature can be used in any F capacity as well as Trial. Multitenant organization (MTO) (public preview) Fabric now supports Entra Id Multitenant Organizations (MTO). Many larger organizations have multiple Entra Id tenants for various reasons such as mergers and acquisitions, compliance and security boundaries, or due to complex organizational structure. The multitenant organizations capability in Entra Id synchronizes users across multiple tenants, adding them as users of type external member. We are excited to announce public preview support for MTO. External members can now sign in to Fabric to consume and create content. MTO users can bring their own licenses from their home tenants. Users that have been assigned Power BI Pro or PPU licenses in their home tenants will not need to acquire an additional license for the other MTO tenants. Click here for more information. Announcing Git integration (generally available) Fabric Git integration is now generally available! This feature allows you to sync workspaces with Git repositories, leverage version control, and collaborate seamlessly using Azure DevOps or GitHub. Though some items are still in preview, additional items will become generally available for Git integration over time. Learn more about what's available now and what’s coming next. A new design for Deployment pipeline (preview) Deployment Pipeline Redesign is now in preview! We are thrilled to announce the launch of Deployment pipelines redesign. This major update brings many improvements and new features designed to make your deployment process more efficient and user-friendly. We’ve reimagined the deployment pipeline from the ground up, ensuring that every aspect of your deployment workflow is optimized for performance and ease of use. Homepage improvements We are excited to share the latest enhancements to our homepage that are designed to streamline your experience and boost your productivity. 1. Enhanced Workspace Focus The first improvement we've made is putting workspaces and related actions in the premium real estate on the homepage. The primary call to action is now dedicated to creating a workspace and navigating to recent ones. In the Get started section, you can now create a workspace with a predesigned template called task flow. The task flow guides you to create specific items in a workspace. The idea is to encourage you to think in terms of projects and what you are trying to build end-to-end, instead of having to think about which workloads or items you need to use. If you want to revisit any of your recent workspaces and continue where you left off, the Quick Access section is now optimized for efficient workspace navigation. 2. Collapsible Learn Section We’ve also refined our Recommended section, transforming it into the Learn section. This new area is packed with sample materials and learning resources designed to help new users get up to speed quickly. Recognizing that these resources are particularly useful for newcomers, we've made the Learn section collapsible. For our experienced users, collapsing this section provides more space for the Quick Access area, ensuring a more streamlined experience. Watch a demo for Core OneLake Access Databricks Unity Catalog tables from Fabric (public preview) You can now access Databricks Unity Catalog tables directly from Fabric. In Fabric, you can now create a new data item called “Mirrored Azure Databricks Catalog”. When creating this item, you simply provide your Azure Databricks workspace URL and select the catalog you want to make available in Fabric. Rather than making a copy of the data, Fabric creates a shortcut for every table in the selected catalog. It also keeps the Fabric data item in sync. So, if a table is added or removed from UC, the change is automatically reflected in Fabric. Once your Azure Databricks Catalog item is created, it behaves the same as any other item in Fabric. Seamlessly access tables through the SQL endpoint, utilize Spark with Fabric notebooks and take full advantage of Direct Lake mode with Power BI reports. To learn more about Databricks integration with Fabric, see our documentation here. Google Cloud Storage shortcuts and S3 Compatible shortcuts (generally available) GCS shortcuts and S3 Compatible shortcuts are now generally available. Utilize shortcuts in OneLake to quickly and easily make data accessible in Fabric. No need to set up pipelines or copy jobs, just create a shortcut and your data is immediately available in Fabric. Don’t forget to enable shortcut caching, GCS and S3 Compatible shortcuts both support caching. This can be enabled in your workspace setting. By enabling shortcut caching, you can reduce your egress costs when accessing data across clouds or service providers. GCS and S3 Compatible shortcuts also support the On-Premises Gateway. You can utilize the gateway to connect to your on-prem S3 compatible sources as well as GCS buckets that are protected by a virtual private cloud. To learn more about shortcuts see our documentation here. REST APIs for OneLake shortcuts (generally available) We recently made big improvements to the REST APIs for OneLake shortcuts, including adding support for all current shortcut types and introducing a new List operation. With these improvements, you can programmatically create and manage your OneLake shortcuts. We’re excited to announce that these APIs are now Generally Available! OneLake SAS (public preview) Support for short-lived, user-delegated OneLake SAS is now in public preview. This functionality allows applications to request a User Delegation Key backed by an Entra ID, and then use this key to construct a OneLake SAS token. This token can be handed off to provide delegated access to another tool, node, or user, ensuring secure and controlled access. Data Warehouse Copilot for Data Warehouse (public preview) Copilot for Data Warehouse in public preview! Copilot for Data Warehouse is an AI assistant that helps developers generate insights through T-SQL exploratory analysis. Copilot is contextualized to your warehouse’s schema. With this feature, data engineers and data analysts can use Copilot to: Generate T-SQL queries for data analysis. Explain and add in-line code comments for existing T-SQL queries. Fix broken T-SQL code. Receive answers regarding general data warehousing tasks and operations. Learn more about Copilot for Data Warehouse. Copilot for Data Warehouse is currently only available in the Warehouse. Make sure you have Copilot enabled in your tenant and capacity settings to take advantage of these capabilities. Copilot in the SQL analytics endpoint is coming soon. Delta column mapping in the SQL analytics endpoint (public preview) SQL analytics endpoint now supports Delta tables with column mapping enabled for public preview. Column mapping is a feature of Delta tables that allows users to include spaces, as well as any of these characters, in the table’s column names: ,;{}()\n\t=. The extra characters in the column names are shown in the Lakehouse, brought through into the SQL Analytics endpoint, the semantic model, and into Power BI reports. Enabling SQL analytics endpoint on schema enabled Lakehouse’s (public preview) We are enabling the SQL analytics endpoint on schema enabled lakehouses. This allows delta tables in schemas to be queried in the SQL analytics endpoint. New editor improvements for Data Warehouse and SQL analytics endpoint We are excited to share key improvements to our new editor in Fabric Data Warehouse and SQL analytics endpoint to improve the consistency and efficiency of SQL developers' experiences! Starting with the ribbon, the actions presented would change depending on your context in our previous editor. This lacked consistency with other Fabric tools and required extra click stops to launch a certain experience from the ribbon. To make the ribbon more intuitive, we have improved our ribbon to be unified by consolidating all dev tools in a single location split by two tabs for a streamline workflow. The Home and Reporting tab now consists of features for an end-to-end developer experience with no overlaps and no added actions depending on your context. Our new data grid within data preview and displaying query results now provides added capabilities at the column level including sort by ascending and descending order, select, and specify values to filter out per column along with many more features to come. These capabilities can help you quickly analyze and filter your data without running any T-SQL. We’ve listened to your feedback on scenarios where you need to look at multiple experiences at once when writing queries. For example, checking on the data in the tables and the relationships between tables in the BI models to decide how to better structure the query. Within the editor, multitasking between dynamic tabs is now supported between different experiences such as data preview, querying, and modeling for a more efficient data analyzing experience. The multitasking navigation between warehouses and SQL analytics endpoints has also been improved so that whether you’re on a data preview, SQL query, or modeling tab, you can smoothly transition between warehouses/SQL analytics endpoints, and it persists in your last activity. Database Migration Experience (private preview) We are excited to announce the opening of a Private Preview for a new Migration Experience. Designed to accelerate the migration of SQL Server, Synapse dedicated SQL pools, and other warehouses to the Fabric Data Warehouse, users will be able to migrate the code and data from the source database, automatically converting the source schema and code to Fabric Data Warehouse, helping with data migration, and providing AI powered assistance. Please contact your Microsoft account team if you are interested in joining the preview. TSQL Notebook (public preview) You can now use Fabric Notebooks to develop your Fabric warehouse and consume data from your warehouse or SQL analytics endpoint. The ability to create a new notebook item from the warehouse editor lets you carry over your warehouse context into the notebook and use rich capabilities of notebook to run T-SQL queries. T-SQL notebook enables you to execute complex T-SQL queries, visualize results in real-time, and document your analytical process within a single, cohesive interface. The embedded rich T-SQL IntelliSense and easy gestures like Save as table, Save as view or Run selected code provides familiar experiences in the notebook experience to increase your productivity. Learn more here. Nested Common Table Expression (public preview) Fabric Warehouse customers now can use Nested Common Table Expression (NCTE) to deconstruct ordinarily complex queries into smaller reusable blocks. NCTE simplifies complex query code, improves query readability, and query code investigation. With this addition, Fabric Warehouse now supports three types of CTE. They are standard, sequential, and nested CTE. A standard CTE doesn't reference or define another CTE in its definition. A nested CTE's definition includes defining another CTE. A sequential CTE's definition can reference an existing CTE but can't define another CTE. Watch the Data Warehouse demo Data Engineering High Concurrency mode for Notebooks in Pipelines (public preview) We are excited to announce the public preview of High Concurrency mode for Notebooks in Pipeline. This new feature in Microsoft Fabric enables users to share Spark sessions across multiple notebooks within a pipeline. Pipelines are primarily used for orchestrating data engineering tasks for production workloads and scheduled jobs. For enterprise data teams, optimizing resource utilization and achieving the best price-performance ratio is crucial for faster job start times and efficient compute usage. With High Concurrency Mode, users can trigger pipeline jobs, and these jobs are automatically packed into existing high concurrency sessions. Subsequent notebook steps benefit from a 5-second session start experience, even with custom compute configurations and custom pools, resulting in a 30x performance boost and instant session start. Note: Session sharing is always confined to a single user, workspace, and pipeline boundary. Sessions are selected based on matching compute configurations, library management dependencies, and file system dependencies. Learn more about high concurrency mode for notebooks in pipelines from our documentation. Workspace Level Setting to Reserve Maximum Cores for Jobs in Fabric Data Engineering (public preview) We're pleased to introduce a new workspace-level setting that allows you to reserve maximum cores for your active jobs for Spark workloads. By default, optimistic job admission is enabled in all workspaces, and it allows jobs to start with their minimum node configuration and scale up to multiple executors based on available capacity. In cases where there are excessive jobs running, pushing the capacity to its maximum limits, scale-up requests may be rejected. For enterprise customers requiring absolute maximum core reservations for critical jobs, you can now enable a compute reservation model. Workspace administrators can enable this option by navigating to the Data Engineering/Science section of the workspace settings and activating the "Reserve maximum cores for active Spark jobs" setting. Once enabled, the maximum auto scale size of the Spark pool is considered during job admission. Even if a job is currently running with 2 nodes, if the pool's maximum limit is 5, the other 3 nodes will be reserved for the job's potential growth throughout its lifetime. This will ensure that each job that’s submitted has the cores available for it to grow its maximum scale. Users now have the flexibility to choose between optimistic job admission and compute reserved mode, tailoring their workspace settings to meet specific workload requirements. Learn more about settings to reserve maximum cores for your Fabric data engineering jobs from our documentation. Session Expiry Control in Workspace Settings for Notebook Interactive Runs (public preview) We're pleased to introduce a new session expiry control in Data Engineering/Science workspace settings. This feature empowers administrators to set the maximum expiration time limit for notebook interactive sessions. Notebooks are a popular choice for interactive querying, and by default, sessions expire after 20 minutes. With this new setting, you can now customize the maximum expiration duration, helping to prevent unused sessions from consuming unnecessary capacity and potentially impacting the performance of new incoming job requests. If users require additional time, they can extend the session duration using the "extended session time" option available in the monitoring status view within the notebook experience. Learn more about session expiry settings for interactive notebooks sessions from our documentation. Spark Connector for Fabric DW – New Features Recently, we launched Fabric Spark connector for Fabric Data Warehouse (DW) in Fabric Runtime to empower Spark developers or data scientists to access and work on data from Fabric DW and SQL analytics endpoint of the lakehouse (either from within the same workspace or from across workspaces) with a simplified Spark API. This initial version supported reading data from a table or view only in Scala. We are happy to announce that we have released these additional capabilities: Support for custom or pass-through query Support for PySpark Support for Fabric Runtime 1.3 (Spark 3.5) To learn more about Spark Connector for Fabric Data Warehouse (DW) with its recent updates, please refer to the documentation at: Spark connector for Fabric Data Warehouse. T-SQL Notebook (public preview) T-SQL notebook is now available for public preview. This enhancement broadens our language support, extending from a Spark-centric approach to including T-SQL as well. With this update, T-SQL developers can now utilize Notebook for crafting their T-SQL queries to develop a warehouse. They can organize extensive queries into separate code cells and use Markdown cells for enhanced documentation, offering more comprehensive documentation experience. Just like we can add Lakehouse into the notebook, it's enabled to add a Data Warehouse or SQL analytics endpoint into the notebook. This allows you to run T-SQL code directly against the connected warehouse or SQL analytics endpoint. BI Analysts can also take advantage of the update by utilizing T-SQL Notebook to execute cross-database queries. This will enable them to compile business insights from various warehouses and SQL analytics endpoints. Most of the existing features are directly accessible for T-SQL notebooks. For instance, T-SQL developers can take advantage of comprehensive charting tools to visualize the results of their T-SQL queries, as well as collaborate with colleagues to jointly develop the notebook. (collaborate in a notebook) To create a T-SQL notebook from an existing data warehouse, a new entry named “New SQL query in notebook” is added under the “New SQL query” menu group. This action generates a new Notebook in the same workspace, with the data warehouse automatically added into it. You can create a notebook code cell with T-SQL as the language and run the query against the connected Data Warehouse. Fabric Spark Diagnostic Emitter: Collect Logs and Metrics (public preview) Fabric Apache Spark Diagnostic Emitter is now in public preview, a powerful new feature that allows Apache Spark users to collect logs, event logs, and metrics from their Spark applications and send them to various destinations, including Azure Event Hubs, Azure Storage, and Azure Log Analytics. This feature provides robust support for monitoring and troubleshooting Spark applications, enhancing your visibility into application performance. What Does the Diagnostic Emitter Do? The Fabric Apache Spark Diagnostic Emitter enables Apache Spark applications to emit critical logs and metrics that can be used for real-time monitoring, analysis, and troubleshooting. Whether you're sending logs to Azure Event Hubs, Azure Storage, or Azure Log Analytics, this emitter simplifies the process, allowing you to collect data seamlessly and store it in the destination that best suits your needs. Key Benefits of the Apache Spark Diagnostic Emitter Centralized Monitoring: Send logs and metrics to Azure Event Hubs, Azure Log Analytics, or Azure Storage for real-time data streaming, deep analysis and querying, as well as long-term retention. Flexible Configuration: Easily configure Spark to emit logs and metrics to one or more destinations, with support for connection strings, Azure Key Vault integration, and more. Comprehensive Metrics: Collect a wide range of logs and metrics, including driver and executor logs, event logs, and detailed Spark application metrics. To learn more, see: Monitor Apache Spark applications with Azure Log Analytics Collect your Apache Spark applications logs and metrics using Azure Event Hubs Collect your Apache Spark applications logs and metrics using Azure Storage account Environment Artifact integration with Synapse VS Code extension Microsoft Fabric environments is a consolidated item for all your hardware and software settings. In an environment, you can select different Spark runtimes, configure your compute resources, install libraries from public repositories or local directories and more. To learn more, see Create, configure, and use an environment in Fabric. By supporting the Environment item within the Synapse VS Code extension, you can explore and manage the Environment from VS Code side. Expanding the new node of the Environment within VS Code, you can see all the Environment items from the selected workspace and easily identify the workspace default one. You can switch the workspace default to some other item by hovering over the environment and selecting the Set Default Workspace Environment button. Hovering the environment and selecting the Inspect button, the environment details should be displayed in the right panel in JSON format. You can find out the association between the code item, such as Notebook, and the environment from the code item detail property panel. Notebook debug within vscode.dev (public preview) Vscode.dev is a lightweight version of VS Code running fully in the browser. We released the extension Synapse VS Code -Remote last year to enable the edit and run Fabric Notebook within vscode.dev. We are excited to announce the introduction of the debug feature for Notebook today. This update brings us closer to achieving a pro-dev experience with Notebook. Once you've opened the Notebook through the VS Code (Web) interface, you can place breakpoints and debug your Notebook code, which is helpful for troubleshooting. This update first starts with the Fabric Runtime 1.3 Adding Python support in Fabric User Data Functions Introducing support for Python functions in Fabric User Data Functions. Private preview users can now create and run functions that will run on Python version 3.11. This enables users to leverage powerful Python libraries such as pandas, numpy, seaborn and more. These functions are compatible with the existing Fabric data source integrations, so users can interact with Data Warehouses as well as Lakehouses. This feature is available for users of the Private Preview of User Data Functions. To participate in this preview, please fill out the application form. Invoke Fabric User Data Functions in Notebook You can now invoke User Defined Functions (UDFs) in your PySpark code directly from Microsoft Fabric Notebooks or Spark jobs. This integration makes it easy to call reusable functions across different workspaces, reducing the need to rewrite or refactor code. With NotebookUtils integration, invoking UDFs is as simple as writing a few lines of code. Whether you’re working in a Notebook or running batch jobs using Spark Job Definitions, the process is incredibly straightforward. Below is an example of invoking UDFs in Notebooks. Functions Hub is now available in Fabric User Data Functions Functions Hub provides a single location to view, access and manage your User Data Functions. Users can access this feature in the left navigation bar, or in the Data Engineering experience. In Functions Hub, users can see all the functions they have access to and perform actions such as manage access permissions, create a new function, view lineage and more. This is helpful to manage User Data Functions artifacts on scale. This feature is available for users of the Private Preview of User Data Functions. To participate in this preview, please fill out the application form. Support for spaces in Lakehouse Delta table names We are pleased to announce the support for spaces in Lakehouse Delta table names, enhancing usability for all users across Fabric. This feature allows you to create and query Delta tables with spaces in their names, such as "Sales by Region" or "Customer Feedback". This aligns with the expectations of Power BI and SQL customers, who are used to working with spaces in table names. Now, you can seamlessly use spaces in table names across Lakehouse Explorer and its experiences, Spark, Shortcuts, and other experiences. All Fabric Runtimes and Spark authoring experiences support table names with spaces. Code completion authoring works as expected, properly escaping the table names with the required backtick characters. Fabric Runtime 1.3 GA We are extremely excited to announce the advancement of Fabric Runtime 1.3 from Public Preview to General Availability. Our Apache Spark-based big data execution engine, optimized for both data engineering and data science workflows, has been fully updated and seamlessly integrated into the Fabric platform, along with all our components. With this announcement, all new workspaces will, by default, be based on the latest GA runtime version, which is now Fabric Runtime 1.3. Existing workspaces will remain in their current version; however, we strongly encourage you to migrate to Runtime 1.3 as soon as possible to take full advantage of the newest functionalities. The latest enhancements in Fabric Runtime 1.3 include the upgrade to Delta Lake 3.2, updates and upgrades to Python libraries, and improvements to the R language. Additionally, we've implemented query-specific optimizations to further enhance performance and efficiency. Read more here. Native Execution Engine on Runtime 1.3 (public preview) Native Execution Engine for Fabric Runtime 1.3 is now available in public preview. Previously, the Native Execution Engine supported earlier versions of Runtime 1.3, but it is now fully compatible with the latest GA runtime version, which is also based on Delta Lake 3.2. The Native Execution Engine can significantly enhance the performance of your Spark jobs and queries. This engine has been completely rewritten in C++, operates in columnar mode, and utilizes vectorized processing. It offers superior query performance across data processing, ETL, data science, and interactive queries—all directly on your data lake. Importantly, this engine is fully compatible with Apache Spark™ APIs, including the Spark SQL API. The current release of the Native Execution Engine excels particularly in the following scenarios: Working with data in Parquet and Delta formats. Handling queries that involve complex transformations and aggregations, leveraging the engine’s columnar processing and vectorization capabilities. Running computationally intensive queries, as opposed to simple or I/O-bound operations. Best of all, no code changes are required to take advantage of the Native Execution Engine. Read more here. Acceleration tab and UI enablement for the Native Execution Engine No code changes are required to speed up the execution of your Apache Spark jobs when using the Native Execution Engine. You have the flexibility to activate the Native Execution Engine either through your environment settings or selectively for an individual notebook or job. By enabling this feature within your environment settings, all subsequent jobs and notebooks associated with that environment will automatically inherit this configuration. We are also excited to share that enabling the Native Execution Engine is now even more accessible. You can activate it not only through Spark settings but also via the Environment tab, specifically under the "Acceleration" section in the UI, ensuring a seamless and straightforward enablement process. Fabric Spark Runtimes Release Notes We are committed to ensuring that our Microsoft Fabric Spark runtimes remain at the forefront of performance and security. In our ongoing efforts, we occasionally introduce updates that may affect your experience. While we understand the critical nature of these updates, we recognize the challenges in keeping up with changes, identifying affected components, and understanding the rationale behind these modifications. To enhance your experience, we are now providing detailed, automated release notes for each Apache Spark-based runtime we deliver. These notes will be readily accessible, offering complete transparency and clarity regarding the updates, including their purpose. This will empower you to seamlessly adapt your workloads and fully leverage the benefits of each update. For the most current information, including a comprehensive list of changes and specific release notes for each runtime version, we encourage you to check and subscribe to Spark Runtimes Releases and Updates. Enable/Disable Functionality in API for GraphQL The Enable/Disable feature for queries and mutations in Microsoft Fabric's GraphQL API provides administrators and developers with granular control over API access and usage. This functionality allows you to selectively activate or deactivate specific queries and mutations within your GraphQL schema, giving you the ability to manage API capabilities dynamically without altering the underlying code or deploying changes. By leveraging this feature, you can permanently or temporarily disable certain operations for maintenance, gradually roll out new functionality, or restrict access to sensitive data operations as needed. This level of control enhances security, aids in API versioning, and provides flexibility in managing your GraphQL API's behavior to align with your application's evolving requirements and operational needs. To disable a query or mutation within a GraphQL item, simply select the ellipses next to the query or mutation you want to disable. Then select the Disable option from the pop-up menu. If a query or mutation has been disabled, you will see it grayed out on the Schema Explorer. You can enable it by simply choosing Enable from the entries’ pop-up menu. Public REST API of Livy Endpoint The Fabric Livy endpoint lets users submit and execute their Spark code on the Spark compute within a designated Fabric workspace, eliminating the need to create any Notebook or Spark Job Definition artifacts. This integration with a specific Lakehouse artifact ensures straightforward access to data stored on OneLake. Additionally, Livy API offers the ability to customize the execution environment through its integration with the Environment artifact. When a request is sent to the Fabric Livy endpoint, the user-submitted code can be executed in two different modes: Session Job: A Livy session job entails establishing a Spark session that remains active throughout the interaction with the Livy API. This is particularly useful for interactive and iterative workloads. A Spark session starts when a job is submitted and lasts until the user ends it or the system terminates it after 20 minutes of inactivity. Throughout the session, multiple jobs can run, sharing state and cached data between runs. Batch Job: A Livy batch job entails submitting a Spark application for a single execution. In contrast to a Livy session job, a batch job does not sustain an ongoing Spark session. With Livy batch jobs each job initiates a new Spark session, which ends when the job finishes. This approach works well for tasks that don't rely on previous computations or require maintaining state between jobs. The endpoint of the session job API would look like: https://api.fabric.microsoft.com/v1/workspaces/ws_id/lakehouses/lakehouse_id /livyapi/versions/2023-12-01/ batches The endpoint of the session job API would look like: https://api.fabric.microsoft.com/v1/workspaces/ws_id/lakehouses/lakehouse_id /livyapi/versions/2023-12-01/sessions ws_id: this is the workspace in which the hosting Lakehouse artifact belongs to. lakehouse_id: this is the artifact id of the hosting Lakehouse. All the capacity consumption history from this batch API call will be associated with this artifact. To access the Livy API endpoint, you need to create a Lakehouse artifact. After it's set up, you'll find the Livy API endpoint in the settings panel. The “Recent Run” section of the Lakehouse will now display all submitted requests. Click on the “Application name” column to open the detailed monitoring page and view additional logs. Typically, the Livy request is directed to the workspace starter pool by default. However, users can customize the execution by including the environment artifact ID in the payload body, utilizing the settings from the specified environment artifact. In this example, the value of the id is the artifact id of the environment, the workspaceId is the id of the current workspace. In the current version, the environment and Lakehouse artifacts should belong to the same workspace. { "conf": { "spark.fabric.environmentDetails": "{\"id\":\"558bd4a3-5107-413e-ad6d-48a4b80678f6\",\"workspaceId\":\"ea0f47a3-8f30-4cf9-bfb6-0b3af37d9eb8\"}" } } Watch the Data Engineering demos Data Science Announcing Public Preview: Share Feature for Fabric AI Skill The highly anticipated feature for Fabric AI Skill, the "Share" capability is now in public preview. This powerful addition allows you to share the AI Skill with others using a variety of permission models, providing you with complete control over how your AI Skill is accessed and utilized. With this new feature, you can: Co-create: Invite others to collaborate on the development of your AI Skill, enabling joint efforts in refining and enhancing its functionality. View Configuration: Allow others to view the configuration of your AI Skill without making any changes. Query: Enable others to interact with the AI Skill to obtain answers to their queries. Additionally, we are introducing flexibility in managing versions. You can now switch between the published version and the current version you are working on. This feature facilitates performance comparison by running the same set of queries, providing valuable insights into how your changes impact the AI Skill’s effectiveness. We’ve also refined the publishing process. You can now include a description that outlines what your AI Skill does. This description will be visible to users, helping them understand the purpose and functionality of your AI Skill. We are excited for you to explore these new capabilities and look forward to hearing about your experience. Your feedback is crucial as we continue to enhance your experience with Fabric AI Skill. Data Wrangler now supports Spark DataFrames and PySpark code generation (generally available) Data Wrangler, a notebook-based tool for exploratory data analysis, now works for both pandas DataFrames and Spark DataFrames in general availability. You can use the tool to explore and transform data in your notebook with an immersive visual interface, generating either Python code or PySpark code in just a few steps. If you’ve used Data Wrangler before, the steps will be familiar. You can begin by opening any active pandas or Spark DataFrame from the “Data Wrangler” prompt in the notebook ribbon. Data Wrangler will automatically convert Spark DataFrames to pandas samples for performance reasons. However, all the generated code will ultimately be translated to PySpark when you save it back to your notebook. (You can also modify the sample size and sampling method by selecting “Choose custom sample” from the Data Wrangler dropdown.) As with a pandas DataFrame, Data Wrangler will show you descriptive insights about your data in the right-hand “Summary” panel and in each column’s header. You can use the left-hand “Operations” panel to browse and apply common data-cleaning transformations. Selecting one will prompt you to provide a target column or columns, along with any necessary parameters. As you fill in those values, a preview of the applied operation, along with the corresponding code, will be automatically generated. If you apply the previewed step, Data Wrangler’s display grid and summary statistics update to reflect the results. The code appears in a running list of committed operations. After applying a series of steps, you can copy or save the generated code using options in the toolbar above the display grid. For Spark DataFrames, all the code generated on the pandas sample is translated to PySpark before it lands back in the notebook. Data Wrangler will display a preview of the translated PySpark code and provide an option to export the pandas code as well. Announcing new usability improvements for Data Wrangler You may have noticed that Data Wrangler has a new entry point under the “Home” tab of the Fabric notebook ribbon. We’re excited to share additional usability updates designed to improve your experience. Launch Data Wrangler directly from a cell You can now open Data Wrangler directly from a notebook cell when you use the Fabric display() command to print a pandas or PySpark DataFrame. Just click on the “Data Wrangler” prompt in the interactive display output. Work faster with improved performance Thanks to new engineering updates, you’ll notice that Data Wrangler takes less time to apply an operation once you’ve loaded the preview. We’ve cut down the latency to speed up your workflow. Modify your display using the new “Views” prompt Data Wrangler works best on large monitors, but you can now use the “Views” dropdown above the display grid to minimize or hide parts of the interface based on your preferences or screen size. File editor in Notebook We are excited to share the launch of the file editor feature in Fabric Notebook. This new feature allows users to view and edit files directly within the notebook's resource folder and environment resource folder in notebook. Supported file types include CSV, TXT, HTML, YML, PY, SQL, and more. With the file editor, you can: View and edit files: Easily access and modify files within the notebook. Keyword highlighting for module files: Provide necessary language service when opening and editing code files like .py and .sql. Enhanced user experience: Integrated with the notebook pane switcher to help you easily navigate different features inside notebook. Watch the Data Science demos Real-time Intelligence Real-Time Intelligence is a powerful service that empowers everyone in your organization to extract insights and visualize their data in motion. With capabilities across ingestion, processing, transformation, analytics, visualization and action, Real-Time Intelligence transforms your data into a dynamic, actionable resource that drives value across the entire organization. Learn more here. Creating a Real time Dashboard by Copilot From the list of tables in Real-Time hub, users can click on the three dots menu and select create real-time dashboard. Copilot will review the table and automatically create a dashboard with two pages, one with insights about the data in the table and one page that contains a profile of the data with a sample, the table schema and more details about the values in each column. This can be further explored and edited to make it easy for users to find insights on their time-series data without having to write a single line of code. Insight page: Profile page: Adding a real-time dashboard to an org app Real-time dashboards can be included in the new version of org app. The org app can include PBI reports, PBI paginated reports, Notebooks and Real time dashboards. This makes it very easy to distribute RTD to large user cohorts outside a specific Workspace. An org app with a PB report and an RTD: Introducing A New Real-Time Hub User Experience We are thrilled to announce the launch of the new Real-Time Hub user experience (UX), a redesigned and enhanced experience that will help you get the most out of your real-time data. Whether you want to monitor, process, analyze, or act on your data streams, the new UX will make it easier and faster than ever. What's new? A new left navigation provides you with seamless access to all your data-in-motion. This includes both the data streams that you have access to and the ones that you have integrated into Fabric, as well as Fabric events and Azure resources. With just a few clicks, you can now effortlessly navigate through these resources in a more user-friendly way. A new page called “My Streams” where you can bring in external events from various sources and integrate them with your real-time data. You can use My Streams to create custom streams that combine data from different sources, including Microsoft sources, Database CDC, and Fabric events. “Get Events” button is now rebranded as “Add source”, which aligns with the value proposition of Real-Time hub to allow customers to not only discover, manage, and consume events, but also all the other types of data-in-motion. Four new Eventstream connectors have been introduced into the Real-Time hub. Now you can stream data from Azure SQL MI DB (CDC), SQL Server on VM DB (CDC), Apache Kafka, and Amazon MSK Kafka. These sources will enable you to enrich your real-time data with historical and analytical data from your data warehouse or database. To learn more about these four connectors, continue reading 'New Stream Sources Now Available' below. Simply sign in to Microsoft Fabric and start experiencing the new Real-Time hub UX! New Streaming Sources Now Available via Eventstream Connectors We’re excited to announce the expansion of our Eventstream Connectors with four powerful new streaming sources, enabling even more seamless data streaming across your ecosystem: Azure SQL MI CDC: Capture data changes in your Azure SQL Managed Instance (MI) database and stream them into Eventstream for real-time processing, analysis, and monitoring. SQL Server on VM CDC: Capture data changes in your SQL Server on Virtual Machines and stream them directly into Eventstream for processing, analysis, and monitoring. Apache Kafka: Easily stream data from Apache Kafka cluster into your Eventstream, enabling unified processing and analysis across platforms. Amazon MSK Kafka: Seamlessly stream data from Amazon Managed Streaming for Apache Kafka (MSK) into Eventstream and perform real-time processing. These new sources empower you to build richer, more dynamic Eventstreams, ensuring that your real-time streaming applications can fully leverage the reporting and analysis capabilities in Fabric. Introducing Eventhouse as a new Destination in Eventstream As part of our ongoing efforts to enhance Eventstream’s capabilities, we are excited to introduce Eventhouse as a new destination for your data streams. Eventhouses, equipped with KQL Databases, are designed to handle and analyze large volumes of data, particularly in scenarios that demand real-time analytics and exploration. With the Eventhouse destination in Eventstream, you can efficiently process and route data streams into an Eventhouse and analyze the data in near real-time using KQL. Eventstream’s Integration with Managed Private Endpoint We’re excited to introduce the Private Network feature for Fabric Eventstream! With Fabric’s Managed Private Endpoint, you can now establish a private connection between your Azure services, such as Azure Event Hub, and Fabric Eventstream. This integration ensures your data is securely transmitted within a private network, allowing you to explore the full power of real-time streaming and high-performance data processing that Eventstream offers. The diagram below shows how Eventstream pulls data from your Azure Event Hub within a virtual network using Fabric’s Managed Private Endpoint. To learn more about Managed Private Endpoint, visit here: Overview of managed private endpoints for Microsoft Fabric - Microsoft Fabric | Microsoft Learn Introducing the new look and feel of KQL Database We’re excited to unveil the newly redesigned KQL DB experience in Microsoft Fabric Real Time Intelligence! This update brings a clean, modern interface packed with powerful features to help you work smarter, whether you’re managing several databases, dealing with streaming data, or exploring massive datasets. A UI Tailored to Your Workflow The new KQL DB interface is built to accommodate the diverse needs of data professionals. Whether you're exploring a simple database or navigating one with hundreds of tables, the design remains intuitive and adaptable. An Enhanced Database Page Experience The new KQL database page has undergone a significant transformation. When you open the Tables tab of the database, you’re greeted with visually rich cards (tiles) that present key metrics briefly.4 These tiles display ingestion trends, data size, recent active users, and the availability status in OneLake. For those who prefer more detail, you can switch to the list view, offering a comprehensive table-focused perspective. Exploring your data is now more interactive with the updated Data Preview tab. Here, you can instantly see the most recent records across all tables in a convenient JSON view, allowing for quick insights without needing to dive too deep. If you’re looking to drill down further, the updated histograms and time filters let you see the overall data ingestion trend in more depth. Diving Deeper into Tables The individual table view has seen improvements, too. The Data Preview of a table lets you explore records with easy-to-use interface as the database-level preview. Also, the Schema tab gives you a complete picture of the data structure, including distribution stats and column details. Histograms in the table view bring your data to life, allowing you to explore different time ranges and customize binning to suit your analysis needs. All this is wrapped in a design that’s both accessible and easy to navigate. Flexible Views for Any Data Scope The new look and feel of KQL DB in Microsoft Fabric offers a more intuitive, powerful, and visually engaging way to work with your data. We can’t wait for you to try it out and see how these enhancements elevate your experience! To read more about the new KQL DB look and feel, check out the documentation. Set alerts on KQL Querysets with Data Activator triggers Now you can set up Data Activator alerts directly on your KQL queries in KQL querysets. Whether you’re tracking real-time metrics or keeping an eye on important logs, this feature makes sure you get notified the moment something important happens. Set Alerts Based on KQL Query Results or Specific Conditions With this new feature, you can set alerts to trigger based on specific results or conditions from a scheduled KQL query. For example, if your KQL DB tracks application logs, you can configure an alert to notify you if the query, scheduled at a frequency of your choice (e.g., every 5 minutes), returns any logs where the message field contains the string "error". This feature also lets you monitor live data trends by setting conditions on visualizations, similar to how you can set alerts on visuals within Real-Time Dashboards. For instance, if you visualize sales data distribution across product categories in a pie chart, you can set an alert to notify you if the share of any category drops below a certain threshold. This helps you quickly identify and address potential issues with that product line. You can choose whether to receive alerts via email or Teams messages when the condition is met. To read more about setting alerts for KQL querysets, check out the documentation. Data Exploration made easier with Top values feature The "Top Values" feature in our no-code data exploration UI is available for Real-Time Dashboards’ tiles! This new component makes it easier than ever to explore and understand your data without writing a single line of code. Instantly view the top values for each column in your query result set, allowing you to see value distributions and gain insights into the nature of your data. Plus, seamlessly add new filters directly from this component to refine your analysis. Real-Time Dashboard lower than ever refresh rate We are pleased to share an enhancement to our dashboard auto refresh feature, now supporting continuous and 10 seconds refresh rates, in addition to the existing options. This upgrade, addressing a popular customer request, allows both editors and viewers to set near real-time and real-time data updates, ensuring your dashboards display the most current information with minimal delay. Experience faster with data refresh and make more timely decisions with our improved dashboard capabilities. As the dashboard author you can enable the Auto refresh setting and set a minimum time interval, to prevent users from setting an auto refresh interval smaller than the provided value. Note that the Continuous option should be used with caution. The data is refreshed every second or after the previous refresh is completed if it takes more than 1 second. Multivariate anomaly detection Now you can use the power of Eventhouse, Spark, and OneLake to perform real-time monitoring and multivariate anomaly detection of time series data. We added a workflow that is based on the algorithm that is used in the AI Anomaly Detector service (which is being retired as a standalone service). Multivariate anomaly detection is a method of detecting anomalies in the joint distribution of multiple variables over time. This method is useful when the variables are correlated, thus the combination of their values at specific times might be anomalous, while the value each variable by itself is normal. Multivariate anomaly detection can be used in various applications, such as monitoring the health of complex IoT systems, detecting fraud in financial transactions, identifying unusual patterns in network traffic and more. Multivariate anomaly detection in Fabric takes advantage of the powerful Spark and Eventhouse engines on top of a shared persistent storage layer. The initial data can be ingested into an Eventhouse and exposed in OneLake. The anomaly detection model can then be trained using the Spark engine. The trained model is stored in Fabric MLflow models registry. Predictions of anomalies on new streaming data can be done in real time using the Eventhouse engine. The interconnection of these engines that can process the same data in the shared storage allows for a seamless flow of data from ingestion, via model training, to anomalies prediction. For more details see an overview and tutorial. Enhanced Time Series Analysis: More Control, Better Insights Whether you are using Real-Time Dashboards or KQL Querysets for time series analysis, you’ll find this new way of interacting with your data visualization valuable. You can now interact with the data by selecting specific items from the legend using the mouse, using Ctrl to add or remove selections, or holding Shift to select a range. The Search button helps you quickly filter items, while the Invert button allows you to reverse your selections. Navigate through your selections with ease using the Up and Down arrows to refine your data view. Real-Time Intelligence Copilot conversational mode We'd like to share an upgrade to our Copilot assistant, which translates natural language into KQL. Now, the assistant supports a conversational mode, allowing you to ask follow-up questions that build on previous queries within the chat. This enhancement enables a more intuitive and seamless data exploration experience, making it easier to refine your queries and dive deeper into your data, all within a natural, conversational flow. New Rule Creation Experience We’re excited to announce the launch of a new rule creation experience within Data Activator. This innovative experience unlocks new capabilities to monitor your events with more fine-tuned controls at greater scale, ensuring you solve and stay on top of your operational business challenges with ease. The new unified experience allows you to easily bring in your data, identify the business object you want to monitor, quickly and easily create business rules on top of them, and automatically carry out actions when the rule is met. This feature is not only a new and improved experience with the same functionality as is available today, but also lays the groundwork for new modeling capabilities in the future. With this update you’ll be able to: Stay on top of your critical metrics by monitoring your business objects. You can track and analyze key business objects such as individual packages, households, refrigerators, and more in real-time, ensuring you have the insight needed to make informed decisions. Whether it’s understanding how individual instances of your business objects impact sales figures, inventory levels, or customer interactions, our monitoring system provides detailed insights, helping you stay proactive and responsive to changes in your business environment at a fine-tuned level of granularity. Unlock the full potential of creating business rules on your data with advanced data filtering and monitoring capabilities. This update offers a wide array of options for filtering, summarizing, and scoping your data, allowing you to tailor your analysis to your specific needs. You can set up complex conditions to track when data values change, exceed certain thresholds, or when no new data has arrived within a specified timeframe. Ensure your communications are perfect before hitting send by seeing a preview of your Email and Teams messages. This will allow you to see a preview of your message exactly as it will appear to the recipient. Review your content, check formatting, and make any necessary adjustments to ensure clarity. With this feature, you can confidently have Data Activator send messages on your behalf knowing they look just the way you intended. Set up rules that trigger automatically with every new event that comes in on your stream of data. Whether you need to send notifications or initiate workflows, this feature ensures that your processes are always up-to-date and responsive. These are just some of the highlights the new Data Activator rule creation experience provides, but there is much more to explore. To learn more about how this update can help you create and manage your business rules more easily and efficiently, check out our detailed blog post that covers all the features and benefits in depth. We’ve made it easier to alert your teammates in Power BI We’ve streamlined the process of creating Data Activator alerts for your teammates in Power BI. Previously, you had to create your alert in Power BI, then open it in Data Activator to add your teammates as recipients. Now, you can add recipients directly within Power BI. We’re rolling this out gradually during the coming few weeks, so look out for it during the month of September when you click “Set Alert” in Power BI. For the latest updates and what's coming in Real-Time Intelligence check our out documentation and roadmap. Watch the Real-Time Intelligence demos Data Factory Dataflow Gen2 Copilot in Dataflow Gen2 (generally available) Earlier this year, we released a Public Preview of the Copilot for Data Factory capabilities in Dataflows Gen2. Today, we are excited to announce that these capabilities are now Generally Available. Copilot in Fabric enhances productivity, unlocks profound insights, and facilitates the creation of custom AI experiences tailored to your data. As a component of the Copilot in Fabric experience, Copilot in Data Factory empowers customers to use natural language to articulate their requirements for creating data integration solutions using Dataflow Gen2. Essentially, Copilot in Data Factory operates like a subject-matter expert (SME) collaborating with you to design your dataflows. Copilot for Data Factory is an AI-enhanced toolset that supports both citizens and professional data wranglers in streamlining their workflow. It provides intelligent Mashup code generation to transform data using natural language input and generates code explanations to help you better understand earlier generated complex queries and tasks. Learn more about how to get started with Copilot for Data Factory: Copilot for Data Factory overview - Microsoft Fabric | Microsoft Learn. Fast Copy in Dataflow Gen 2 (general available) The Fast Copy feature is now generally available. This functionality enables efficient ingestion of large amounts of data, utilizing the same backend as the Copy Activity in data pipelines. It significantly reduces data processing time and enhances cost efficiency. Learn how to enhance performance and cut costs with Fast Copy in Dataflows Gen2. Fast Copy supports numerous source connectors, including ADLS Gen2, Blob storage, Azure SQL DB, On-Premises SQL Server, Oracle, Fabric Lakehouse, Fabric Warehouse, PostgreSQL, and more. An on-premises gateway is also supported, allowing for high-performance data transfer from on-premises sources. For details, see Fast Copy with On-premises Data Gateway Support in Dataflow Gen2. Resources: Docs: Fast copy in Dataflows Gen2 - Microsoft Fabric | Microsoft Learn Incremental refresh for Dataflow Gen2 We are happy to share the introduction of incremental refresh for Dataflows Gen2. This significant enhancement in Microsoft Fabric’s Data Factory is designed to optimize data ingestion and transformation, particularly as your data continues to expand. About Incremental Refresh Incremental refresh enables you to refresh only new or updated data, thus reducing refresh times, enhancing reliability by avoiding long-running operations, and minimizing resource usage. This feature is particularly valuable in scenarios where data volume is substantial, and efficiency is required. Prerequisites To leverage incremental refresh in Dataflows Gen2, ensure the following prerequisites are met: You must have a Fabric capacity. Your data source should support query folding and must include a Date/DateTime/DateTimeZone column for data filtering. Your data destination must support incremental refresh. Supported destinations include Fabric Warehouse, Azure SQL Database, and Azure Synapse Analytics. How to Use Incremental Refresh Create a new Dataflow Gen2 or open an existing one. In the dataflow editor, create a new query that retrieves the data you need to refresh incrementally. Ensure your query returns data with a Date/DateTime/DateTimeZone column for filtering. Verify that the query fully folds to the source system. Right-click the query and select Incremental refresh. Provide the required settings, including the DateTime column, data extraction parameters, and bucket size. Configure any advanced settings if necessary. Publish Dataflow Gen2. Resources: Incremental Refresh Docs Incremental Refresh Blog Certified connector updates Check out the new and updated connectors in this release: New connectors: Kyvos Clickhouse Updated connectors: InformationGrid Are you interested in creating your own connector and publishing it for your customers? Learn more about the Power Query SDK and the Connector Certification program. Watch the Data Factory demo Data pipeline Fabric Pipeline Integration in On-premises Data Gateway (generally available) On-premises connectivity for Data pipelines in Microsoft Fabric is now Generally Available. Using the on-premises Data Gateway, customers can connect to on-premises data sources using data pipelines with Data Factory in Microsoft Fabric. This enhancement significantly broadens the scope of data integration capabilities. In essence, by using an on-premises Data Gateway, organizations can keep databases and other data sources on their on-premises networks while securely integrating them into Microsoft Fabric (cloud). Learn more: How to access on-premises data sources in Data Factory - Microsoft Fabric | Microsoft Learn Invoke remote pipeline in Data pipeline We’ve been working diligently to make the very popular data pipeline activity known as “Invoke Pipeline” better and more powerful. Based on customer feedback, we continue to iterate on the possibilities and have now added the exciting ability to call pipelines from Azure Data Factory (ADF) or Synapse Analytics pipelines as a public preview! This opens tremendous possibilities to utilize your existing ADF or Synapse pipelines inside of a Fabric pipeline by calling it inline through this new Invoke Pipeline activity. Using cases that include calling Mapping Data Flows or SSIS pipelines from your Fabric data pipeline will now be possible. We will continue to support and include the previous Invoke Pipeline activity as “legacy”, without the support for ADF or Synapse remote pipeline invocation and without child pipeline monitoring in Fabric. But for the latest features like remote invocation and child pipeline monitoring, you can use the new Invoke Pipeline. Spark Job environment parameters One of the most popular use cases in Fabric Data Factory today is automating and orchestrating Fabric Spark Notebook executions from your data pipelines. A very common request has been to reuse existing Spark sessions to avoid any session cold-start delays. And now we’ve delivered on that requirement by enabling “Session tags” as an optional parameter under “Advanced settings” in the Fabric Spark Notebook activity! Now you can tag your Spark session and then reuse the existing session using that same tag. Watch the Data pipeline demos Mirroring Mirroring Azure SQL Database Mirroring Azure SQL Database in Fabric now extends our capabilities to support mirrored tables within additional SQL’s Dynamic Definition Language (DDL). Now, operations such as Drop Table, Rename Table, and Rename Column can be seamlessly executed while tables are in the process of mirroring. Click here to watch a demo. New Azure Data Factory Item Bring your existing Azure Data Factory (ADF) to your Fabric workspace! We are introducing a new preview capability that allows you to connect to your existing ADF factories from your Fabric workspace. By clicking “Create Azure Data Factory” inside of your Fabric Data Factory workspace, you will now be able to fully manage your ADF factories directly from the Fabric workspace UI! Once your ADF is linked to your Fabric workspace, you’ll be able to trigger, execute, and monitor your pipelines as you do in ADF but directly inside of Fabric. Copy Job (public preview) We'd like to introduce Copy Job, elevating the data ingestion experience to a more streamlined and user-friendly process from any source to any destination. Now, copying your data is easier than ever before. Moreover, Copy job supports various data delivery styles, including both batch copy and incremental copy, offering flexibility to meet your specific needs. With Copy Job, you can enjoy the following benefits: Intuitive Experience: Seamless experience data copying with no compromises, making it easier than ever. Efficiency: Enable incremental copying effortlessly, reducing manual intervention. This efficiency translates to less resource utilization and faster copy times. Flexibility: Empower yourself to customize your data copying behavior. From selecting tables and columns to data mapping and read/write behavior, you have the flexibility to tailor the process to your specific needs. Additionally, enjoy the freedom to set flexible schedules, whether it's a one-time copy or at a regular cadence. Click here to learn more about Copy Job. Watch the Mirroring demosMicrosoft Fabric 2025 holiday recap: Unified Data and AI Innovation
As 2025 ends, we’re taking a moment to reflect on Microsoft Fabric’s second year in the market and the collective progress made alongside our community, customers, and partners. What began as a unified vision for data and AI has grown into a platform adopted by more than 28,000 organizations worldwide, anchored by OneLake and shaped through continuous collaboration and feedback. This year brought meaningful advancements across security, modernizations, professional developer tooling, and more—strengthening Fabric’s role as a trusted foundation for modern data and AI solutions. As we look ahead to 2026, we’re encouraged by the momentum built over these first two years and grateful for everyone who has contributed to Fabric’s evolution. Below, we revisit the highlights that defined 2025. Fabric Data Days We extend our sincere thanks to everyone who has taken part in the challenges for Fabric Data Days—a global, community‑driven skilling initiative offering, hands‑on activities, certification preparation, and expert‑led sessions. Your creativity, energy, and outstanding work highlights the remarkable talent and innovation across the Fabric community. Congratulations to Lahiru Wimalarantha, winner of the Notebooks Contest – Cities of Tomorrow; Juan Bohorquez and Pierre Dieng, winners of the Dataviz Contest for Pros; and Nayara Hellen, winner of the Student Dataviz Contest. Running through December 31, 2025, Fabric Data Days also offers exclusive certification vouchers, including 100% discounts for DP‑600 and DP‑700 and 50% discounts for PL‑300 and DP‑900 (available to eligible participants). Breakthrough advancements 2025 was a defining year for Microsoft Fabric, marked by several highly anticipated innovations that expanded the platform’s security, modernization capabilities, AI integrations, and professional developer experience. These capabilities strengthened Fabric’s position as a unified data and AI platform, offering customers greater operational control, more streamlined modernization paths, and improved tooling for building and managing data and AI solutions. The highlights below outline some of the most impactful advancements introduced throughout the year. Strengthening platform security Security continued to be a core area of investment in 2025, with several enhancements introduced to strengthen network isolation and data protection across the Fabric platform. Outbound Access Protection reached general availability, giving organizations the ability to govern and restrict outbound connections, ensuring that data movement aligns with corporate compliance and governance policies. Private links for Fabric workspaces also became generally available, enabling Fabric workspaces to be accessed exclusively through private Azure endpoints and ensuring that all traffic remains within a customer’s trusted network boundary. This year also marked the public preview of OneLake security, introducing a unified, role‑based model for controlling access to data stored in OneLake. This approach allows organizations to define granular permissions that apply consistently across Fabric’s compute engines, improving alignment between governance requirements and day‑to‑day operations. Together, these capabilities reinforce Fabric’s security posture and its commitment to providing a secure, enterprise‑ready foundation for modern workloads. Streamlining the modernization path Modernizing data estates remained a top priority for customers looking to take advantage of Fabric’s integrated platform, and in 2025 Microsoft introduced new capabilities to make that modernization more efficient and outcome‑focused. The Migration Assistant for Fabric Data Warehouse reached general availability this year, providing a guided experience for organizations modernizing analytical workloads—such as those running on Azure Synapse dedicated SQL pools—into Fabric’s lake‑native, fully integrated environment. By simplifying the process of bringing forward existing schemas and workloads, the assistant enables organizations to more quickly benefit from Fabric’s elastic performance model, unified governance, and the efficiency of building directly on OneLake. To complement seamless modernization efforts, Fabric also introduced the Azure Data Factory pipeline assessment (Preview), giving organizations clearer insight into how their existing pipelines align with Fabric Data Factory. The assessment helps teams identify which orchestration assets are ready to modernize and when it may be appropriate to continue using Azure Data Factory or Azure Synapse Analytics to copy, transform, and land data in Fabric lakehouses or warehouses until full pipeline feature parity becomes available. Together, these capabilities reflect Fabric’s commitment to providing customers with practical, low‑friction onramps that enable organizations to build toward a unified, scalable, and future‑ready data foundation. AI and Copilot: Expanding AI access 2025 brought several major AI announcements, most notably Copilot being made available across all paid SKUs—making generative AI accessible to every user and every workload. Copilot now streamlines tasks across the platform: generating code in notebooks, offering in‑line SQL completions, creating KQL queries through a chat pane, assisting with no-code data preparation, troubleshooting orchestration pipelines, enabling conversational data exploration, and more. These capabilities accelerate work across Data Engineering, Data Science, Data Factory, SQL, Real‑Time Intelligence, and Power BI, helping users move from intent to impact faster. The general availability of AI functions further expands access to AI by allowing both pro‑code and no‑code developers to apply advanced operations—such as summarization, extraction, and classification—directly within their solutions. Whether it’s a line of code in a notebook or a single step in Dataflow Gen2, AI‑powered transformation is now easier and more approachable than ever. Together, these advancements make AI more deeply integrated, broadly accessible, and increasingly impactful within organizational solutions. Empowering professional developers For developers, Fabric introduced a series of enhancements designed to improve productivity and support more sophisticated development workflows. The platform’s web experience advanced significantly, adding IDE‑like capabilities such as multitasking tabs, an object explorer, and a dedicated focus mode that streamlines development and reduces visual distractions. Beyond the browser, Fabric strengthened its command‑line tooling with the general availability of the Fabric CLI, enabling developers to navigate and manage Fabric resources programmatically and integrate them more easily into CI/CD pipelines. Fabric also broadened its support for familiar environments through integrations with VS Code (Microsoft Fabric and Microsoft Fabric MCP Server extensions) and continued compatibility with SQL Server Management Studio (SSMS), allowing developers to work within the tools they already use. Additionally, Fabric’s DevOps and release automation capabilities grew meaningfully this year: Terraform support for Fabric enables declarative provisioning and governance of Fabric artifacts, while the fabric-cicd deployment tool standardizes environment promotion, approvals, and repeatable releases across projects—simplifying workspace topology, configuration, and multi‑stage deployments at scale. Together, these investments reduce developer friction, improve consistency across tooling, and strengthen Fabric as a platform for rigorous engineering experiences. And more… While we highlighted only a handful of updates, we encourage you to subscribe to the Fabric blog feed—or use your preferred RSS reader—from the righthand menu of the Microsoft Fabric Blog. It’s the easiest way to stay informed about what’s coming next, especially as we look toward 2026, and the many improvements that will continue to shape the product and excite our community. Join us in Atlanta for FabCon + SQLCon 2026 Mark your calendar for March 16–20, 2026, when FabCon and SQLCon come together in Atlanta, Georgia for an unforgettable week that unites the entire data community. From Microsoft Fabric developers to SQL Server and Azure SQL professionals, this co‑located experience brings everyone under one roof to learn, connect, and explore the full power of Fabric, Power BI, SQL Server, Azure SQL, Real‑Time Intelligence, AI, data engineering, and analytics. With inspiring keynotes at State Farm Arena, deep‑dive workshops, more than 200 technical sessions, a buzzing expo hall, and a vibrant community lounge at the Georgia World Congress Center, there’s truly something for every data professional. Even better, one registration grants full access to both FabCon and SQLCon— giving you the freedom to mix sessions, meet experts, and engage with peers across the entire Microsoft data ecosystem. With SQLCon now officially integrated, attendees will be immersed into the latest innovations across SQL Server and Azure SQL, including roadmap updates, performance tuning, security advancements, and AI‑powered applications. Whether you want to sharpen your skills, expand your career, or simply reignite your passion for data, FabCon + SQLCon 2026 is the place to be. We can’t wait to welcome you to Atlanta as we continue building the future of data together. Register now with code FABCOMM to get $200 off registration! What’s Ahead in 2026! As we wrap up 2025, we want to thank you—our incredible community, customers, and partners—for being a part of Fabric’s journey. Your enthusiasm, ideas, and collaboration have helped us build a strong foundation in these first two years: one that unifies disparate data services into a single, cohesive platform, with rapid innovation, growing adoption, and openness guiding Fabric’s continued evolution. We’re excited to keep shaping the future of data and AI together. Happy holidays—and we’ll see you in 2026 with even more to come.Microsoft Fabric August 2024 Update
Welcome to the August 2024 Update. Here are a few, select highlights of the many we have for Fabric. V-Order behavior of Fabric Warehouses allows you to manage the V-Order behavior at the warehouse level. Monitor ML Experiments from the Monitor Hub allows you to integrate experiment items into Monitoring Hub with this new feature. You can easily browse and connect to your Azure resources automatically with the modern data experience of Data Pipeline. There is more to explore, please continue to read on. European Fabric Community Conference Join us at Europe’s first Fabric Community Conference, the ultimate Power BI, Fabric, SQL & AI learning event in Stockholm, Sweden from September 24 -27, 2024. With 120 sessions, daily keynotes, 10 pre-conference workshops, an expo hall with community lounge, and “ask the expert” area, the conference offers a rich learning experience you don’t want to miss. This is a unique opportunity to meet the Microsoft teams building these products, customers betting their business on them, and partners at the forefront of deployment and adoption. Register today using code MSCUST for an exclusive discount on top of early bird pricing! Attention Power BI users! If you are accessing Power BI on a web browser version older than Chrome 94, Edge 94, Safari 16.4, Firefox 93, or equivalent, you need upgrade your web browser to a newer version by August 31, 2024. Using an outdated browser version after this date, may prevent you from accessing features in Power BI. Contents https://youtu.be/36U44YK68G0?si=_k5aMYvPG2oeDCYx Copilot and AI Ask Copilot questions against your semantic model (preview) You can now ask Copilot for data from your entire semantic model in Desktop! Just tell Copilot what you’re looking for, and Copilot will query your model to answer your question with a visual. Since the Copilot pane in Desktop is still in preview, you do not need to turn on the preview toggle to use this new capability. To find out more about how this feature works and the types of questions that are supported check out our previous blog post and documentation page. Reporting Visual level format strings (preview) Visual level format strings are here, providing you with more options to configure formatting. Originally built for visual calculations, the core ability that visual-level format strings provide is the ability to format visual calculations. Since visual calculations are not in the model, you could not format them, unless you were using them in data labels or in specific parts of the new card and new slicer visuals. With visual level format strings, you can! Visual level format strings, however, are useful even without using visual calculations. With the introduction of visual-level format strings, Power BI now has three levels for format strings: Model. You can set a format string for columns and measures in the model. Anywhere you use that column or measure the format string will be applied, unless it’s overridden by a visual or element level format string. Visual. This is what we’re introducing today. You can set format strings on any column, measure or visual calculation that is on your visual, even if they already had a format string. In that case the model level format string will be overridden, and the visual level format string is used. Element. You can set a format string for data labels and for specific elements of the new card and the new slicer visuals. This level will be expanded to include much more in the future. Any format string you set here will override the format string set on the visual and model level. These levels are hierarchical, with the model level being the lowest level and the element level the highest. A format string defined on a column, measure or visual calculation on a higher-level override what was defined on a lower level. Since visual calculations are not in the model, they cannot have a format string set on the model level but can on the visual or element level. Measures and columns can have format strings on all three levels: Level Impacts Available for Measures, columns Visual calculations ELEMENT Selected element of the selected visual X X Visual Selected visual X X Model All visuals, all pages, all reports on the same model X The image below summarizes this and shows that higher level format strings override lower-level format strings: Take a look at this example using a measure. There is a Profit measure in the model, which is set to a decimal number format. To do this, you might have set the formatting for this measure using the ribbon: Alternatively, you could have made the same selections in the properties pane for the measure in the model view or entered the following custom formatting code: If you put this measure on a visual it now returns a decimal number, as expected: However, on a particular visual you want that measure to be formatted as a whole number. You can now do that by setting the format code on the visual level by opening the format pane for that visual and the Data format options found there under General: Now that same measure shows as a whole number, but just on that visual: On top of that, you might want to use a scientific notation for that measure but only in the data label on a particular visual. No problem, you set the format code on the data label for that measure: Now the total shows in scientific notation, but only in the data label and not in other places (such as the tooltip as shown below). Notice how the element level format is used in the data label but the visual or model level format string is still used for the other elements in the same visual. For visual calculations the same principle applies but of course without the model level. For example, if you have a visual calculation that returns a percentage, you can now format it as such using the Data Format options in the General on the visual in the format pane: The ability to set visual level format strings makes it much easier to get the exact formatting you need for your visualizations. However, this is only the first iteration of the visual level format strings. We are planning to add the settings you’re used to for the model level format strings to the visual level soon. Since visual level format strings are introduced as part of the visual calculations preview, you will need to turn on the visual calculations preview to use them. To do that, go to Options and Settings > Options > Preview features. Select Visual calculations and select OK. Visual calculations and visual level format strings are enabled after Power BI Desktop is restarted. Please refer to our docs to read more about format strings or visual calculations. Dynamic per recipient subscriptions (Generally Available) Dynamic per recipient subcriptions are now generally available for Power BI and paginated reports. Dynamic per recipient subscriptions is designed to simplify distributing a personalized copy of a report to each recipient of an email subscription. You define which view of the report an individual receives by specifying which filters are applied to their version of the report. The feature is now available in Sov. Clouds as well. Create a dynamic per recipient with a simple drag and drop experience. First, subscribe to the report by selecting “Subscribe to report”, then “Create Subscriptions”. Select “Dynamic per recipient” subscription. Connect to data that has recipient email, names or report parameters. Then, select and filter data that you want in your subscription. You probably only want to send emails conditionally. To do that, you can filter the data in the “Filter” pane. You can select the recipient email addresses and the email subject from the dataset that you connected to by selecting “Get Data”. Then map your data to the subscription. Next, schedule the subscription and save it. The subscriptions will be triggered based on the schedule that you have set up. Personalized reports can be sent to up to a thousand recipients! Learn more about Dynamic per recipient subscriptions for Power BI reports, and paginated reports. Deliver subscriptions to OneDrive and SharePoint (Generally Available) Do you have reports that are too large to be delivered by email? Do you have reports that are eating into your email in just a few weeks, or do you need you to move it to a different location? You can now deliver Power BI and paginated report subscriptions to OneDrive or SharePoint. With this capability, you can schedule and send full report attachments to a OneDrive or SharePoint location. Learn more about how to deliver report subscriptions to OneDrive or SharePoint. Updated Save and Upload to OneDrive Flow in Power BI Beginning the first weeks of August, desktop users should see a preview switch starting in SU8 to turn on the updated Save and Upload to OneDrive experience in Power BI. To enable this, navigate to the Preview features section of Options in Power BI. Users will then need to select “Saving to OneDrive and SharePoint uploads the file in the background”. With these updates, we’ve improved the experience of uploading new Power BI files to OneDrive, and easily upload new changes in the background. Preview switch that needs to be selected. For uploading new files, after navigating to the correct location in the OneDrive file picker and saving, a dialog box appears while the file is being uploaded. The option to cancel the upload is there if needed. This dialog will only show up the first time a new file is uploaded to OneDrive. Dialog for saving a new file to OneDrive. When new changes are saved to a file uploaded to OneDrive, the top of the toolbar indicates that the new changes are also being uploaded to OneDrive. ric_August_2024_Update Additional changes being uploaded in the background to the existing file. If you click on the title bar flyout in the toolbar, you can also now access more information about the file. Clicking “View your file in OneDrive” will provide a direct link to where the file is stored in OneDrive. Drop down including the link to the file in OneDrive. Data limit We are introducing the data limit capability to help you manage performance issues. This feature allows you to set the maximum data load for a single session per visual displaying only the rows of data in an ascending order by default. To use this feature: Go to the 'Filters on this visual' menu in the filter pane. Click on the data limit menu to open a new filter card. 3. Set your desired data limit value. The filter card features include: Removing, locking, or clearing filters. Hiding or showing filters. Expanding or collapsing filter cards. Applying filters. Renaming and reordering filters. Report consumers can see any data limits applied to a visual in the filter visual header, even if the filter pane is hidden. Visuals, shapes and line enhancements Over the past few months, we have been fine-tuning the visual elements of your reports, including columns, bars, ribbons, and lines. We have given you the ability to craft these Cartesians with precision. However, we noticed that the legends and tooltips were not quite accurate. With the latest update, the legend and tooltip icons will now automatically and accurately reflect per-series formatting settings, such as border colors, shapes, and line styles. This makes it easier to match series to their visual representations. Additionally, we have added consistency to how per-series formatting is applied to line charts, column/bar charts, scatter charts, and other Cartesian formatting options for common items like error bars and anomalies. Modeling DAX query view in the web Write DAX queries on your published semantic models with DAX query view in the web. DAX query view, already available in Power BI Desktop, is now also available when you are in the workspace. Look for Write DAX queries on your published semantic model. Right-click on the semantic model and choose Write DAX queries. Click on the semantic model to open the details page, then click Write DAX queries at the top of the page. This will launch DAX query view in the web, where you can write DAX queries, use quick queries to have DAX queries written for you on tables, columns, or measures, or use Fabric Copilot to not only write DAX queries but explain DAX queries, functions, or topics. DAX queries work on semantic models in import, DirectQuery, and Direct Lake storage mode. Write permission, that is permission to make changes to the semantic model, is currently needed to write DAX queries in the web. And, the workspace setting, User can edit data models in the Power BI service (preview), needs to be enabled. DAX query view in the web includes DAX query view’s way to author measures. Define measures with references, edit any of them, and try out changes across multiple measures by running the DAX query, then update the model with all the changes in a single click of a button. DAX query view in web brings this functionality for the first time to semantic models in Direct Lake mode! If you do not have write permission, you can still live connect to the semantic model in Power BI Desktop and run DAX queries there. Try out DAX query view in web today and learn more about how DAX queries can help you in Power BI and Fabric. Deep dive into DAX query view in web DAX queries Work with DAX query view Deep dive into DAX query view and writing DAX queries Write DAX queries with Copilot Deep dive into DAX query view with Copilot Overview of Copilot for Power BI Direct Lake Embedded Analytics Narrative visual with Copilot available in SaaS embed Narrative visual with Copilot is available for user owns data scenarios (SaaS) and secure embed. When a user embeds a report containing the narrative visual in a solution where users must sign in – they will now be able to refresh the visual with their data. The first step on our Copilot embed journey! When you embed a Power BI report in an application in the "embed for your organization" scenario, it allows organizations to integrate rich, interactive data visualizations seamlessly into their internal tools and workflows. Now this solution supports the Copilot visual. A sales team might want to embed a Power BI report in their internal CRM application to streamline their workflow. By integrating sales performance dashboards directly into the CRM, team members can easily monitor key metrics like monthly sales targets, pipeline status, and individual performance, without switching between different tools. This integration enables quicker access to actionable insights, helping the team make informed decisions, identify trends, and react swiftly to market changes, all within the secure environment of their organization's data infrastructure. Supported Scenarios: Embed a report in a secure portal or website Power BI. User owns data: A user embeds a report containing the narrative visual in a solution where users must sign in. They need a license to do so. This action is also known as embed for your organization. It includes when users want to embed visuals in solutions like PowerPoint as well. Unsupported Scenario: App owns data: A customer embeds a narrative visual on a website where users visit, and don't need to sign in. Also known as embed for your customer's application. To get this set up, there are a few steps to follow - so make sure to check out the documentation. You will need to Edit your Microsoft Entra app permissions to enable the embedded scenario to work. From here you’ll need to add the MLModel.Execute.All permission. Once completed, your visual should work in your embedded experiences where users still sign in. Check out the documentation for additional details. Other Paginated Reports: Sharing of reports connecting to Get Data data sources made easy We announced the ability to create paginated reports from Power BI Report Builder by connecting to over 100 data sources with the Get Data experience. You can learn more about Connect paginated reports to data sources using Power Query (Preview) - Power BI | Microsoft Learn. You no longer need to share the shareable cloud connection. You only need to share the report and ensure that those consuming the report have access to view the report. This update will be rolling out in the coming weeks. Core Workspace filter improvement to support nested folders Since folders were introduced in workspaces, it can be hard to find items sometimes because they are all hidden in the hierarchy. We have upgraded the filter experience to support filtering through the entire workspace or through a specific folder with all its nested folders. OneLake OneLake data access role improvements OneLake data access roles allow for granular security to be defined within a lakehouse. This month, we’ve updated data access roles based on key feedback. The assign roles page in the user interface has been redesigned to make it easier to understand access. The page now has the “Add people or groups” control front and center, with less emphasis on assigning users via item permissions. Item permissions control has been rebuilt for ease of use, including a new “user count” to show how many users will get added to the role based on the permissions you select. The members list now includes sort options and shows newly added users and groups with an icon to help validate pending changes. Schema support for lakehouses was announced last month, and OneLake data access roles now allows for defining security for schemas. Schema support follows the same inheritance model as folders, allowing multiple tables to be managed easily by securing the parent schema. To get started with schema level security, create a new lakehouse and check the “enable schemas” preview button, and then enable OneLake data access roles for that lakehouse. To learn more or get started, view the documentation on OneLake data access roles. Synapse Data Warehouse Managing V-Order behavior of Fabric Warehouses V-Order is a write-time optimization for the Parquet file format, enabling Direct Lake mode with Power BI. It applies special sorting and compression to Parquet files, offering query benefits but potentially increasing data ingestion time in Fabric Warehouse. We are excited to share a new feature in Fabric Warehouse that allows you to manage V-Order behavior at the warehouse level. You can now disable V-Order, providing better control over ETL performance when V-Order is unnecessary. This enhancement is particularly beneficial for scenarios where V-Order optimization may not offer significant advantages, such as in typical ETL processes with write-intensive staging tables. Disabling V-Order on a warehouse is a non-reversible operation. You should thoroughly test the performance of your warehouse queries and ETL processes to determine if this option is suitable for your scenario before disabling it. Currently, there is no way to re-enable V-Order. For more details and to determine if disabling V-Order is suitable for your scenarios, refer to the product documentation. Delta Lake Log Publishing Pause and Resume Fabric Warehouse publishes Delta Lake Logs for every table created in your Warehouses. Any modifications made to a Warehouse table will be reflected in the Delta Lake Log within a minute of the transaction being committed. This enables other analytical engines in Microsoft Fabric to read the latest data on user tables without any data duplication. This new feature allows you to pause and resume the publishing of Delta Lake Logs for Warehouses. When publishing is paused, Microsoft Fabric engines that read tables outside of the Warehouse will see the data as it was before the pause. This ensures that reports remain stable and consistent, reflecting data from all tables as they existed before any changes were made to the tables. It is especially beneficial if you have many tables and reports that are frequently changing, it will minimize the risk of data inconsistencies. Once your ETL/ELT process is complete, you can resume Delta Lake Log publishing to make all recent data changes visible to other analytical engines. To learn more about Delta Lake Log publishing pause and resume, refer to the product documentation. Alter Table add Nullable Column Alter Table add Nullable Columns is now generally available! This feature allows for adding new nullable columns to previously created Delta Parquet-backed tables in a warehouse in Microsoft Fabric. In the ever-evolving data landscape of data organizations, schemas are shifting and changing to keep up with the influx of new data. Whether your schema modifications are few and far between, or a regular occurrence that constantly needs to adapt to changing requirements, we have you covered. Our goal is to ensure that customers have everything they need for a seamless warehousing experience, and we continue to strive towards ensuring our TSQL surface area meets the needs of our customers. Note that today, only the following subset of ALTER TABLE operations in Warehouse in Microsoft Fabric are supported: ADD nullable columns of supported column data types. ADD or DROP PRIMARY KEY, UNIQUE, and FOREIGN_KEY column constraints, but only if the NOT ENFORCED option has been specified. All other ALTER TABLE operations such as drop / rename/ adding non nullable columns are currently blocked Here are some example syntaxes to get started with ALTER TABLE functionality. ALTER TABLE [AdventureWorksSales].[dbo].[Product] ADD [status] VARCHAR(50); -- any supported data types Learn more here. Truncate Table Truncate Table in Fabric Datawarehouse, enhancing data management capabilities for the users is now generally available. Truncate removes all rows from any warehouse user table that the user has permission to update, while preserving the table's metadata. This command is beneficial for situations that demand quick data removal and table maintenance. It is ideal for scenarios where staging tables need regular data clearing without altering the table's structure. Truncate operation is only allowed on a parquet backed Datawarehouse user table. There are few Limitations such as: Truncating a specific table partition is not allowed in Fabric Datawarehouse Truncation of a Lakehouse table, views, table referenced by materialized views, system tables, system DMVs are also not allowed. Syntax for truncate table to remove rows from a Parquet-backed user table in Microsoft Fabric Warehouse: TRUNCATE TABLE { database_name.schema_name.table_name | schema_name.table_name | table_name } [;] Learn more here Mirroring Azure SQL Database We’ve heard your requests and we're excited to extend our capabilities with support for mirrored tables within additional SQL’s Dynamic Definition Language (DDL). Now, operations such as Drop Table, Rename Table, and Rename Column can be seamlessly executed while tables are in the process of mirroring. Click here to watch a demo. Mirroring integration with modern get data experience You can use a more intuitive experience with Modern Get Data to connect data source for Mirroring. On the homepage of Data Warehouse experience, click any Mirroring module to get started. Let’s use the “Mirrored Azure SQL Database” to get started. After entering the “Mirrored Azure SQL Database”, you can use the Modern Get Data experience by choosing all the available databases in the OneLake data hub. More details of how to do this, you can take reference on this document Tutorial: Configure Microsoft Fabric mirrored databases from Azure SQL Database (Preview). Data Engineering Announcing MsSparkUtils upgrade to NotebookUtils Beginning in August, the library MsSparkUtils will be rebranded as NotebookUtils. This change reflects our commitment to providing the best tools and utilities for your data processing needs. While NotebookUtils will be backward compatible with MsSparkUtils, we want to inform you that new features will only be added to the NotebookUtils namespace. Therefore, we strongly recommend that you start replacing the old namespace with the new one in your projects, and the NotebookUtils will be only supported on runtime version 1.2 and above, please note to upgrade your runtime version as well. The transition to NotebookUtils is a step forward in our journey to enhance your experience and efficiency. We appreciate your cooperation in this change and are here to support you through the transition process. Thank you for your continued support and happy coding! Import Notebook UX improvement The Import Notebook feature has recently been enhanced making it more accessible and intuitive. With this update, you can effortlessly import notebooks, reports, or paginated reports using the unified entry in the workspace toolbar. This improvement streamlines the process, ensuring a seamless experience for developers. Fabric Runtime Lifecycle We recently released the Lifecycle of Apache Spark runtimes in Fabric. Our team diligently delivers new versions, ensuring they are of high quality, well-integrated, and supported continuously. Each version includes about 110 components, and as the runtime grows, we make sure it integrates smoothly into Microsoft Fabric and Azure. We cautiously approach new preview runtime releases, aiming for an experimental preview in roughly 3 months, although the exact timeline varies per case. The latest runtime, Version 1.3 based on Apache Spark 3.5, is currently in Public Preview, and we’re preparing it for General Availability (GA). Runtime 1.2, based on Apache Spark 3.4, is already stable in GA. We have announced the end-of-support date for Runtime 1.1 on Apache Spark 3.3, which will be deprecated and not available after March 31, 2025. Data Science Semantic Link Labs is now live! Recently, Michael Kovalsky released a python library called ‘fabric_cat_tools‘ for Fabric notebooks. This library includes 120+ additional functions which extend semantic-link’s capabilities ranging from automating the migration of semantic models to Direct Lake, analyzing semantic models via a Best Practice Analyzer, showing Vertipaq Analyzer statistics, wrapping the full Tabular Object Model…and much more. You can even automatically translate your entire semantic model’s metadata into any language in seconds! All of this is self-contained inside of the Fabric ecosystem. Check out this GIF Figure 1: Using Semantic Link Labs to analyze semantic models via Vertipaq Analyzer. We are excited to announce that this library has been renamed to Semantic Link Labs and open-sourced on Microsoft’s official GitHub page. As its name implies, semantic-link-labs is now an official extension of Semantic Link, offering early access to many features not yet available in Semantic Link but having the reassurance of a Microsoft-branded, open-sourced product. All functions within Semantic Link Labs are fully documented here. We are confident Semantic Link Labs will help Power BI developers and admins easily automate previously complicated tasks as well as make semantic model optimization tooling more easily accessible within the Fabric ecosystem. The overwhelmingly positive feedback from this library (in its time as fabric_cat_tools) shows that semantic-link and Semantic Link Labs can offer a great deal to the Power BI community, not just the data science community. We warmly welcome your contributions to our GitHub repository. Check out this GIF Figure 2: Using Semantic Link Labs to analyze semantic models via a Best Practice Analyzer. Check out this GIF Figure 3: Using Semantic Link Labs to automatically translate a semantic model’s metadata. Apply MLFlow tags on ML experiment runs and model versions We have released a new feature that allows users to apply MLflow tags directly on their ML experiment runs and ML model versions from the user interface. This enhancement empowers users to add annotations, track changes, and incorporate additional metadata seamlessly. Whether you are fine-tuning a model or running extensive experiments, tagging will enable you to organize and contextualize your results more effectively. Moreover, these tags are easily accessible from the run or model version details page, providing a comprehensive overview briefly. Users can also utilize these tags to compare views within the item or inline notebook authoring, making it effortless to compare across multiple runs or model versions. This streamlined tagging capability is designed to enhance your workflow, offering deeper insights and improved management of your machine learning projects. To learn more about tagging, refer to the product documentation. Track related ML Experiment runs in your Spark Application We have a new feature for you that is designed to help users track related ML experiment runs within their Spark applications. This feature allows users to monitor the progress of their Spark applications, which can create one or multiple experiments and related experiment runs. By navigating to the Monitoring Hub, users can click on a Spark application and go to item snapshots to find experiments and runs created within that application. This enhancement makes it easier to debug Spark applications, especially when they impact specific machine learning workflows. For instance, if a Spark application fails, users can quickly identify which ML training runs were affected, facilitating efficient troubleshooting and resolution. This new capability ensures a smoother and more transparent process for managing and debugging ML workflows within Spark applications. To learn more about monitoring data science items, refer to the product documentation. Monitor ML Experiments from the Monitor Hub Integrate Experiment items into the Monitoring Hub with this new feature! With this enhancement, users can track experiment runs directly from the Monitoring Hub, providing a unified view of all their activities. This integration includes powerful filtering options, enabling users to focus on experiments or runs created within the last 30 days or other specified periods. This feature offers a comprehensive "single pane of glass" experience, giving users a holistic overview of all activities in their workspace. Whether managing multiple projects or tracking specific experiment runs, this integration simplifies the process, making it easier to stay organized and informed. Enhance your workflow with this seamless monitoring capability, designed to provide clarity and efficiency in managing your machine learning experiments. To learn more about monitoring data science items, refer to the product documentation. Use PREDICT with Fabric AutoML models We're releasing an update to code-first AutoML in Fabric. With this update, we now automatically log the input and output schema for non-Spark models trained using AutoML. This enhancement allows users to seamlessly move from training with AutoML to making predictions by leveraging the built-in Fabric PREDICT UI and code-first APIs for batch predictions. This streamlined process ensures a smooth transition from model training to making accurate and efficient predictions, enhancing overall productivity and ease of use. To learn more about AutoML in Fabric Data Science, refer to the product documentation. The AI skill is now in public preview You can now build your own generative AI experiences over your data in Fabric with the Public Preview of the AI skill! With this new capability, you can build question and answering AI systems over your Lakehouses and Warehouses. You can configure the AI to respond to questions in the way that works for you and your organization by providing instructions and examples. Many generative AI experiences are generic; they do not understand all the nuances that come along with your data. These systems can output very reasonable looking answers that are ultimately incorrect. The issue is that this nuance is not expressed in the schema or the data itself. How should the AI count sales across different time zones? Or what should you do if the customer ID is not unique? These kinds of issues arise frequently in real world data systems, and only you have all the context needed to provide these answers. With the new AI skill experience, domain experts and data professionals now have the power to configure and guide the AI by providing instructions and examples. Your colleagues can then ask their questions and have the AI generate reliable queries for them. The AI skill parses the generated queries before the queries are executed, to ensure they do not alter or delete data. Also, the AI skill uses your credentials to execute the query, which ensures that data governance rules remain adhered to. For more details on the AI Skill, read the full update here. Real-Time Intelligence Fabric Real-Time Hub Teaching Bubble The Fabric Real-Time Hub now introduces 'teaching bubbles' that provide a step-by-step guide through its major functionalities. These interactive guides allow you to seamlessly navigate each tab of the Real-Time Hub, offering a clear understanding of the value and core functions. By following these teaching bubbles, you can efficiently kick off the GetEvents flow, ensuring a best-in-class streaming data ingestion experience. This feature is designed to enhance your learning curve and maximize the capabilities of the Real-Time Hub. KQL Queryset REST API support KQL Queryset is a data analysis tool that enables users to write and execute KQL queries against large datasets. It allows for advanced data exploration, manipulation, and visualization, making it easier to derive insights from structured and semi-structured data. The new Fabric Queryset REST APIs allow you to create/update/delete KQL Querysets in Fabric, and programmatically manage them without manual intervention. Read more here. Data Factory Dataflow Gen2 Certified connector updates Below are the new and updated connectors in this release: Update Denodo Delta Sharing Eduframe Funnel LEAP SiteImprove LinkedIn Learning Wolters Kluwer CCH Targetik SolarWinds Service Desk Bloomberg Deprecated Emplifi Metrics Socialbakers Metrics Are you interested in creating your own connector and publishing it for your customers? Learn more about the Power Query SDK and the Connector Certification program. Data pipeline New connectors We are excited to announce the release of two powerful new connectors in Fabric Data Factory data pipeline: Salesforce Connector Vertica Connector The Salesforce connector using Bulk API 2.0 in Fabric Data Factory offers a high-performance solution for integrating Salesforce data with Microsoft Fabric. Bulk API 2.0 is designed to handle large volumes of data efficiently by processing batches of records asynchronously, making it ideal for large-scale data migrations and synchronizations. This connector enables users to seamlessly extract, transform, and load extensive datasets from Salesforce into various destinations or vice versa. By utilizing this connector, organizations can accelerate data operations, streamline workflows, and leverage the full potential of their Salesforce data alongside other data sources, ensuring robust and scalable data integration and analytics. The Vertica connector offers unparalleled ease of connection to Vertica, a high-performance, distributed SQL database designed for big data analytics through on-premises gateway. By leveraging the Vertica connector, organizations can harness the robust analytical capabilities of Vertica alongside the scalability and flexibility of Fabric Data Factory, thereby enhancing their data workflows and accelerating insights across their enterprise data ecosystem. Elevate your data integration experience with these new connectors in Fabric Data Factory and unlock the full potential of your data ecosystems. Learn more here. Data Warehouse Connector Supports TLS 1.3 The introduction of TLS 1.3 support in the Data Warehouse connector marks a significant improvement in data security for data integration processes. TLS 1.3, the latest version of the Transport Layer Security protocol, provides improved encryption, reduced latency, and enhanced privacy over previous versions. By supporting TLS 1.3, the Data Warehouse connector ensures that data transmitted between the data warehouse and other systems is protected with state-of-the-art encryption, mitigating the risk of cyber threats and data breaches. This update not only strengthens the security posture of data operations but also optimizes performance, enabling faster and more secure data transfers. Organizations can now confidently leverage the Data Warehouse connector in data pipeline to integrate and analyze their data, knowing that their information is safeguarded with the most advanced security protocols available. Learn more here. Easily Connect to your Azure Resources by Modern Get Data Experience in Data pipeline You can easily browse and connect to your Azure resources automatically with the modern data experience of Data Pipeline. With the Browse Azure experience, you can quickly connect to Azure resources without manual filling in some information, such as endpoint, URL, etc. Currently we have supported connecting Azure Blob, Azure Data Lake Gen 2, Azure Cosmos DB and Synapse through Browse Azure experience. To browse the Azure resources, you need at least a Reader access role on the data source. To be able to connect to the resource with OAuth, you need at least the Storage Blob Data Reader role on the data source. For more information on access roles. Learn more here. Let’s get started with the Geta Data experience in Fabric Pipeline. After you navigate to the Azure module, you will land into browse Azure module. In the browsing Azure module, you can select subscription, resource group and resource types to filter on the resources that you want. In this example, my account is Azure Blob storage, so I will connect the resource with the Azure Blobs storage type. After connecting to Azure Blob storage, you can easily select the file to preview data. You can also use a similar easy way to connect with other supported Azure resources. Learn more here. Get certified on Fabric! We’d like to thank the thousands of you who completed the Fabric AI and 30 Days to Learn It Skills Challenge and earned a discount voucher for Exam DP-600 which leads to the Fabric Analytics Engineer Associate certification. If you earned a discount voucher, you can find redemption instructions in your email. We recommend that you schedule your exam promptly, before your discount voucher expires. If you need a little more help with exam prep, visit the Fabric Career Hub which has expert-led training, exam crams, practice tests and more. Fabric Sticker Challenge Winners Announced! The Fabric Community Sticker Challenge launched August 1-23 and winners are in! All Fabric Community members were invited to create unique stickers showcasing their enthusiasm and creativity under the following categories: Community Enthusiasm, Inspirational, “Inside Joke” for developers and data, and Super Users. To see winning designs, check out our Community News. Thank you all who participated in this challenge; it was great to see so much involvement! Fabric Influencers Spotlight Check out our latest initiative, the Fabric Influencers Spotlight. Each month, we’ll be highlighting some of the great blog, videos presentations and other contributions submitted by members of Microsoft MVP & Fabric Super User communities that cover the Fabric Platform, Data Engineering & Data Science in Fabric, Data Warehousing, Power BI, Real-Time Intelligence, Data Integration, Fabric Administration & Governance, Databases and Learning.
Lakehouse vs Data Warehouse vs Real-Time Analytics/KQL Database: Deep Dive into Use Cases, Differences, and Architecture Designs
With the general availability of Microsoft Fabric this past Ignite, there are a lot of questions centered around the functionality of each component but more importantly, what architecture designs and solutions are best for analytics in Fabric. Specifically, how your data estate for analytics data warehousing/reporting will change or differ from existing designs and how to choose the right path moving forward. This article will be focused on helping you understand the differences between the Data Warehouse, Data Lakehouse, and a KQL Database, Fabric solution designs, warehouse/lakehouse/real-time analytics use cases, and to get the best of the Data Warehouse, Data Lakehouse, and Real-Time Analytics/KQL Database. Topics covered in this article: High level description of Microsoft Fabric to have a baseline understanding of the architecture/relevant components. High level description of the Data Warehouse, Data Lakehouse, Real-Time Analytics/KQL Database, and OneLake. Difference between the Data Warehouse, Data Lakehouse, and Real-Time Analytics/KQL Database. Use Cases for Data Warehouse, Data Lakehouse, Real-Time Analytics/KQL Database, or used together. Architecture designs for using the Data Warehouse, Data Lakehouse, and Real-Time Analytics/KQL Database. First, we will cover what Microsoft Fabric is from a high level to have a baseline understanding of the architecture/relevant components for this article as well as the relevant components for discussing the Data Lakehouse, Data Warehouse, and KQL Database in further detail. What is Microsoft Fabric? Microsoft Fabric is an all-in-one analytics solution for enterprises that covers everything from data movement to data science, Real-Time Analytics, and business intelligence. It offers a comprehensive suite of services, including data lake, data engineering, and data integration, all in one place. Microsoft Fabric is built on a SaaS foundation as it brings together new and existing components from Power BI, Azure Synapse, and Azure Data Factory into a single integrated environment. You can interact/build on these components through the experiences such as Data Engineering, Data Factory, Data Warehouse, Data Science, Real-Time Analytics, and Power BI. The advantages of having these components/experiences on one shared platform are: An extensive range of deeply integrated analytics in the industry. Shared experiences across experiences that are familiar and easy to learn. Developers can easily access and reuse all assets. A unified data lake that allows you to retain the data where it is while using your preferred analytics tools. Centralized administration and governance across all experiences. Although the components of Fabric use ‘Synapse’ as a prefix for Data Engineering, Data Warehousing, Data Science, and Real-Time Analytics -> this is NOT the same as Azure Synapse Analytics. This can lead to confusion because of naming. The technology is built on top of the existing technology of Synapse but there are a lot of significant changes/improvements to the architecture/functionality that makes it a revolutionized version. Those experienced with Azure Synapse Analytics (Dedicated Pools, Serverless Pools, Spark Pools) will find the concepts of the Microsoft Fabric components familiar, but that the technology/functionality is significantly improved and optimized. What is OneLake, Data Lakehouse, Data Warehouse, and Real-Time Analytics/KQL Database? OneLake OneLake is the data lake that is the foundation on which all Fabric services are built. OneLake is built on top of ADLS (Azure Data Lake Storage) Gen2 and services as the tenant-wide store for data and can support any type of file, structured or unstructured. Every Fabric tenant automatically provisions OneLake, with no extra resources to set up or manage. All Fabric data items like data warehouses and lakehouses store their data automatically in OneLake in Delta Parquet format. It is a single unified storage system for all developers (professional and citizen) where all compliance, security, and policies are enforced centrally and uniformly. OneLake eliminates the data silos across organizations and eliminating the need to physically copy or move data for different teams/engines to use whether the data is in OneLake or if it is stored in a shortcut compatible location. You can think of OneLake as the OneDrive for you for your organization’s data. Data Lakehouse Microsoft Fabric Lakehouse is a data architecture platform for storing, managing, and analyzing structured and unstructured data in a single location, in open format. The default file format is delta parquet. You can store data in 2 physical locations that are provisioned automatically, files (unmanaged) or tables (managed delta tables). Tables A managed area for hosting tables of all formats in Spark (CSV, Parquet, or Delta). All tables, whether automatically or explicitly created, are recognized as tables in the Lakehouse. Also, any Delta tables, which are Parquet data files with a file-based transaction log, are recognized as tables as well. Files An unmanaged area for storing data in any file format. Any Delta files stored in this area aren't automatically recognized as tables. If you want to create a table over a Delta Lake folder in the unmanaged area, you'll need to explicitly create a shortcut or an external table with a location that points to the unmanaged folder that contains the Delta Lake files in Spark. The Lakehouse allows you to perform transformations via Spark computes or use SQL endpoints for analysis/exploration of your data. The Data Lakehouse’s default file format is delta parquet which is optimal for analytic workload performance. Data Lakehouses are not a new concept or proprietary to Microsoft Fabric, as they are very common in analytic workload architectures using the recommended medallion architecture (more on this later). This same concept of a Data Lakehouse is present in Microsoft Fabric with some new functionality and seamless integration with the components inside of Microsoft Fabric and a few other services outside of Microsoft Fabric. Including the ability to provide one copy of your data and share amongst other teams with zero data duplication. Data Warehouse Microsoft Fabric Data Warehouse is a lake centric data warehouse built on an enterprise grade distributed processing engine. One of the major advantages of Fabric Data Warehouse compared to other data warehousing solutions is that there is no need to copy the data from a Data Warehouse for other compute engines or teams to consume since the warehouse uses OneLake as its storage. One single copy of the data stored in Microsoft OneLake in a delta parquet format. Because of this, you can have cross-database querying to leverage different sources (OneLake or other sources through shortcuts) of data seamlessly and with zero data duplication. At its core, the Data Warehouse is a SQL MPP Engine (massively parallel processing engine) with Delta Tables and TDS endpoints, which will provide you with full T-SQL DDL and DML support (check availability of specific functions/features). The compute is a serverless infrastructure to allow for infinite scaling with dynamic resource allocation, instant scaling up or down with no physical provisioning involved, and physical compute resources are assigned within milliseconds to jobs. Real-Time Analytics/KQL Database Real-Time Analytics is a fully managed big data analytics platform optimized for streaming, and time-series data. It utilizes a query language and engine with exceptional performance for searching structured, semi-structured, and unstructured data. Real-Time Analytics is fully integrated with the entire suite of Fabric products, for both data loading, data transformation, and advanced visualization scenarios. Just like the rest of the Microsoft Fabric components, Real-Time Analytics is a SaaS experience. With Real-Time Analytics in Microsoft Fabric, you enable your organization to focus and scale up their analytics solution while democratizing data for the needs of both the citizen data scientist all the way to the advanced data engineer. Real-time analytics have become essential in many scenarios in the enterprise world, such as cybersecurity, asset tracking and management, predictive maintenance, supply chain optimization, customer experience, energy management, inventory management, quality control, environmental monitoring, fleet management, and health and safety. This is achieved through reducing the complexity of data integration. You are able to gain quick access to data insights within a few seconds of provisioning, automatic data streaming, indexing, and partitioning for any data source or format, and on-demand query generation and visualizations. Real-Time Analytics lets you focus on your analytics solutions by scaling up seamlessly with the service as your data and query needs grow. Read more here What makes Real-Time Analytics unique? To discover more about the advantages of Real-Time Analytics. Real-Time Analytics uses a KQL database as a data store. We will cover the architecture/components of a Real-Time Analytics solution later in this article, but for now I will give a brief overview of the KQL database. KQL stands for Kusto Query Language, and it is a query language used to interact with data located in a KQL database. KQL database can be used as a general term to apply to both Fabric Real-Time Analytics and Azure Data Explorer since it is referring to a database/store that is interacted with using KQL. At its core, Real-Time Analytics shares the same core engine with identical core capabilities as Azure Data Explorer, just with different management behavior. (Azure Data Explorer being a PaaS offering and Real-Time Analytics a SaaS offering). So, if you are familiar with Azure Data Explorer then that knowledge is translatable to Fabric Real-Time Analytics, with differences in management. Azure Data Explorer provides unparalleled performance for ingesting and querying telemetry, logs, events, traces, and time series data. It features optimized storage formats, indexes, and uses advanced data statistics for efficient query planning and just-in-time compiled query execution. Azure Data Explorer offers data caching, text indexing, row store, column compression, distributed data query, separation of storage and compute. See this documentation to view the differences in functionality/management between the Real-Time Analytics and Azure Data Explorer: Differences between Real-Time Analytics and Azure Data Explorer - Microsoft Fabric | Microsoft Learn What is the difference between the Data Lakehouse, Data Warehouse, and Real-Time Analytics/KQL Database? Now that we have a basic high-level understanding of what Microsoft Fabric is, and specifically OneLake, Data Lakehouse, Data Warehouse, and and Real-Time Analytics/KQL Database. It is time to break down the differences between the Data Lakehouse, Data Warehouse, and and Real-Time Analytics/KQL Database so we can better understand the use cases for each and help with designing our solutions. First, we will focus on the comparison between the Data Lakehouse and Data Warehouse before comparing the and Real-Time Analytics/KQL Database. Both the Data Lakehouse and Data Warehouse are open data format – delta parquet, lake centric approaches and have some cross functionality, but there are differences that I have broken down into categories. How to access/work with the data Endpoint being used (more detail below) Types of data stored/used Developer Skillset Endpoints for Data Lakehouse and Data Warehouse There are 3 different endpoints between each the Lakehouse and Data Warehouse. Lakehouse Endpoint for Spark Runtimes/Libraries SQL Analytics Endpoint of the Lakehouse Data Warehouse Endpoint Lakehouse Endpoint for Spark Runtimes/Libraries To interact with Lakehouse files/tables for analysis, transformations, or processing using Spark, you will connect to the endpoint for your Lakehouse that is separate from the SQL Analytics Endpoint. Just like standard methods outside of Fabric for interacting with files/delta tables, you will connect using either the URL, ABFS path, or mounting the Lakehouse directly in your explorer. Using Spark allows you to perform write operations with your choice of Scala, PySpark, Spark SQL, or R. However, if you wish to use T-SQL, you will need to use the SQL Analytics Endpoint where you can only perform read-only operations. SQL Analytics Endpoint of the Lakehouse This endpoint provides a SQL-based experience for the Lakehouse delta tables. The SQL Analytics Endpoint offers the ability to interact, query, and serve data within the Lakehouse using SQL. This experience is read-only and applies only to delta tables. This would be the ‘Tables’ section of the Lakehouse, and the ‘Files’ section are not readable/discoverable through the SQL endpoint. The SQL Analytics Endpoint allows you to analyze the delta tables using T-SQL, save functions, generate views, and apply SQL object level security. It enables data engineers to build a relational layer on top of physical data in the Lakehouse and expose it to analysis and reporting tools using the SQL connection string. A SQL Analytics Endpoint is automatically created when you create a Lakehouse, which points to the delta table storage. Each Lakehouse has only one SQL Endpoint, and each workspace can have more than one Lakehouse. This means that the number of SQL Endpoints in a workspace matches the number of Lakehouses. It is important to note that the creation/modification of delta tables (and the data within the delta tables) must be completed using Apache Spark, Delta Tables that are created through Spark within the Lakehouse are automatically discoverable through the endpoint. And if there are external delta tables that are created with Spark code then they will not be visible to the SQL analytics endpoint until you make a shortcut to the external delta table for it to be visible. You can set object-level security for accessing data using SQL analytics endpoint. These security rules will only apply for accessing data via SQL analytics endpoint. This means that if you want to make sure your data is not accessible in other ways (through different endpoints or directly) then you must set workspace roles and permissions. You can connect to this endpoint outside of Fabric as well with tools such as SSMS or Azure Data Studio by supplying your authentication and endpoint connection string just like any other SQL server connection. Behind the scenes, the SQL analytics endpoint is using the same engine as the Warehouse endpoint to serve high performance, low latency SQL queries. Which means that it is also a TDS endpoint, just with restrictions on the DML/DDL functionality and T-SQL surface limitations when compared with the Data Warehouse endpoint. It is important to note that MSFT documentation will describe the SQL Analytics Endpoint for the Lakehouse as a Data Warehouse. Although this can be true depending on your context, I will not refer to it as a data warehouse to avoid confusion when discussing the Synapse Data Warehouse. I will explicitly refer to it by the SQL Analytics Endpoint or SQL querying on the Lakehouse. Within my Fabric workspace, I have Lakehouse called ‘BronzeLakehouse’. Within that Lakehouse, I have 2 items (different item types) associated with my Lakehouse. The SQL Analytics Endpoint (red) and the Lakehouse endpoint (green) What can I see using the ‘Lakehouse’ endpoint? Files (red) and Delta Tables (green). You can check which endpoint you are currently viewing in the top right corner via a drop-down selection (shown expanded) e_Analytics_KQL_Database_Deep_Dive_into_U What can I see using the SQL Analytics Endpoint of the Lakehouse? Delta Tables only. Data Warehouse Endpoint The Synapse Data Warehouse or Warehouse Endpoint functions in a ‘traditional’ SQL data warehouse fashion. This means that this endpoint supports the full T-SQL capabilities like your enterprise data warehouse. Unlike the SQL Analytics Endpoint, the Data Warehouse Endpoint offers you: Read and write support to Delta Tables You can read data using spark or T-SQL You can write data with only T-SQL Full DML and DDL T-SQL support Including data ingestion, modeling, and development through TSQL or the UI You are fully in control of creating tables, loading, and transforming Can use COPY INTO, pipelines, dataflows OR Cross Database ingestion using CREATE TABLE AS SELECT (CTAS), INSERT..SELECT, or SELECT..INTO Full ACID compliance and transaction support Lakehouse offers ACID compliance support for only Delta Tables. So there can be files/data that would be in a Lakehouse that wouldn’t support ACID compared. Multi-table transaction support Pairing the full read/write capabilities and the cross-database ingestion capabilities, you are able to ingest from multiple Data Warehouses or Lakehouses seamlessly. When ingesting data into the Data Warehouse, it will create a Delta Table that is stored in OneLake. Here is a diagram that can help explain the difference between the SQL Analytics Endpoint and the Data Warehouse Endpoint. Example of a cross database query to load data into the Data Warehouse from the Lakehouse. Warehouse table ‘holiday.Warehouse_Holiday_Clean’ is created and loaded with a CTAS statement with the ‘SilverLakehouse.dbo.Holiday_Clean’ delta table as the source. KQL Database Endpoint The KQL Database is not a new concept or as complex as the endpoints for the Lakehouse or Data Warehouse. We will discuss the use cases and other factors that differentiate Real-Time Analytics/KQL database from the other offerings, but this section is for the endpoint differences. There are no limitations on read/write capabilities depending on your language or endpoint as there are not different endpoints for a KQL database. You will be able to read/write using KQL, Spark, connector ecosystem (no code), or T-SQL. Depending on exactly what you are trying to do, there may be a better language or process that you will have to evaluate in a case-by-case basis. The limitations of read/write capabilities would be applied to the account permissions. KQL databases offer the ability to apply ONLY row level security, which is account specific and not compute engine specific like the Data Warehouse and Lakehouse. Below are some of the main benefits of using a KQL Database that I want to highlight: Manage an unlimited amount of data, from gigabytes to petabytes, with unlimited scale on concurrent queries and concurrent users. Built-in autoscale adjusts resources to match workload factors like cache, memory, CPU usage, and ingestion, optimizing performance and minimizing cost. Query raw data without transformation, with high performance, incredibly low response time, while using a wide variety of available operators. Queued and Streaming ingestion has a couple of seconds latency. Automatic indexing and partitioning of your ingested data to further support high performance at unlimited scale without you managing like in traditional RDMS. Any data format can be ingested/analyzed through analytical queries such as free text and JSON structures. Support for advanced analytics such as time Series native elements and full geospatial storing and query capabilities There is also a preview feature called ‘One Logical Copy’. This provides you the ability to create a one logical copy of KQL Database data by enabling data availability in OneLake. Enabling data availability of your KQL database in OneLake means that you can query data with high performance and low latency in your KQL database, and query the same data in Delta Lake format via other Fabric engines such as Direct Lake mode in Power BI, Warehouse, Lakehouse, Notebooks, and more. One logical copy (Preview) - Microsoft Fabric | Microsoft Learn Types of Data Stored Another difference between the Data Lakehouse, Data Warehouse, and KQL Database is the type of data that is stored and how it is organized. Within the Data Lakehouse: You can store unstructured, semi-structured, or structured data. The data is organized by folders and files, lake databases, and delta tables. Within the Data Warehouse: You can store structured data. The data is organized by databases, schemas, and tables (delta tables behind the scenes) Within the KQL Database: You can store unstructured, semi-structured, or structured data. The data is organized by databases, schemas, and tables. Developer Skillset In many architecture and design sessions, the right service/pattern may not be the best for your team due to skillsets of the team members. For instance, transforming data in Spark may be the best design choice for performance, cost, etc. However, no one on the team has experience or knowledge with Spark or any language besides T-SQL. That will affect your design to consider the availability/uses of your developer’s skillset. Within the Data Lakehouse: The primary skill is Spark(Scala, PySpark, Spark SQL, R) This is for write operations and the majority of the workloads There can be a secondary skill of T-SQL for read-only operations or analysis thanks to the SQL Analytics Endpoint on the Lakehouse The interaction with the data will primarily be through Spark notebooks and Spark job definitions. Within the Data Warehouse: The primary skill is SQL This includes T-SQL and related SQL knowledge like data modeling, DDL/DML statements, SQL MPP engine understanding, SQL DBA knowledge etc. The interaction with the data will be through SQL scripts. Stored procedures, views, ad hoc queries, etc. You can read from the Data Warehouse with Spark but it would not be used for consumption/serving. Within the KQL Database: The Primary skills are KQL, SQL, or No Code. You will be able to read data with KQL, T-SQL, Spark, and Power BI You will be able to write data with KQL, Spark, and the connector ecosystem. The interaction with data will be through a KQL Queryset or KQL Database directly. Here is a chart that provides a comparison between the Data Warehouse, Lakehouse, and KQL Database. You can view the full documentation here: Fabric decision guide - choose a data store - Microsoft Fabric | Microsoft Learn Data warehouse Lakehouse KQL Database Data volume Unlimited Unlimited Unlimited Type of data Structured Unstructured,semi-structured,structured Unstructured, semi-structured, structured Primary developer persona Data warehouse developer, SQL engineer Data engineer, data scientist Citizen Data scientist, Data engineer, Data scientist, SQL engineer Primary developer skill set SQL Spark(Scala, PySpark, Spark SQL, R) No code, KQL, SQL Data organized by Databases, schemas, and tables Folders and files, databases, and tables Databases, schemas, and tables Read operations Spark,T-SQL Spark,T-SQL KQL, T-SQL, Spark, Power BI Write operations T-SQL Spark(Scala, PySpark, Spark SQL, R) KQL, Spark, connector ecosystem Multi-table transactions Yes No Yes, for multi-table ingestion. See update policy. Primary development interface SQL scripts Spark notebooks,Spark job definitions KQL Queryset, KQL Database Security Object level (table, view, function, stored procedure, etc.), column level, row level, DDL/DML, dynamic data masking Row level, table level (when using T-SQL), none for Spark Row-level Security Access data via shortcuts Yes (indirectly through the lakehouse) Yes Yes Can be a source for shortcuts Yes (tables) Yes (files and tables) Yes Query across items Yes, query across lakehouse and warehouse tables Yes, query across lakehouse and warehouse tables;query across lakehouses (including shortcuts using Spark) Yes, query across KQL Databases, lakehouses, and warehouses with shortcuts Advanced analytics Time Series native elements, Full geospatial storing and query capabilities Advanced formatting support Full indexing for free text and semi-structured data like JSON Ingestion latency Queued ingestion, Streaming ingestion has a couple of seconds latency Use Cases for Data Warehouse, Data Lakehouse, and Real-Time Analytics/KQL Database Now that there is a deeper understanding of the differences between the Data Warehouse, Data Lakehouse, and Real-Time Analytics/KQL Database, it is time to review some of the use cases to determine which option to use. There are a lot of specific use cases out there and it would be impossible to cover every possible scenario, so I am going to cover some general scenarios. To determine whether you should use the Lakehouse alone, Data Warehouse alone, KQL Database alone, or mixing them together, will generally depend on these requirements for your solution. How the data is going to be consumed/used Requirements for your application or ETL/ELT Skillset of developers Inclusive of data engineers, data scientists, citizen developers, and professional developers How the data is going to be consumed/used When designing a solution, it is important to understand how the data you are working with will be consumed. A Lakehouse, Data Warehouse, and KQL Database have a lot of flexibility/overlapping ability for serving data to users. However, the specific needs/nuances can vary from solution to solution so here are some common examples of different consumption methods for the data: Users consuming through Power BI reports/dashboards. Lakehouse, Data Warehouse, and KQL Database all can serve Power BI. Lakehouse and Data Warehouse both can serve Power BI with the same direct lake mode semantic model capability. (Direct lake mode not discussed in detail here.). Or using import or DirectQuery mode through the SQL Analytics Endpoint of the Lakehouse or the Data Warehouse endpoint. KQL Database can only serve Power BI with Direct Lake Mode when the preview feature 'One Logical Copy' is enabled. More on this later. Business users analyzing/exploring with ad-hoc T-SQL Lakehouse, Data Warehouse, and KQL Database all offer the ability for SQL read functionality. Other factors will determine what solution is best. Allowing users to consume with Spark directly on files, including unstructured, semi-structured, and structured data and all file types (not just delta tables or delta parquet). This can be for business users, data scientists, etc. Lakehouse is the correct use case for this, since any file format can be stored, and Spark can be used to interact with these files. The Data Warehouse allows Spark to read only to the tables, which might be enough but generally, you would not use a Data Warehouse for this scenario. KQL Database has the ability to interact with tables using Spark functionality for read and write, however it will depend on other factors like use cases, needs of the ETL/ELT or application, size of the data, etc. to determine if KQL should be used over a lakehouse. Mix of consumption: PySpark skills from the Data Engineering team who perform the transformation work/data modeling and business users/developers who consume read-only T-SQL queries whether ad hoc directly, cross-database queries to other Lakehouses/Warehouses in Fabric, or in Power BI. Lakehouse would be best for this scenario. Because of the business users consuming read only T-SQL, then they can use the SQL Analytics Endpoint on the lakehouse for their consumption. And the data engineering team can continue to work with the data in Spark. There would be no need to use the Data Warehouse, unless there were other requirements/needs that would force the use of the Data Warehouse such as using the warehouse endpoint for third party reporting tools or analytic queries or requiring functionality only available in Data Warehouse Ex. Multi-table transactions or DML/DDL capability via SQL You can technically use a KQL database for your Real-Time Analytics solution in this scenario, as you can serve/consume data via KQL/SQL queries or in Power BI while performing transformations. And can cross query between the data warehouse, lakehouse, and other KQL databases (via Shortcuts with 'One Logical Copy' enabled) KQL databases fulfills the requirements of this scenario, however I would start with a lakehouse first then determine based on other requirements of the solution. More on that as we go on. Ex. Needing real-time analytics with high performance and low latency (not ‘near real-time’) or time series and geospatial storing/query support. Each database, Data Warehouse, Lakehouse, KQL, SQL Server, Cosmos DB, etc., are all optimized for different read/write sizes and workloads. So, understanding these optimizations is key to determining which solution is best based on the requirements. Requirements for your application or ETL/ELT For different applications and ETL/ELTs, there can be solution needs that require specific functionality of either a Lakehouse, a Data Warehouse, or Real-Time Analytics/KQL Database. Again, the nuances of every solution make it impossible to cover all scenarios, but I will cover 4 common requirements for applications or ETL/ELT workloads. ACID transaction compliance ACID stands for: Atomicity Each statement in a transaction is treated as a single unit. Either the entire transaction is executed or none of it is. Ex. a bank transfer deducting money from your account and transferring it to the other account. Without atomicity, you can deduct money from your account without it reaching the other account. Consistency Transactional consistency. Requiring the changes made in a transaction are consistent with any constraints. Prevents any of these errors or corruption from entering your data. Ex. You try to withdraw more money from an ATM than is in your account. Transaction fails since it prevents an overdraft, violating a constraint. Isolation All transactions are run in isolation without interfering with each other. If there are multiple transactions applying changes to the same source, then they will occur one by one. Ex. You have $1,000 in a bank account. You send 2 simultaneous withdrawals for $500 each. If these happen at the same time, then your ending balance would be $500. But with isolation, the ending balance would be $0 since each transaction impacted the other. Durability Ensuring that the changes made to the data persist. Whether this is writing to a database, saving a file, etc. Just that the changes are available in case of a crash or system failure. Why ACID is important? Being ACID compliant allows for the best data integrity and reliability possible by never allowing your data to be in an inconsistent state. What does ACID compliance look like in the Lakehouse and Data Warehouse? Lakehouse ACID Compliance ACID transaction capabilities are ONLY for Delta formatted tables. This means you must utilize the managed Delta Tables to have ACID functionality. Delta Tables extend parquet files with a file-based transaction log for ACID transactions. You can interact/have ACID support for Delta Tables with the Spark or SQL engine. Non-Delta Tables AKA, all other files, will not have ACID support in the Lakehouse. Data Warehouse ACID Compliance ACID transactions are fully supported in the Data Warehouse for all tables. All tables are Delta Tables stored in OneLake as delta parquet files with a file-based transaction log. KQL Database ACID Compliance ACID transactions are NOT supported in KQL databases. A KQL Database is a distributed system supporting eventual consistency and does not support transactions. There is an ability to use an ‘update policy’ to apply ingestion updates to multiple tables however this is not ACID as there are limitations to this and they do not function as transactions to support ACID compliance. Multi-table Transactions Multi-table transactions are a way to group changes to multiple tables into a single transaction. This allows you to control the commit or rollback of read and write queries, and modify data that is stored in tables using transactions to group changes together. Ex: You change details about a purchase order that affects three tables (dbo.ItemStock, dbo.PurchaseOrderHistory, dbo.OrderShippingInfo). You can group the transaction of cancelling an order so that the changes will apply to all 3 tables or none at all. So, when a user queries any of the tables, they will see the consistent change across all tables, and not contradicting/incorrect data. Multi-table transactions are ONLY supported in the Data Warehouse. The Lakehouse currently does not support this functionality. KQL Databases do not support transactions, it's a distributed system supporting eventual consistency. A KQL Database can achieve an update behavior similar to multi-table transaction via the ‘update policy’ but this is not ACID compliant nor guaranteed order of execution/durability. Dynamic Data Masking Dynamic data masking limits sensitive data exposure by masking it to nonprivileged users. It helps prevent unauthorized viewing of sensitive data by enabling administrators to specify how much sensitive data to reveal, with minimal effect on the application layer. Some sensitive fields that are commonly masked for business users are SSNs, emails, account numbers, PHI, etc. Lakehouse Data Masking: Supported only through the SQL analytics endpoint of the Lakehouse. Files or engines other than SQL (Ex. Spark) will not be able to use Dynamic Data Masking. Can also apply object and row level security (along with DDM) but only through the SQL Analytics Endpoint. Spark engines will not be able to use the object and row level security. This may change in the future through OneLake Security. Can view more about the public roadmap here. What's new and planned for OneLake in Microsoft Fabric - Microsoft Fabric | Microsoft Learn Data Warehouse Data Masking: Fully supported. In addition to dynamic data masking, the data warehouse supports SQL granular permissions, column and row level security, and audit logs. KQL Database Data Masking: None. You can apply row level security or restricted view access policy on a table or try to create views or functions that only expose select columns however there is no support for masking or encrypting sensitive data. Real-Time Analytics The general explanation of what Real-Time Analytics is being able to process, view, analyze, and measure data as soon as it has been collected. This means that data must be available for consumption with extremely low latency, such as within a few seconds. The solution must be able to handle large amounts of data while providing that few second latency from collection. The size you want to think of is going to be in petabytes and higher with the data format widely varying. Some of the general use cases for Real-Time Analytics involve security audit logs, inventory data, supply chain information from assets on a production floor to the delivery trucks and streaming smart device information in any industry. Data Warehouse: The Data Warehouse is not a great use solution for Real-Time Analytics. The data size, format, and latency requirements do not make the Data Warehouse an option for Real-Time Analytic solutions. However, you can ingest the data from your Real-Time Analytics solution into your Data Warehouse for further analysis or combine with your existing data for further value. Lakehouse: The Lakehouse could be an option for Real-Time Analytics, but it will depend on the requirements of the solution if it is even possible. The Lakehouse could be a choice, but it is not the first one. IF you have millions of transactions per second with only 100-500 rows of data per transaction, well the Lakehouse may be able to support the needs of the Real-Time Analytics solution. As this falls into the potential performance/latency expectations of a Lakehouse solution. However, if those millions of transactions per second have hundreds of thousands or millions of rows per transaction, then the Lakehouse will not be able to handle this and is not a possible option. It will depend on the concurrency and size of the data to determine whether the Lakehouse would even be able to achieve the low latency and high-performance requirements. KQL Database: KQL Database is the best choice for Real-Time Analytics solutions. Hands down. KQL databases are optimized for very large writes and high concurrency. The example of the millions of transactions per second with hundreds of thousands of rows in each transaction will be able to be handled in a performant manner with KQL databases. With the automatic indexing and partitioning of the tables/data within the KQL database and the support for all data formats, this is the best solution for handling any Real-Time Analytics solutions. Each database, Data Warehouse, Lakehouse, KQL, SQL Server, Cosmos DB, etc., are all optimized for different read/write sizes and workloads. So, understanding these optimizations is key to determining which solution is best based on the requirements. Summary of Requirements for your application or ETL/ELT These are just 4 examples of common requirements within analytic solutions that will dictate which component to use, Lakehouse, Data Warehouse, or KQL Database. For ACID compatibility, you will need to review the requirements of the solution/other factors since utilizing the ACID capability in either the Data Warehouse and Lakehouse; you will need to create a Delta Table (all tables in the Lakehouse and Data Warehouse are Delta Tables). This means there is likely a need to either convert a file into a Delta Table or load directly into one for both components. You will not be able to use a KQL Database. So, the choice will depend on the other factors for the use cases such as skillset, other requirements, and consumption. For multi-table transactions, you will need to use the Data Warehouse over the lakehouse and KQL Database to perform these transactions. Understanding this can save you a lot of headaches and time when designing your solution/functionality for your users. For dynamic data masking, you can use either the Data Warehouse or the Lakehouse and KQL Database to achieve this functionality. IF using a SQL endpoint for the Lakehouse. If you have need for files to have Dynamic Data Masking or other engines like Spark, then you can certainly try to design your Lakehouse to remove sensitive columns and expose different versions of the data to the business users, however that can quickly become unmanageable depending on the scale. The data warehouse offers the ability to have a one stop shop for business users and protects sensitive information via dynamic data masking without worrying about the use of files or other engines. For Real-Time Analytics, you will want to use KQL Database. You may be able to use the Lakehouse depending on the exact requirements, but KQL Databases are optimized/designed for this exact scenario. As always, you can certainly try to design a solution in the Lakehouse, but the KQL database is just more performant for this scenario and has many optimizations out of the box. Data Warehouse is not a good choice for Real-Time Analytics solutions. Skillset of Developers The last major category of the use case decisions is the skillset of your developers. This includes citizen developers, professional developers, data engineers, data scientists, etc. An important piece of information when I advise customers on architecture decisions is understanding the skillset of the individuals working with the data. The most optimal service/design for performance and cost may not be the best solution for the team since no one would know how to develop it. A silly example of this concept is if you are trying to have your team individually drive a car from point A to point B. You decide that the fastest way to get there is to provide your team with only manual transmission vehicles (stick shift) because they are faster than automatic transmission vehicles. If you know how to drive stick shift, then you won’t have a problem getting there quickly. However, if your entire team doesn’t know how to drive stick shift, then they will take even longer/run into issues using the manual transmission. Supplying an automatic transmission vehicle instead of a manual transmission may not be the best option, objectively, but it may be the best option for your team. High-level breakdown of the skillsets used and the types of developers for the Lakehouse, Data Warehouse, and KQL Database. Lakehouse Developers/Users with Spark knowledge and preference. Working directly with files and or Delta Tables for the ETL/ELT or other workloads. This includes working with Spark notebooks and Spark job definitions. Usually data engineers, data scientists, and professional developers Developers/Users with only T-SQL knowledge to only read the data curated for them (in Delta Tables via the SQL Analytics Endpoint) for consumption or analysis. Usually, citizen developers and professional developers Data Warehouse Developers/Users with T-SQL knowledge and preference while building an ETL/ELT. Data Warehousing experts working with stored procedures, functions, and DBA tasks. Usually data warehouse engineers, and SQL developers Developers/Users with only T-SQL knowledge to only read the data curated for them for consumption or analysis. Usually, citizen developers and professional developers KQL Database Developers/Users with KQL knowledge and preference. You will be able to interact with the data (read/write) with Spark or T-SQL statements however KQL is the best choice for interacting with the data at this time. Developers/Users can consume via Power BI connectors or different views, queries, or functions as a source. Use Case Conclusion After reviewing some general use cases combined with the different requirements that you may have, it should be clearer when to use a Lakehouse vs a Data Warehouse vs a KQL Database. Often these decisions are complicated and require a lot of factors to be considered. By starting with analyzing how the data is going to be consumed/used, the requirements for your application or ETL/ELT, and the skillset of developers/users; your design/architecture decisions can be made quicker. And if not, then at the very least, you will find out what other questions to ask. Solution Architectures/Designs For Data Lakehouse, Data Warehouse, and Real-Time Analytics/KQL Database Understanding the functionality, feature support, use cases, and differences between the Data Lakehouse, Data Warehouse, and a KQL Database will assist you in building an architecture for different types of solutions. We will combine everything we have learned so far and apply it to architectures for different workloads/scenarios. These examples are not intended to be reference architectures or recommendations of specific best practices across all scenarios, as we covered just some of the complexities that need to be considered. The goal is to demonstrate architectures using the Lakehouse exclusively, Data Warehouse exclusively, Real-Time Analytics/KQL Database exclusively, the Lakehouse and Data Warehouse together, and Real-Time Analytics/KQL Database and a Lakehouse or Data Warehouse together to provide a better understanding of different design patterns based on your criteria. The solution architecture/design will be broken into these categories: Medallion architecture concept review Lakehouse example architecture Data Warehouse example architecture Lakehouse and Data Warehouse combined architecture Real-Time Analytics/KQL Database architecture Medallion Architecture Concept Review A medallion architecture is not unique to Fabric and has been best practice/very common for lakehouse/data warehousing architectures in the cloud. And it continues to be best practice in Fabric. So it is important that we review the basics of a medallion architecture before diving into specific architectures using it. The medallion architecture is the recommended approach in Fabric. The medallion architecture describes a series of data layers that denote the quality of data stored in the lakehouse. A medallion architecture comprises of three distinct layers or zones: bronze (raw), silver (validated), and gold (enriched). Each layer indicates the quality of data stored in the lakehouse, with higher levels representing higher quality. The names of the zones, like ‘validated’ for the silver zone, can vary across sources and designs but concepts are the same. Bronze Raw zone. First layer that stores the source data in its original format. Typically, there is little to no user access at this layer. Silver Enriched zone. Raw data that has been cleansed and standardized. The data is now structured with rows and columns, and can be integrated with other sources across the enterprise. Transformations at the silver level should be focused on data quality and consistency, not on data modeling. Gold Curated zone. Final layer sourced from the silver layer. This data is refined, “business ready”, and meets analytical/business requirements. This can be the Lakehouse or a Data Warehouse depending on needs. More on this design below. You can have multiple gold layers for different users or domains with specific optimizations/designs. For example, you might have separate gold layers for your finance and sales which is utilizing a STAR schema and optimized for analytics/reporting. And you might also have a gold layer for your data scientists that is optimized for machine learning. Lakehouse Example Architecture Here is an example of a Lakehouse architecture utilizing a medallion architecture. Below are notes on different sections of the diagram. General decision points for using this architecture pattern: Developer team skill set is primarily Spark. There is no need for additional capabilities of the Data Warehouse like multi-table transactions and dynamic data masking. Consumption does not need to use the Data Warehouse endpoint for functionality that some third-party reporting tools require. T-SQL DDL/DML capability is not required by users/developers. Data Sources: Can be any source from files to streaming data and everything in between. This data can be sourced from an on-premises location, Azure location, other cloud providers locations, shortcuts, or inside of Fabric itself. Prepare and Transform: Ingestion methods: We did not cover ingestion methods in detail, but you can choose a variety of no code, low code, or code options. The goal is to land the data in the Bronze Lakehouse (raw layer). Fabric pipelines 200+ native connectors are part of the Microsoft Fabric Pipelines. Dataflows Shortcuts Data source is either: OneLake, Azure Data Lake Store Gen2 (ADLS Gen2), Amazon S3, or Dataverse. Spark notebooks/jobs Includes streaming data and file/database connection. Transformation/data promotion methods for all zones: Will depend on the skillset for no code, low code, or code options. Spark Can be notebooks or Spark Job Definition Preferred method for complex scenarios and high code option. Performance will be best here for complex transformation scenarios. Dataflows Low code option to perform simple transformations and best for small datasets. Orchestration by Microsoft Fabric Pipelines Bronze Lakehouse: Whenever possible, keep the data in its original format. If that is not possible then use Parquet or Delta Parquet as the conversion target. The reason is performance. Loading the data into the Lakehouse will allow you to leverage better performance from OneLake/Azure location and the optimizations and scalable serverless compute available in Fabric to do any transformations or conversions necessary. Versus doing that conversion midflight where you may not be able to leverage the optimized scalable compute/low latency. If the source data is compatible for a shortcut then create one in the bronze zone instead of copying the data across. Silver Lakehouse: Recommended to use Delta tables if possible. There are extra capabilities and performance enhancements built into Fabric specifically for Delta Tables. Such as, V-Order write-time optimization to the parquet file format. Which allows extremely fast reads by the different compute engines available in Fabric. Ex. SQL engine, Power BI engine, Spark engine This area you will start to enrich your data. The specifics will depend on what is needed to be done. Whether it is combining other data sources together, transforming your existing data, data cleansing, etc. the silver zone is the area to perform these actions. Gold Lakehouse: Same recommendation for using Delta Tables as the Silver Lakehouse. This area will be “business ready” Lakehouse. Meaning that you have your data in a STAR schema data model, data is normalized, and business logic has been applied so the data is ready for consumption. *It is recommended that each Lakehouse be separated into its own Fabric Workspace. This allows better control and governance at each zone level compared to having all the Lakehouses in a single workspace. Analyze: Users will consume and analyze the data through 2 methods: SQL Analytics Endpoint of the Gold Lakehouse Users/teams can pull data into different reporting tools and even cross database query between Lakehouses and Data Warehouses for different reporting need. This is also the method for exploration or ad hoc analysis. Whether it is business users or SQL analysts/developers, the SQL Analytics endpoint offers the ability to have a full, read-only, SQL experience all from the Gold Lakehouse. Through the SQL Analytics Endpoint, privileged users can apply object level and row level security to protect sensitive data. You can create views and functions to customize and control your end user’s experience an access just like a traditional SQL environment that many users are used to. Power BI Users/teams will consume data through Power BI via reports, datasets, dashboards, apps, etc. That are sourced from the Gold Lakehouse. Direct lake mode is a new semantic model capability for parquet formatted files to load directly from the data lake without having to query a Lakehouse endpoint OR import/duplicate data within the Power BI model. Learn about Direct Lake in Power BI and Microsoft Fabric - Power BI | Microsoft Learn You can still use Import mode and DirectQuery if needed as well from the Gold Lakehouse. Data Warehouse Example Architecture Here is an example of a Data Warehouse architecture. Below are notes on different sections of the diagram. General decision points for using this architecture pattern: Developer team skill set is primarily T-SQL/Data Warehousing skillset. Transforming data with stored procedures There is a requirement to support functionality like multi-table transactions which is only supported by the Data Warehouse. Such as transactional workloads (OLTP vs OLAP). Does not mean that transactional workloads can ONLY use Data Warehouse. They can use Lakehouse depending on the requirements of the application or ETL/ELT. Consumption requires a Data Warehouse endpoint for functionality not available in the Lakehouse SQL endpoint that are required for third-party reporting tools or processes. T-SQL DDL/DML capability is required by users/developers. Your workload requires users to be able to modify data even after it has been normalized or transformed. Data Source: ‘Mount Enabled’ is referring to shortcuts. It is recommended if you are able to use a shortcut then use it. Shortcuts allow you to use the data without physically copying or moving it. Data source is either: OneLake, Azure Data Lake Store Gen2 (ADLS Gen2), Amazon S3, or Dataverse. Structured/Unstructured Includes data from other Azure Data Services, other cloud platforms, on-premises sources, etc. Ingestion: ‘Mounts’ are referring to mounting an existing Azure Data Factory to leverage existing infrastructure to load data into Fabric and specifically the Data Warehouse. Pipelines & Dataflows Fabric Pipelines offers the ability to ingest data into your warehouse with 200+ native connectors. You can use copy activities in pipelines to land the data or dataflows to land the data with in-flight transformations if needed. Those familiar with Azure Data Factory will find similar functionality and concepts within Fabric Pipelines. Store: Warehouse Design: Although not shown in the diagram, it is still recommended to have clear separation of data zones within your Data Warehouse design. Just like a medallion architecture but using the Data Warehouse instead of a Lakehouse. Data Warehouses often have the same concept as a medallion architecture, with some concept of a landing zone, staging zone, and production zone which provide the same benefit as the medallion architecture. It is recommended to implement this concept into your Fabric Data Warehouse as well. To accomplish this, there are 2 options depending on the amount of administrative oversight tolerable. Separate the bronze, silver, and gold layers into separate Data Warehouses within their own workspace. You can still cross query to different warehouses to facilitate the data movement/use data, but this offers a natural security and governance border. Single warehouse with schema/table enforcement for the different zones. This can cause a lot of overhead and sprawl but can work for smaller data warehouses with not a lot of objects/security requirements. Each zone would be depicted in a schema and provide the bronze, silver, and gold within the single warehouse through schemas. Ex. bronze.Patient, silver.Patient, gold.Patient You will want to enforce different object level security to prevent users from discovering/interacting with data they shouldn’t, like sensitive data, raw data, etc. SQL granular permissions, object/schema security, row/column level security, and dynamic data masking can all be used Can be a lot to manage at scale. Data Transformations: SQL stored procedures are recommended. This is through the full DDL/DML capability in the warehouse. Your ETL (transforming with SQL stored procedures) can be orchestrated through Fabric pipelines like ADF now. Expose: Warehouse: Users/teams can pull data into different reporting tools and even cross database query between Lakehouses and Data Warehouses for different reporting needs. This is also the method for exploration, ad hoc analysis, or users performing DDL/DML statements. Whether it is business users or SQL analysts/developers. Using SQL granular permissions, you can define the level of access/permission each user/security group/user has. You can apply object level, row level, and column level security with dynamic data masking to prevent exposing sensitive data. You can create views and functions to customize and control your end user’s experience and access just like a traditional SQL environment that many users are used to. 3 rd party reporting tools or other processes that require functionality only available through the Data Warehouse endpoint. Power BI: Users/teams will consume data through Power BI via reports, datasets, dashboards, apps, etc. That are sourced from the gold warehouse or tables. Direct lake mode is a new semantic model capability for parquet formatted files to load directly from the data lake without having to query a warehouse endpoint OR import/duplicate data within the Power BI model. Learn about Direct Lake in Power BI and Microsoft Fabric - Power BI | Microsoft Learn You can still use Import mode and DirectQuery if needed as well from the gold warehouse or tables. Data Warehouse & Lakehouse Example Architecture Here is an example of a Data Warehouse and Lakehouse combined architecture. Below are notes on different sections of the diagram. General decision points for using this architecture pattern: Developer team skill set is primarily Spark. Transforming data with Spark notebooks There is a requirement to support functionality like end user DDL/DML ability via T-SQL which is only supported by the data warehouse. Your workload requires users to be able to modify data even after it has been normalized or transformed. Consumption requires functionality from the Data Warehouse endpoint that some third-party reporting tools or processes. This architecture diagram is the same as the Lakehouse medallion architecture diagram except with 2 differences. Differences between this architecture and the Lakehouse only architecture: Gold Lakehouse is replaced by a Gold Data Warehouse. Consumption through the SQL Analytics Endpoint of the Lakehouse is replaced by the Data Warehouse endpoint. Why would you combine the architectures? Combining both the Lakehouse and Data Warehouse into a single architecture can provide you with the best of both worlds (depending on your requirements). With this architecture, you allow the consumption layer (through Power BI or warehouse endpoint) to take full advantage of the Data Warehouse like DDL/DML support for users/developers at the gold layer and Data Warehouse endpoint for other reporting tools/processes – while not having to sacrifice performance or duplicate data just for a different consumption method. All the data transformations within the ETL/ELT are performed with Spark to achieve the best performance and flexibility of your code while leveraging the primary skillset of the developers and still allowing the best end user experience/functionality. Real-Time Analytics/KQL Database Example Architecture Here is an example of a Real-Time Analytics/KQL Database architecture. Below are notes on different sections of the diagram. General decision points for using this architecture pattern: Requirement for Real-Time data analysis (latency in seconds) Often customers explain a need for “Real-Time” requirements, and after discussion, the requirement is to be “Near-Real-Time”. Meaning that the latency requirements are not in seconds, but in hours, or minutes. The “Near-Real-Time” is sometimes misunderstood to mean Real-Time. This distinction is important as Real-Time Analytics is referring to the latency in seconds. Depending on other requirements, Near-Real-Time solutions may not use KQL Databases. KQL Databases offer high performance, low latency, and high freshness even at very large scale. There is a requirement to have unlimited scale. This unlimited scale applies to query concurrency/size, ingestion streams/size, and storage size. Everything is automatically indexed and partitioned within a KQL database. Requirements for real-time data transformation of complicated data structures. Requirements to use any data source and/or any data format. Requirements to use a timeseries database and/or geospatial functionality built in. Consumption can be through Power BI, Notebooks in the Lakehouse, Data Pipelines/Dataflows, or user queries. Data Source: Any data source and any data format can be used here. Lakehouses and Data Warehouses, can have cross database queries and also be a source for the KQL Database. Event streaming sources like Event Hubs, IoT Hubs, or other open-source ecosystems (Kafka, Logstash, Open Telemetry, etc.) are possible to use KQL Databases as their streaming targets. Ingestion: Streaming data into Real-Time Analytics through an Event Hubs cloud connections Get data from Azure Event Hubs - Microsoft Fabric | Microsoft Learn The event streams feature in Microsoft Fabric gives you a centralized place in the Fabric platform to capture, transform, and route real-time events to various destinations with a no-code experience. Everything in Fabric event streams focuses on event data. Capturing, transforming, and routing event data are the essential capabilities of eventstreams. The feature has a scalable infrastructure that the Fabric platform manages on your behalf. Shortcuts. It is recommended if you are able to use a shortcut then use it. Shortcuts allow you to use the data without physically copying or moving it. Data source is either: OneLake, Azure Data Lake Store Gen2 (ADLS Gen2), Amazon S3, or Dataverse. Pipelines & Dataflows Fabric Pipelines offers the ability to ingest data into your warehouse with 200+ native connectors. You can use copy activities in pipelines to land the data or dataflows to land the data with in-flight transformations if needed. Those familiar with Azure Data Factory will find similar functionality and concepts within Fabric Pipelines. Data Residing in OneLake Data loaded into Real-Time Analytics is reflected in OneLake as one logical copy. One logical copy (Preview) - Microsoft Fabric | Microsoft Learn Data from OneLake can be queried from Real-Time Analytics as a shortcut. Data from OneLake can be loaded into Real-Time Analytics via pipelines, queries, dataflows, or manually in the Fabric UI. Store: KQL Database: KQL Database is the data store for Real-Time Analytics. All the benefits and use cases are above. A medallion architecture would not be used in the KQL Database, as this is meant for raw telemetry data and modeling/cleansing occurs either midflight or at the consumption layer. It is still recommended to have some sort of organization or naming convention in your KQL objects. One important concept for designing your Real-Time Analytics solution is archiving. You will want to have an archiving strategy set in place to prevent the data from growing/incurring costs if not needed. Why would you want to archive data? Well, most likely, the data in your Real-Time Analytics solution is valuable for only a short period of time, and for more historical analysis you have different assets. Each solution has different requirements and business needs so the amount of data to be retained can vary but very rarely is there a true requirement to keep all the data in a Real-Time Analytics solution. For example, in manufacturing, knowing the current status of production floor machinery is of paramount importance for the dashboard/analysis. However, what the status was 30 days ago at a specific time or even 3 days ago is not relevant or useful for the scope. This is where archiving the data will allow you to either delete the data or move it somewhere else to provide deeper analysis/trending, combine with warehouse data, or any other uses you may have for it. This will help performance and cost. Within Real-Time Analytics, you can define data retention policies and export the data in different ways to various destinations. Lakehouse & Data Warehouse: The reason for including the Lakehouse and Data Warehouse is for that deeper analysis or further use of analytics that fall outside of the scope of the Real-Time Analytics reporting. The benefits of this are a solution that provides you with Real-Time Analytics and the historical retention of the data to be used in other analytical reporting. One Logical Copy (Preview) allows you to interact with the data from the KQL Database in OneLake, which allows you to work with the data without physically copying it for different compute engines. Provides the ability of the lakehouse or Data Warehouse to use a shortcut to connect to the data in KQL without physically copying it to a lakehouse. Now if you want to load that data into different files or warehouses for historical retention or other reasons, you would be physically moving/copying the data. This can be accomplished via Pipelines, Dataflows, Spark Notebooks, and even cross database query functionality between the Lakehouse, Datawarehouse, and KQL database when enabling the one logical copy in KQL (allows you to create a shortcut). If you are using the KQL database as a source of data for your data warehouse or lakehouse, review the above architectures for guidelines when implementing these solutions. Expose: Power BI: Users/teams will consume data through Power BI via reports, datasets, dashboards, apps, etc. That are sourced from the gold warehouse or tables. When you enable ‘One Logical Copy’ then you will be able to use Direct lake mode. Direct Lake Mode is a new semantic model capability for parquet formatted files to load directly from the data lake without having to query a warehouse endpoint OR import/duplicate data within the Power BI model. Learn about Direct Lake in Power BI and Microsoft Fabric - Power BI | Microsoft Learn Otherwise, you will have to use import mode or direct query mode. KQL Queryset: The KQL Queryset is the item used to run queries, view, and customize query results on data from a KQL database. Uses KQL for creating queries, views, functions, control commands, customized results and many SQL functions. There are tabs in a KQL Queryset that allow you to connect/associate with different KQL Databases and switch them out to run the same query in different states. You can collaborate with others and save queries. Notebooks: You can also explore data using Notebooks via Spark/KQL. This allows you the ability to ingest, analyze, use in Machine Learning, or any of your processes via Notebooks directly from the KQL Database. Conclusion When thinking about creating new solutions, or migrating existing ones, within Fabric, understanding the capabilities of the Fabric components and your solution requirements are critical. By examining different use cases for the Data Warehouse, Lakehouse, and Real-Time Analytics/KQL Database, examples of different architecture patterns, and deep diving into the capabilities of each the Data Warehouse, Lakehouse, and Real-Time Analytics/KQL Database, there should be a clearer architecture design path when combined with the knowledge of your user/workload requirements. Sources for article: What is OneLake? - Microsoft Fabric | Microsoft Learn What is a lakehouse? - Microsoft Fabric | Microsoft Learn What is data warehousing in Microsoft Fabric? - Microsoft Fabric | Microsoft Learn Fabric decision guide - choose a data store - Microsoft Fabric | Microsoft Learn Tutorial: Ingest data into a lakehouse - Microsoft Fabric | Microsoft Learn Fabric decision guide - copy activity, dataflow, or Spark - Microsoft Fabric | Microsoft Learn OneLake shortcuts - Microsoft Fabric | Microsoft Learn Implement medallion lakehouse architecture in Microsoft Fabric - Microsoft Fabric | Microsoft Learn Overview of Real-Time Analytics - Microsoft Fabric | Microsoft Learn Differences between Real-Time Analytics and Azure Data Explorer - Microsoft Fabric | Microsoft Learn Sense, analyze, and generate insights with Synapse Real-Time Analytics in Microsoft Fabric | Microsoft Fabric Blog | Microsoft Fabric Fabric Change the Game: Real – time Analytics | Microsoft Fabric Blog | Microsoft FabricFabric January 2026 Feature Summary
If you’ve been trying to keep up with everything shipping in Microsoft Fabric, this January 2026 round-up is for you—covering the biggest updates across the platform, from new AI-powered catalog experiences and OneLake governance improvements to enhancements in Data Engineering, Data Warehouse, Real-Time Intelligence, and Data Factory. If you haven’t already, make sure FabCon Atlanta is on your calendar (March 16–20, 2026)—it’s shaping up to be an epic week for the Fabric community. Contents Events & Announcements Upgrade your skills and certifications before FabCon Join live sessions with your favorite MVPs and other Fabric experts and request voucher to take the Fabric Analytics Engineer (DP-600) exam or the Fabric Data Engineer (DP-700) exam. Voucher supplies are limited. Register for a live session. Submit your request for a Fabric exam voucher. Have you registered for FabCon Atlanta yet? Join us from March 16-20, 2026, in Atlanta, GA! For the first time ever, SQLCon will be co-located with FabCon, bringing together the entire data community in one place. Two Conferences. One pass. One Epic Week. Join us for the ultimate Microsoft Fabric, SQL, Power BI, Real-Time Intelligence, AI, and Databases community-led event. Master SQL Server 2025 internals in the morning, dive into Fabric innovations in the afternoon, attend Power Hour before dinner and network with peers from both communities. The sessions you choose are totally up to you. Register with code FABCOMM to save $200. Prices go up on February 14th. General Microsoft acquires Osmos to extend Fabric with agentic data engineering Microsoft has acquired Osmos, an agentic AI data engineering platform designed to help simplify complex and time-consuming data workflows. Together, we’re accelerating the future of autonomous data engineering directly in Microsoft Fabric, helping customers turn data in OneLake into analytics- and AI-ready assets faster. To learn more, read the acquisition announcement. Fabric Platform AI Auto-Summary for semantic models (Preview) The Auto-Summary for semantic models is an AI-generated high-level summary that helps you quickly understand an item’s purpose and main characteristics without opening the item or reviewing its full metadata. It makes it easier to understand unfamiliar items and compare them directly in the OneLake catalog Explorer. The summary is created based on the item’s metadata and structure. Users with the appropriate Copilot capacity and permission can generate the summary from the quick actions in the main explore tab or directly from the semantic model’s item details page. Each time you return to the Catalog, a new summary can be generated so that you always see the most up-to-date version. After a summary is generated, you can generate another version, copy the text for use elsewhere, or provide feedback on the quality. Parent-Child hierarchy in OneLake catalog OneLake catalog now includes a clear Parent–Child item structure that makes it easier to understand how your data items relate to each other. Instead of showing everything in one flat list, the catalog now groups connected items together and displays them in the appropriate hierarchy. For example, a Lakehouse appears with its autogenerated SQL Analytics Endpoint, and an Eventhouse appears with its related KQL DBs. You can easily view or hide these related items using the expand and collapse feature, giving you a cleaner view when you need it and more detail when you want it. This helps you quickly find what you need, reduces confusion between similarly named items, and makes it clearer which item you should connect to for each task. Learn more from the OneLake catalog documentation. Item Reference Variable Type (Preview) The Variable Library in Microsoft Fabric continues to evolve, and one of the most impactful additions arriving with the upcoming release is the Item Reference variable type, a new way to reference Fabric items in a structured, resilient, and more secure way. This feature simplifies managing configuration across different environments by allowing you to reference Fabric items directly instead of hard‑coding values as strings. We will also validate permissions to the referenced item, ensuring stricter governance on changing the values being used within variables. Why this matters The new Item Reference variable addresses these issues: Stronger security: Variables can only reference items that the user is authorized to access. Better clarity: Instead of obscure GUIDs shown in the UI, references clearly provide meaningful information of the item name, type, and its location. For different value sets, simply choose the item in the proper workspace. Improved reliability: Structured metadata eliminates issues caused by renaming items or manually editing IDs. Better experiences across Fabric The Fabric ‘Select variable’ dialog lets you choose a reference from only the items you’re permitted to access and surfaces helpful details such as item name, type, and location, so you can confidently select the right one. In addition, the experience of configuring it across Fabric in UDF, Lakehouse shortcuts, and other Fabric items is unified through the following dialog where you can easily filter and search your variable: Key points You must have at least READ permission on an item to set it as a value. When deploying or updating, Fabric validates that all referenced items in the target stage active value-set exist, and that you have permission for them. Early support will be available in Lakehouse Shortcuts, UDF code and Notebook code (Not supported in %%Configure). Upcoming items can be tracked in the documentation. Looking ahead Item reference is the first step toward unified, scalable configuration in Fabric. Next, we will bring Connection reference, a new variable type for managing external connections (AWS, Blob Storage, and more) with the same secure and consistent experience. Stay tuned! Ready to get started? The Item Reference variable type is more than “just another variable”. is a foundational step toward a more maintainable, predictable, and scalable Fabric ecosystem. With its structured design, CI/CD-friendly behavior, and shared resolution model, it removes many long-standing pain points around configuration, item linking, and environment consistency. Explore the Variable Library in your Fabric workspace and try creating your first Item reference variable today! Git integration – enhanced support for GitHub Enterprise Cloud with data residency Fabric workspaces can now connect to GitHub Enterprise Cloud instances with data residency (ghe.com), allowing regulated customers to use Microsoft Fabric – Git Integration. To learn more, refer to the Microsoft Fabric and GitHub Enterprise Cloud with data residency support. Git integration – Commit to standalone branch We have introduced a highly requested flexibility feature that allows users to create a new branch from their last synchronization point and commit current changes to it in a single action. In practice, this means: No commitment to the connected branch Branch off on the fly without switching context Commit to another branch even if there are incoming updates This new functionality can be useful for scenarios like: Handling conflict situation Working in isolation Backing up your current work on specific branch Sharing your work with others To learn more, refer to the Microsoft Fabric – Get started with Git Integration – commit to standalone branch. Python SDK for Microsoft Fabric REST API (Preview) Python SDK is now available on PyPI as microsoft-fabric-api and you can get started today. The Microsoft Fabric Python SDK is a client library that wraps the Fabric REST APIs, allowing developers to manage, automate, and interact with Microsoft Fabric resources directly from Python. Instead of hand-crafting HTTP requests and managing authentication manually, you can use idiomatic Python objects and methods to perform common tasks. This SDK is currently in preview and is installed from PyPI, microsoft-fabric-api · PyPI The Fabric REST APIs support automation of tasks: Listing and managing workspaces Automating deployment processes Interacting with Fabric items programmatically Integrating workflows into CI/CD and DevOps pipelines The new SDK offers: Built-in support for authentication via Azure identity libraries Typed API clients for REST endpoints Simplified serialization/deserialization Higher-level helpers that make script automation easier Install the SDK You can install the SDK using pip: Bash pip install microsoft-fabric-api This will grab the latest public preview release from PyPI. Getting Started The following is a quick example to help you start exploring Fabric resources. Step 1: Authenticate The SDK uses Azure identity libraries (such as DefaultAzureCredential) to get tokens automatically. Make sure you’re logged in via Azure CLI or have environment credentials set. python from azure.identity import DefaultAzureCredential from microsoft_fabric_api import FabricClient # Create your credential (Azure CLI login, managed identity, etc.) credential = DefaultAzureCredential() Initialize the client fabric_client = FabricClient(credential) Step 2: List Workspaces Once authenticated, you can call Fabric REST APIs through the SDK. For example, to list workspaces you have access to: python workspaces = fabric_client.core.workspaces.list_workspaces() print(f"Found {len(workspaces)} workspaces:") for ws in workspaces: print(f"- {ws.display_name} (Capacity ID:{ws.capacity_id})") This example demonstrates how straightforward it is to work with Fabric resources using Python. OneLake Granular APIs for OneLake security (Preview) Microsoft Fabric is introducing new granular REST APIs for security role management, giving developers finer control over how OneLake permissions are created, retrieved, and managed programmatically. In addition to the existing batch role API, Fabric now supports discrete Get, Create, and Delete role operations, enabling users to work with individual roles without submitting a full role collection. These new APIs make it easier to build automation‑friendly security workflows and integrate Fabric security into CI/CD pipelines. By combining bulk and granular role management options, Fabric offers greater flexibility for organizations managing security at scale—reinforcing our commitment to open, developer‑first, and interoperable security in OneLake. Granular APIs to get, create, and delete a single role. Check out the OneLake Data Access Security documentation. OneLake security support for Mirrored item types Microsoft Fabric now supports defining OneLake data access roles on all mirrored item types, extending granular, role-based security to data replicated into OneLake from transactional systems. With this update, customers can control access to mirrored data using table-, row-, or column-level security, ensuring permissions are enforced consistently at the OneLake layer regardless of how the data is consumed. Manage data access roles in your mirrored database. By attaching security directly to mirrored data in OneLake, this release enables secure reuse across teams through shortcuts and downstream analytics experiences. Organizations can mirror data once, apply fine-grained access controls at the source, and confidently share data without duplication—simplifying governance while scaling analytics across Fabric. You can learn more about OneLake security in the documentation. OneLake diagnostics immutable logs OneLake diagnostic events can now be made immutable, which means that the JSON files that contain diagnostic events can't be tampered with, or deleted, during the immutability retention period. To learn more, please refer to OneLake diagnostics documentation. Data Engineering High Concurrency mode for Lakehouse operations High Concurrency mode for Lakehouse operations in Microsoft Fabric is a new capability designed to dramatically optimize Spark resource utilization for common tasks like Load to Table and Preview. Previously, when these operations fell back to Spark execution—especially in environments with Managed Virtual Networks—users often faced significant startup latency of three to five minutes per job or found that a single preview operation could hold a session for up to 20 minutes, blocking concurrency on smaller capacities. High Concurrency mode solves this by allowing up to five independent Lakehouse jobs triggered by the same user within the same workspace to share a single, underlying Spark session. This innovative approach delivers immediate improvements in both performance and cost-efficiency. By reusing existing sessions, subsequent table loads or previews can start in under five seconds, even with network security features enabled. Furthermore, this mode offers a significant price-performance advantage: only the initiating Spark session that starts the shared application is billed, meaning subsequent operations sharing that session incur no additional compute costs. These optimized sessions are automatically managed by Fabric and are easily traceable in the Monitoring hub via the HC_<lakehouse_name> naming convention, ensuring you have full visibility into your accelerated workflows. To learn more, refer to the High concurrency mode for Lakehouse operations in Microsoft Fabric documentation. Fabric connection inside Notebook (Preview) With this update, notebooks now offer the familiar Get Data feature, making it simpler and safer for users to access data from frequently used sources like Azure Blob Storage, PostgreSQL, Azure Key Vault, and S3. This update supports connections to cloud data sources. If you need to access on-premises data sources, please use the Managed Private Endpoint option. Create a Fabric connection inside notebook with the new “Connection” flow. Once the connection is ready, you can generate the code snippet to access the underlying data source. Apply the connection’s credential detail to other data source, if applicable. Create Fabric connection inside Notebook. Generate code snippet to query data source. The supported authentication types include: Basic Authentication: Supported for Azure SQL Database and other databases that support basic authentication Account Key Authentication: Supported for REST API data sources that require Account key authentication Token Authentication: Supported for data sources that require token-based authentication Workspace Identity Authentication: Supported for Fabric workspace identity authentication Service Principal Authentication (SPN): Supported for data sources that require SPN-based authentication You can also create the connection inside the Fabric data source management page, but you need to ensure the toggle named Allow Code-First Artifacts like Notebooks to access this connection (Preview) has been enabled. This toggle can only be set during the creation of the connection and can't be modified later. Allow code-first artifact like Notebooks to access this connection. After creating the connection, it appears under Global permissions, ready to link to the notebook. Select Connect in the context menu to link this connection to the notebook. To learn more, refer to the Fabric connection inside Notebook documentation. Open and edit workspace’s Notebook inside VS Code Users can now directly open and edit the Fabric Notebook within its VS Code editor. Previously, the Fabric Data Engineering VS Code extension enabled notebook development in VS Code only after downloading the notebook. With this update, the notebook can be accessed and edited from the selected remote workspace, and any changes saved in VS Code will automatically update the content in the remote workspace. To open the notebook, select the Open Notebook Folder icon located on the toolbar of the desired Notebook. Direct Open Notebook Once activated, the VS Code Explorer View opens the selected Notebook and shows the Fabric workspace with its notebooks, including the one currently open. With this new view, you can open multiple notebooks from the same Fabric workspace, and even open multiple different Fabric workspaces. Multi-Notebook and Multi-workspace view Create or replace semantics support for Materialized Lake View Materialized Lake Views now support create or replace semantics, making it significantly easier to evolve data models as business requirements change. This enhancement allows users to update an existing Materialized Lake View—whether that involves adding or modifying columns, adjusting transformation logic, or updating metadata—without the need to drop and recreate the view. As a result, it becomes faster to iterate while avoiding the operational risks and disruptions that typically come with deleting and rebuilding core analytic objects. For more information, refer to the Spark SQL Reference for Materialized Lake Views documentation. Lineage enhancements in Materialized Lake views Lineage for materialized lake views now clearly shows the Notebook Source, making it simple to trace each view to its origin. Deleted sources are also flagged for quicker troubleshooting and more reliable refresh scheduling. Learn more about this feature in the materialized lake views documentation. Data Warehouse Proactive statistics refresh Proactive statistics Refresh for the Data Warehouse and Lakehouse SQL Analytics Endpoint is a built-in optimization that enriches the automatic management of vital column statistics. With this feature enabled, column statistics that were created during SELECT queries may now be updated by the engine proactively as their data changes. This reduces the likelihood of a query being prolonged by statistic maintenance during plan generation, thus reducing query execution time. Now more than ever, there are fewer reasons to maintain statistics manually. See the statistics documentation for more information. Incremental statistics refresh Incremental statistics refresh is a performance enhancement in Data Warehouse and Lakehouse SQL Analytics Endpoint that improves the execution time of certain column statistic updates. Columns in long tables that experience mostly INSERT or ADD operations since the last statistic refresh are eligible for incremental refresh. As statistic operations can sometimes contribute to SELECT queries’ execution time, queries over these types of tables will benefit most from this improvement. Incremental statistics refresh is a quicker mode of automatically updating statistics, compared to before. See the statistics documentation for more information. Result set caching (Generally Available) In 2025, we introduced result set caching (Preview) for Data Warehouse and Lakehouse SQL Analytics Endpoint. This feature expedites repetitive queries by quickly returning cached results instead of recomputing original queries from scratch. Out-of-the-box performance booster is now generally available and enabled by default. No tuning or configurations required. Enjoy the benefits of result set caching today! Result set caching is now enabled by default, boosting eligible queries whenever possible To learn more about this feature, see the result set caching documentation. MERGE Transact-SQL (Generally Available) In September 2025, the MERGE statement was released for preview in Data Warehouse. This command provides a standardized approach to transforming your data by incorporating conditional logic and DML actions all within a single statement. MERGE encapsulates INSERTs, UPDATEs, and DELETEs all within a single statement. For more information, see MERGE (Transact-SQL) - SQL Server. Real-Time Intelligence MQTT v3 support for the Eventstream MQTT Connector We now support MQTT v3 in the Eventstream MQTT Connector, making it compatible with widely used industry protocols. With this update, the connector now supports MQTT v3.1 and v3.1.1, making it easier than ever to stream data from popular IoT platforms and MQTT brokers directly into Eventstream. Once ingested, the data can be immediately leveraged by Fabric Real-Time Intelligence for real-time analytics and alerting, enabling teams to detect patterns and act on IoT events as they happen. By bridging MQTT v3 and Eventstream, teams can take full advantage of Eventstream’s scalability, reliability, and real-time processing capabilities—without modifying their existing broker setup. This update significantly lowers the barrier to adoption and empowers organizations to build robust, event-driven streaming pipelines with confidence. For more details, visit Add MQTT source to an eventstream. Real-Time Weather Connector for Eventstream (Generally Available) Bring production-ready weather data streaming to your Fabric for real-time analytics. As part of this release, we’ve added several key enhancements based on early user feedback. You can now include a location name directly in the event payload, making processing, filtering, and analytics significantly easier. This removes the need for additional enrichment steps and simplifies stream processing logic. We’ve also introduced a tenant-level control switch, allowing tenant admin to enable or disable the Weather Connector for the entire organization. This gives teams better governance, cost control, and operational clarity when managing Eventstream with weather data streams. These features are now fully supported and ready for mission-critical use cases—from real-time analytics to alerting and dashboard. We look forward to seeing your streaming workflow. For more information, refer to Add a real-time weather source to an eventstream. Eventhouse accelerated OneLake shortcuts now support acceleration based on date-time columns Accelerated OneLake shortcuts in Eventhouse indexes and caches data in OneLake, allowing performant queries on top of delta/Iceberg tables in OneLake. By default, the system uses the modificationTime in delta_log to determine the scope of data to accelerate. We are adding a new property HotDateTimeColumn to the query acceleration policy to specify the name of datetime column in the Delta table whose values will be used to determine hot-cache eligibility. When set, data files whose rows have values within the configured Hot period (and/or HotWindows) are selected for caching. You can override the default behavior by changing the query acceleration policy. For more information, refer to the .alter-merge query acceleration policy command documentation. Eventhouse accelerated OneLake shortcuts control data freshness latency MaxAge property in query acceleration policy controls the data freshness. The shortcut returns accelerated data if the last index refresh time is greater than @now - MaxAge. Otherwise, the shortcut operates in non-accelerated mode. The default for MaxAge is 5 minutes. You can now override the default MaxAge at query execution. Users can fine-tune acceleration scope dynamically, balancing freshness and performance without altering policy definitions. Example: This example overrides the MaxAge to 10 sec during querytime. external_table( TableName, 10s) If you use this property, the external table returns accelerated data if the last index refresh time is greater than @now - MaxAgeOverride. Minimum: 1s For more information, refer to the .alter query acceleration policy command documentation. Simplified KQL syntax for querying shortcuts in Eventhouse Eventhouse shortcuts can now be queried just like Tables, eliminating the need for explicit external_table() syntax, making queries cleaner and more intuitive. This simplifies external data access by allowing direct querying of external tables using standard table names. Example: sample 10 rows from shortcut T using the following syntax: T | take 10 Copilot support for querying shortcuts in Eventhouse Copilot in KQL Queryset and Real-Time Dashboards can now generate KQL for shortcuts. In Eventhouse, shortcuts are implemented as external tables. While querying shortcuts was not previously available with Copilot, this functionality is now supported, allowing users to query shortcuts in the same manner as native tables in Eventhouse. Data Factory More connectors supported for incremental copy in Copy job We are expanding Copy job with broader multi‑cloud connectivity and stronger incremental copy support. Incremental copy in Copy job now supports additional connectors, including: Google Big Query Google Cloud Storage DB2 ODBC Fabric Lakehouse table Folder Azure files SharePoint List Amazon RDS for SQL Server Amazon RDS for Oracle Azure Data Explorer ry To learn more, please refer to the Copy job in Data Factory documentation.
Microsoft Fabric, explained for existing Synapse users
Earlier this year, at Microsoft Build, we introduced, in Public Preview, Microsoft Fabric, “the biggest data product announcement since SQL Server”. Today, we are announcing the General Availability of Microsoft Fabric. Arun explains in detail why we all believe Microsoft Fabric will redefine the current analytics landscape. I will focus here on what it means for customers that are using the current Platform-as-a-Service (PaaS) version of Synapse, explaining what it means for your current investments (spoiler: we fully support them), but also how to think about the future. What happens with PaaS Azure Synapse Analytics The PaaS offering of Azure Synapse Analytics is an enterprise analytics service designed to accelerate time to insight across data warehouses and big data systems. It brings together the SQL technologies used in enterprise data warehousing, Azure Data Factory pipelines, Apache Spark technologies for big data, and Azure Data Explorer for log and time series analytics. Microsoft has no current plans to retire Azure Synapse Analytics. Customers can continue to deploy, operate, and expand the PaaS offering of Azure Synapse Analytics. Rest assured, should these plans change, Microsoft will provide you with advanced notice and will adhere to the support commitments in our Modern Lifecycle Policy in order to ensure our customers’ needs are met. The evolution of Microsoft’s big data analytics products The next versions of our big data analytics products are now a core part of Microsoft Fabric. Fabric opens new architectural horizons for our analytical engines. Fabric offers a unified storage abstraction for all your data, OneLake, organized into a logical data mesh, with federated governance and granular control and an intuitive, personalized data hub. All Fabric engines separate storage from compute, and store data in OneLake using a single, open data format. On this new foundation, we can invent new, unprecedented ways of deploying pipelines, data warehousing, data engineering, data science, observability and real-time analytics technologies, to ultimately simplify and increase the efficiency of our customers’ solutions. Fabric allows us, and our customers, to do more. This is why most of our innovation efforts will be focused on Fabric. How to think about your current Azure PaaS Synapse Analytics solutions As mentioned above, there is no immediate need to change anything, as the current platform is fully supported by Microsoft. Your existing solutions will keep working. Your in-progress deployments can continue, all with our full support. However, you probably have already started thinking about a Microsoft Fabric future for your analytics solutions. The following steps may help you with this thought process. Understand Microsoft Fabric Microsoft Fabric represents a significant upgrade to all our analytics engines. All of them are improved, faster, and more scalable. And there is a lot to learn about the new engines and how to best use them. Fabric reimagines collaboration and empowers the business users in an unprecedented way. But it is much more than just better engines or just better integration. The unified, open-source data format means that there is no need to copy data from one engine to another. You can shape data using the technology of your choice, then query it with any other technology. Fabric introduces completely new ways to make your data part of your analytics landscape. Shortcuts (within Azure, or cross cloud), database mirroring, seamless access to Dataverse and M365 data, all these solutions are designed to remove friction and costs. Understanding these technologies will enable you to make the best out of Fabric, in terms of efficiency, agility and costs. Our teams have worked hard to produce detailed documentation for all the Fabric concepts, and the best complement for the documentation is hands-on experience. The easiest way to understand Fabric in depth is to try the product: Microsoft Fabric free trial . Arun’s blog spells out clearly how to learn more about Microsoft Fabric. Understand what it means for your solution Your analytics solution may use different technologies and engines. Fabric is a complete analytics platform, so you will find, inside Microsoft Fabric, new and enhanced analytics capabilities of the products with which you are familiar today. Fabric brings new capabilities, that have no parallel with the current PaaS Synapse Analytics offering. The Fabric SQL Engine can operate, with equal performance, scale, and security, over any OneLake artifact (warehouses, lakehouses, mirrored databases). It also supports cross-artifact operations removing the need for extra copies while Power BI, for example, in DirectLake mode, can now analyze real time streaming data, or Spark output. All these changes enable simpler, more efficient solutions, removing the need for intermediate steps and multiple data copies. Your solution can get significantly simpler and cheaper. Below, I use one example of common PaaS Azure Synapse Analytics architectures, together with a possibly more efficient solution in Fabric, to demonstrate such potential simplifications. Example 1: Data Lake, from Synapse to Fabric Today, you may prepare your data in an Azure Data Lake Storage Gen2 (ADLSg2) lakehouse (typically using Spark, Synapse or Azure Databricks), then use a pipeline to load data into a Synapse SQL Dedicated Pool, then use Power BI or some other BI tool for your report. Building the Lakehouse - Implementing a Data Lake Strategy with Azure Synapse " /> You can keep your current solution intact, and upgrade to Fabric engines. In Fabric, however, this solution can be simplified: A Data Engineering Lakehouse, in Microsoft Fabric, allows you to use your current ADLSg2 data, as prepared with Synapse Spark or Azure Datrabricks (via shortcuts). The SQL Analytics Endpoint allows you to apply the security rules from the Dedicated Pool directly over the Lakehouse. There is no need for a dedicated capacity, nor for the pipeline copying from the lake to your warehouse. Using the new DirectLake mode, Power BI can now operate directly over the Lakehouse, with performance similar to Import. Your other BI tools can continue to operate over the SQL Analytics Endpoint. By migrating your Notebooks and Spark Jobs to Fabric Spark, your Lakehouse data will be automatically optimized for all the other Fabric engines (while also being stored in an open format) To learn more about the Lakehouse pattern in Microsoft Fabric, please visit Lakehouse end-to-end scenario: overview and architecture - Microsoft Fabric | Microsoft Learn Assess our migration tools and processes We are investing significant development efforts in migration processes and tooling. And our migration efforts are prioritizing current PaaS Synapse Analytics customers. The processes and tools we are designing are intended to minimize the friction, disruption and cost for our existing customers. As you will see in the section on Migration Resources, we are developing tools to: Use your data in-place whenever possible Reuse code investments (pipelines, notebooks) when possible Migrate code (stored procedures, views, notebooks) These investments are not complete. We will keep posting updates to our migration tools. Join the fast-growing Fabric community , and our specialists as well as external experts will be ready to work with you. The Fabric Ideas forum, on the community site, is the best way to suggest new features, and it is closely monitored by the Microsoft Fabric product teams. Develop, then plan to deploy a migration strategy After having learned about Fabric and evaluating the product, you will have developed enough confidence in the new Fabric engines and the migration technology to move your solution to Fabric. For some of you this may happen soon, for others it may take years. There is no rush – we will keep supporting your existing solutions – but we are ready for you to migrate whenever the time is right. When you are ready to move your solution to Fabric, you will be able to exchange your existing 1- or 3-year Synapse Reserved Instance (RI) purchases for 1 year Fabric RI purchases to continue to apply your reservation discounts in Fabric. Additionally, if you want to increase your RI commitment for your Fabric portfolio you will have access to discounts of >40% over the Fabric Pay-as-you-go pricing. In the next sections, the product leaders explain how to think about Fabric from the perspective of different PaaS Synapse Analytics workloads. Data Factory Pipelines Data Factory in Microsoft Fabric brings Power Query and Azure Data Factory together into a modern trusted data integration experience, that empowers data and business professionals to extract, load, and transform data for their organization. In addition, powerful data orchestration capabilities enable you to build simple to complex data workflows, that orchestrate the steps needed for your data integration needs. Key concepts in Data Factory in Microsoft Fabric include: Get Data and Transformation with Dataflow Generation 2 is an evolution of Dataflow in Power BI. Dataflow Generation 2 is re-architected to leverage Fabric compute engines for data processing and transformation. This enables Dataflow Generation 2 to ingest and transform data at any scale. Data Orchestration with Data Pipelines – For customers familiar with Azure Data Factory (ADF), data pipelines in Microsoft Fabric use the same technology that powers Azure Data Factory. As part of the GA of Fabric, data pipelines in Microsoft Fabric will have most of the activities available in ADF.See here a list of activities that will be part of data pipelines in Fabric. SSIS activity will be added to data pipelines by Q2 CY2024. Enterprise-ready Data Movement – Whether it is petabyte-scale data to small data, Data Factory provides a serverless and intelligent data movement platform that enables you to move data between diverse data sources and data destinations reliably. With support for 170+ connectors, Data Factory in Fabric enables you to move data between multi-clouds, data sources on-premises, and within virtual networks (VNet). Intelligent throughput optimization enables the data movement platform to automatically detect the size of the compute needed for data movement. To enable customers to upgrade to Microsoft Fabric from Azure Data Factory (ADF), we will be supporting the following: Data pipelines activities – For many of the activities that you use in ADF, we have added these into Data Factory in Fabric. In addition, we have added new activities (e.g. Teams, Outlook) for notifications. See here for a list of activities that are available in Data Factory in Fabric. OneLake/Lakehouse connector in Azure Data Factory – For many ADF customers, you can now integrate with Microsoft Fabric, and bring data into the Fabric Onelake Azure Data Factory Mapping Dataflow to Fabric – We have put together a guide for ADF customers who are looking at building new data transformations in Fabric.Find out more at https://aka.ms/datafactoryfabric/docs/guideformappingdataflowusers In addition, customers looking at migrating their ADF mapping dataflows to Fabric, you can leverage sample code from the Fabric Customer Advisory Team (Fabric CAT) to convert mapping dataflows to Spark code. Find out more at https://github.com/sethiaarun/mapping-data-flow-to-spark As part of Data Factory in Fabric roadmap, we will be working towards the preview of the following by Q2 CY2024: Mounting of Azure Data Factory in Fabric – This enables customers to be able to mount their existing Azure Data Factory in Microsoft Fabric. All ADF pipelines will work as-it-is, and continue running on Azure, while enabling you to explore Fabric, and work out an upgrade plan. Upgrade from Azure Data Factory pipelines to Fabric – We will be working with customers and the community on learning how we can best support upgrades of data pipelines from ADF to Fabric. As part of this, we will deliver an upgrade experience that empowers you to test your existing data pipelines in Fabric using mounting and upgrading the data pipelines. Learn more about how you can upgrade to Data Factory in Fabric - https://aka.ms/datafactoryfabric/upgradetofabric Synapse Data Warehouse Fabric Data Warehouse is the next generation of data warehousing in Microsoft Fabric. It is the first transactional data warehouse to natively support an open data format enabling data engineers and business users to collaborate seamlessly without compromising security or governance. Just like the previous data warehouse generation, SQL provides multi-table ACID transactional guarantees. It is built on the well-established SQL Server Query Optimizer and Distributed Query Processing engine but comes with major improvements that address many of the challenges customers face in enabling workloads associated with modern analytics. These improvements were driven by rearchitecting the data warehouse by leveraging IP from both Dedicated and Serverless SQL Pools along with: Separation of storage and compute: data is stored in OneLake and is clearly separated from the compute used by the SQL engine. There is an elastic allocation of compute resources based on demand, as well as use of distinct compute resources for different workload types on top of the same data. Leveraging the infinite compute capabilities of Azure Cloud: giving us the capability of going beyond a limited topology offered by the Synapse Gen2 architecture. Support for open data format: allowing a single copy of the data to be used by all the Fabric workloads such as Data Science, Data Engineering, and Power BI. With this new architecture, the new engine enables numerous new capabilities that were not possible in either Dedicated and Serverless SQL Pools such as: Cross database querying without any ETL or data movement. Cloning without creating copies of the data. Autoscaling enabling elastic scale up and down of the compute nodes with dynamic resource allocation tailored to data volume, usage, or query complexity. Enabling a pay for what you use pricing model. No knobs performance via automated query optimizations, statistics, and data distributions. All of this with the concepts familiar to SQL users such as Views, Stored Procedures, SQL security (row-level security, column-level security, dynamic data masking) and full benefits of the T-SQL tooling ecosystem. These architectural changes cannot be backported to either one of the old engines. Because of the open format, your data warehouses cannot be upgraded in place either. Data stored in a proprietary format in Gen2 needs to be extracted and stored in the open format of Fabric. A migration can be done at your own pace when you are ready to leverage these new capabilities. To enable this, we have added the following available to you now: Ability to export your Dedicated SQL Pool to a SQL Project and import it in Fabric. PowerShell scripts are available in GitHub that convert Gen2 DDL to Fabric supported DDL. Detailed migration documents with best practices. Find out more at the Azure Synapse dedicated SQL pools to Fabric Migration Guidance whitepaper In addition, we have also started working on an in-product Migration Assistant that will automatically detect and convert your Synapse Gen2 code to Fabric Data Warehouse code. It will also redirect your endpoints, so you don’t have to worry about application migration. We anticipate this to be available in CY24. Synapse Data Engineering Fabric Data Engineering is our big data analytics workload in Fabric, empowering data engineers to leverage the power of Apache Spark to transform their data at scale and build out a lakehouse architecture. The Fabric Data Engineering experience targets users of Apache Spark pools in the Azure Synapse Analytics world. Here are some of the key takeaways regarding the Fabric Data Engineering experience: Runtime for big data workloads Every Fabric workspace comes pre-wired with a ‘starter pool’ (default Spark cluster) with a Fabric Runtime that contains up to date versions of Spark, Delta, Java and Python. Just like in Azure Synapse Analytics, customers can also create their own custom clusters with their own configurations and libraries if they want. The Apache Spark experience in Fabric also contains many new and exciting enhancements: Starter pools in Fabric are automatically kept live meaning users can enjoy sessions that start within ~15 seconds High concurrency mode in Fabric means multiple notebooks can be attached to a single session, accelerating the start-up times and reducing costs Spark clusters start all the way from a single node, further reducing the costs of getting started with Spark Simplified lakehouse architecture Every Fabric workspace also comes pre-wired with OneLake, our SaaSified data lake for the organization. Users can easily create lakehouse items, which are the perfect container for bringing in all your data into OneLake using Spark, dataflows and pipelines. Existing data can be easily included with no data movement through the use of shortcuts. We will also automatically discover metadata of Delta tables for you, making it super easy to start working with existing data with zero friction. Additionally, we have reduced the price for Spark in Fabric by almost 40% vs. the retail price of Synapse Spark. Here are some other exciting things to keep in mind about the lakehouse in Fabric: Every lakehouse comes with a built in SQL endpoint and Power BI dataset. This means that as soon as you transform your data with Spark, you can start querying it using our SQL engine and Power BI, with no data movement necessary Spark (along with every other Fabric engine) will automatically write the data into the lakehouse with v-order enabled, automatically optimizing it for BI reporting First Class Developer Experiences The Synapse Data Engineering experience brings in familiar authoring tools, including notebooks for interactive querying experiences and Spark Job Definitions for submitting batch jobs. These capabilities come with a variety of new enhancements and users even have some new authoring experiences to look forward to: Notebooks in Fabric include numerous usability improvements including auto-save, real time collaboration and commenting, a built-in file system as well as native file format support when checking into git. Users can also make use of light-weight scheduling (in addition to using the pipeline activity). Spark Job Definitions come with retry policy support, making it easier to continuously run long running streaming jobs Native VS Code support makes it easy to work with your Data Engineering items (notebooks, Spark Jobs, lakehouse) all in your favorite IDE, including full debugging support The newly released environment item streamlines the packaging of all of your Spark configurations, libraries, cluster settings and more, and simplifies the re-usability of your hardware and software environment across your code artifacts. To summarize, with Synapse Data Engineering, you can start building on top of your existing Azure Synapse Spark investments quickly and incrementally. Start by leveraging shortcuts to existing data in your data lake and bringing-in your notebooks using the import capability. We are starting work on an in-product migration assistant but in the meantime, please use our newly published Azure Synapse Spark to Fabric Migration Guidance whitepaper. Synapse Data Science Synapse Data Science empowers data scientists to explore their data, build and operationalize their predictive models. Coming from the Azure Synapse Analytics world, you will see many familiar constructs such as Python and R being baked into the runtime including many popular ML packages, the ability to install your own third party & custom libraries as well as the availability of SynapseML, our open source library for creating massively scalable ML pipelines. Fabric Data Science offers a variety of new capabilities data scientists can look forward to: Model & Experiment tracking Data scientists are able to leverage experiments and models as readily available in items in the Fabric workspace. Support for ML models and experiments allows users to manage models and track experiment runs using standard MLFLow APIs. Comparison experiences make it easy to compare different experiment runs and auto logging helps capture key metrics automatically as users author code to train models. Model batch scoring To operationalize their ML models, users can leverage the scalable PREDICT function for distributed batch scoring on Spark. This capability exists in Azure Synapse today and so existing Synapse users should feel right at home. The Fabric Data Science experience provides low code UI for scoring data and tight integration with the lakehouse, making it easy to enrich data and surface it in Power BI reports with zero friction. Data Exploration & Enrichments Fabric Data Science offers many innovative solutions in the space of exploring and transforming your data. These include: Data Wrangler – a low code UI for carrying out data transformations that automatically generate Python code Semantic Link – a library enabling seamless connectivity to the Power BI semantic model through data science tools like notebooks Pre-built AI models – newly released public preview capability providing built-in access to Azure AI services like text analytics and translation services The migration path for a data scientist in Azure Synapse Analytics is like that of a Spark data engineer – they will need to consider their notebooks, Spark pools and data. We recommend starting with the Azure Synapse Spark to Fabric Migration Guidance whitepaper. Synapse Real-time Analytics Synapse Real-time Analytics is a robust platform tailored to deliver real-time data insights and observability analytics capabilities for a wide range of data types. This includes observability time-based data like logs, events, and telemetry data. It's the true streaming experience in Fabric! Building on the same foundation as Azure Synapse Data Explorer, Synapse Real-time Analytics equips both citizen data scientists and professional data engineers with a suite of features and tools to fully unleash the potential of their data. Rapid Deployment Experience unmatched efficiency by creating a database, ingesting data, running queries, and generating Power BI reports, all within a 5-minute timeframe. Real-time Analytics puts speed at the forefront, allowing you to dive into data analysis without delay. Get Data For an authentic streaming experience in Fabric, the "Get Data" feature has received a modern facelift with an intuitive design and user-friendly interface. It simplifies data ingestion, accepting any data format or structure from various sources in either streaming or batch mode. Your data becomes query able within seconds. Query Versatility Whether you're a Kusto Query Language (KQL) enthusiast or prefer traditional SQL, Real-time Analytics accommodates your needs. This service enables you to generate quick KQL or SQL queries, ensuring that you can work in your preferred language and obtain results swiftly. It doesn't matter if you're working with a small dataset (a few gigabytes), a medium-sized one (a few terabytes), or even massive datasets (in the petabytes range). Data Exploration Fabric Real-Time Analytics offers a multitude of innovative solutions for exploring and visualizing your data, including: KQL Queryset: A workbench for creating, managing, and sharing your queries. Power BI Report: A one-click option to generate a Power BI report on top of any query or table. Notebook: Seamlessly connect your Fabric Notebook with the KQL Database for data ingestion and querying. NL2KQL (Coming Soon): Write your query in natural language, and Fabric will generate and execute the corresponding KQL query for you. Real-Time Dashboard (Coming Soon): The Fabric Real-Time Dashboard is a collection of tiles that enable native export of Kusto Query Language (KQL) queries as visuals. This allows for easy query modification and visual formatting, enhancing data exploration and delivering superior query and visualization performance. Fabric Real-Time Analytics is your gateway to real-time insights and a streamlined data analysis experience. Whether you're pioneering new data horizons or looking to optimize your data analytics solutions, this service is your trusted partner. Stay ahead in the data game and embark on your journey with Fabric Real-Time Analytics today. For more information on Fabric Real-Time Analytics, visit the general availability blog. Migration planning Fabric KQL databases are 100% compliant with Azure Data Explorer (ADX) and Azure Synapse Data Explorer (Preview) and our powered by the same technology. It means that all current applications, SDK, integrations, and tools that work with ADX will continue to work smoothly with Fabric KQL databases. There is a broad set of capabilities to support mixed environments and migrations, some are available now and some will light up in the next months. Available now: Full binary compatibility of APIs, SDKs and tools. Create a database shortcut to host a read only, in place, up to date instance of the database in Fabric. Coming over the next months: Migrate an Azure Synapse Data Explorer pool from a Synapse workspace and attach it to a Fabric workspace Attach an Azure Data Explorer cluster to a Fabric workspace Sync Azure Data Explorer user queries and dashboards into a Fabric workspace query sets and dashboards Migration Resources Azure Data Factory Azure Synapse DW to Fabric Migration Guidance Azure Synapse Spark to Fabric Migration Guidance Azure Synapse Data Explorer and Azure Data Explorer to Fabric Migration Guidance
Copilot in Fabric (preview) is available worldwide
During November 2023, we unveiled the public preview of Copilot in Microsoft Fabric. This preview includes Copilot for Power BI, Data Factory and Data Science & Data Engineering. Since then, we have been gradually rolling this feature out and today, we are excited to announce that our Copilot preview is now available to all customers! With the Copilot preview, Microsoft Fabric brings an improved way to transform, enrich and analyze data, and shortens the time to insights. Our current preview includes: Copilot for Data Science and Data Engineering, offers intelligent code completion, automates routine tasks, and provides industry-standard code templates to simplify the process of building robust data pipelines and crafting complex analytical models. Copilot for Data Factory, provides intelligent Mashup code generation to transform data using natural language input and generates code explanations to help you better understand previously generated complex queries and tasks. Copilot for Power BI, which enables you to automatically create reports by selecting a report topic or prompting Copilot on a particular topic and create a narrative visual that summarizes your report using generative AI. To access the Copilot in Fabric, you must have a paid SKUs (F64 or higher, or P1 or higher) with capacity available in the following Fabric regions. The Fabric Copilot preview is not available on trial SKUs. How to enable Copilot in Microsoft Fabric Before you can use Copilot in Fabric, your tenant administrator needs to enable Copilot. This can be done with a new tenant setting group, “Copilot and Azure OpenAI Service (preview)”, in the admin portal. Since December 2023, we shipped a new feature that allows tenant admins to enable AI and Copilot in Fabric for specific security groups in addition to the entire organization. This means that Copilot can now be tailored to the needs of different groups within your organization. More granular AI and Copilot setting on Capacity level is coming. Fabric Copilot is powered by Azure Open AI large language models and are currently deployed to limited data centers. As of January 2024, if your data is located outside of the US or France, the Copilot preview will be disabled by default unless your Tenant administrator enables the “Data sent to Azure OpenAI can be processed outside your tenant's geographic region, compliance boundary, or national cloud instance” tenant setting. To learn how to get to the tenant settings, see About tenant settings. We look forward to your feedback about Copilot in Microsoft Fabric! Please don't hesitate to let us know how Copilot helps you and how you would like to see Copilot improve. Next Steps Overview of Copilot in Fabric (preview) Enable Copilot for Microsoft Fabric (preview) Copilot in Fabric and Power BI: FAQ
Public Preview of Native Execution Engine for Apache Spark on Fabric Data Engineering and Data Science
The Native Execution Engine showcases our dedication to innovation and performance, transforming data processing in Microsoft Fabric. We are excited to announce that the Native Execution Engine for Fabric Runtime 1.2 is now available in public preview. The Native Execution Engine leverages technologies such as a columnar format and vectorized processing to boost query execution performance. Validated by the TPC-DS 1TB benchmark, our internal tests have shown a significant 4x speed enhancement over OSS Spark. This leap in performance is achieved by translating SparkSQL code into optimized C++ code, coupled with Microsoft’s internal efforts on query optimizations embedded in Apache Spark on Fabric. One of the standout features of the Native Execution Engine is its seamless integration with Apache Spark™ APIs. Whether you are working with PySpark, Scala, R, or Spark SQL, enabling this engine requires no code alterations or concerns about vendor lock-in. It works with both Parquet and Delta formats, ensuring your data, whether in OneLake or accessed via shortcuts, is processed efficiently. Users benefit from accelerated processing times, and that leads to significant operational cost savings. Powered by innovations At the heart of the Native Execution Engine are two key open-source components: Velox, a community sustained C++ database acceleration library created and open sourced by Meta, and Apache Gluten (incubating) initiated by Intel, a middle layer responsible for offloading JVM-based SQL engines’ execution to native engines. Gluten offers seamless integration with existing Spark applications and employs modern C++ for accelerated native code execution. Utilizing CPU vectorization and advanced memory management, Gluten provides features such as native query execution, columnar-based shuffle, and compatibility with Spark's Catalyst optimizer. This combination enables the Native Execution Engine to leverage the native capabilities of underlying data sources, minimizing the overhead associated with data movement and serialization in traditional Spark workloads. This high-performance solution is poised to revolutionize big data analytics by significantly improving the efficiency of Spark SQL's operations. Query Optimizations improvements Microsoft has enhanced Apache Spark query optimizations and integrated these into Apache Spark on Fabric, including: Split Block Bloom Filters to reduce false positives when verifying the existence of elements in large datasets. Parquet Footer Caching to reduce I/O operations by caching metadata, which speeds up data retrieval in large-scale analytics projects. Smart Shuffle Optimizations to improve data distribution across nodes, minimizing network overhead. Optimized Sorting for Window Functions to accelerate data sorting within partitions for more efficient processing. For further details on these native optimizations, refer to the article and previous blog posts: New Query Optimization Techniques in the Spark Engine of Azure Synapse, Apache Spark in Azure Synapse - Performance Update, and Speed up your data workloads with performance updates to Apache Spark 3.1.2 in Azure Synapse. The Native Execution Engine architecture In the realm of resource scheduling within the Spark framework, the Catalyst logical plan optimizer remains a pivotal component. Spark begins by parsing a query, applying numerous optimization rules to forge an optimized logical plan. This plan is subsequently transformed into a physical plan slated for execution. Gluten plays a crucial role here, converting this physical plan into a Substrait plan and utilizing JNI (Java Native Interface) calls to initiate execution in Velox. Velox, recognized as a robust C++ database acceleration library, offers reusable, extensible, and high-performance data processing components. In instances where Velox does not support a particular operator, the system reverts to the existing Spark JVM-based operator. This fallback mechanism, however, brings with it the overhead of converting data between columnar and row formats, impacting performance. The architecture diagram shows how Gluten and Velox interact with Apache Spark. Microsoft's recent rollout of a Native Execution Engine marks a significant enhancement to Apache Spark on Fabric, building upon open-source software while also contributing innovations back to the Gluten and Velox projects. This initiative introduces several pivotal improvements: the integration of the Azure Blob Filesystem (ABFS) storage adapter; advanced operators such as Expand, BroadcastNestedLoopJoin, and CartesianProduct; the suite of new functions, including uuid, date_from_unix_date, and from_utc_timestamp; support for INT96/INT64 timestamps in Velox parquet scans; the capability to handle metadata columns within Gluten; significant reliability enhancements achieved by addressing over 300 unit tests across 40 suites for Apache Spark versions 3.3, 3.4, and 3.5. Scenarios for accelerated performance The current release of the Native Execution Engine performs particularly when: working with data in Parquet and Delta formats; handling queries that involve complex transformations and aggregations, benefiting from the engine’s columnar processing and vectorization capabilities; running computationally intensive queries, rather than simple or I/O-bound operations. Enabling the Native Execution Engine To use the Native Execution Engine, no code changes are required. During the preview stage, you have the flexibility to activate the Native Execution Engine either through your environment settings or selectively for an individual notebook or job. Enabling this feature within your environment item ensures that all subsequent jobs and notebooks associated with that environment will automatically inherit this configuration. For comprehensive guidance on how to activate this feature, please visit our documentation. In the meantime, we are actively developing mechanisms to enable the Native Execution Engine at the tenant, workspace, and environment levels, ensuring seamless integration with the UI. Results from early adopters ❝As part of the private preview testing of the Native Execution Engine, I designed typical analytical queries and compared their execution times with the default engine. I was impressed by the performance and reliability of the Native Engine, which handled all queries faster and more efficiently. I did not encounter any case where the queries were slower. All of them showed a significant improvement in speed, ranging from 50% to over 300%.❞ - Damian Widera, Data Platform MVP, Objectivity part of Accenture rk_on_Fabric_Data_Engine The results shared by one of the private preview early adopters of the Native Execution Engine, are split into two charts to properly reflect the "y" axis, which represents execution times. The first chart is for queries running longer than 100 seconds. The second chart is for queries running less than 50 seconds. Miles Cole (Director, Data & AI, Hitachi Solutions America) and Sandeep Pawar (Senior Power BI Architect, Hitachi Solutions America) tested the Native Execution Engine on various Data Engineering and Data Science workloads in Fabric. Miles focused on testing a wide array of Spark SQL functionality representative of typical analytical queries that included complex joins, aggregations, windowing and other expressions. He saw an average of 1.6x improvement in execution duration on the 100M row scale. Sandeep focused on testing data science workflows involving NLP applications, such as text processing and transformations, and training ML models. He saw on average a 1.3-2x speed up. Hitachi Solutions is excited about the Native Execution Engine and anticipates using it for all Spark workloads since it provides a significant improvement in execution time without any code changes. ❝Our tests with the Fabric Spark Native Execution Engine on complex gold layer transformations led to a 30% performance improvement. Daily processing times for tables exceeding 5 billion rows dropped from 3 hours to 2, with zero code changes!❞ - Andrew Dakhov, Managing Partner, Cloud Services Better together - acknowledging our partners We want to recognize the invaluable contributions of our partners in developing the components of the Native Execution Engine: Meta - for their active contributions to Velox. IBM and members of the Presto foundation - for their active contributions to Velox. Intel - for their active contributions to Apache Gluten (incubating) and over the past year of close-knit collaboration with the Microsoft team, seamlessly incorporating the Native Execution Engine from Gluten and Velox into Apache Spark on Microsoft Fabric. Additionally, we extend our gratitude to the more than 150 contributors to Velox and Gluten. Their commitment and ongoing efforts to accelerate big data query execution empower every individual and organization that processes big data to achieve more, thanks to elevated performance. This collaboration has also been essential in bringing the Native Execution Engine into the hands of our customers. ❝Composable Data Management Systems point to an exciting future for Big Data. We are excited to partner with so many contributors and innovators across the industry to build this future and thank them for their efforts. With the Native Execution Engine Spark on Fabric brings together all these layers into a seamless offering that showcases the value across all aspects of data processing in a manner that only Fabric can. We see this as the early days of an emerging stack that already delivers amazing value to our customers. We look forward to a continued partnership with these communities.❞ - Ashit Gosalia, Partner – Group Engineering Manager, Spark on Fabric ❝Fragmentation in data management has hindered innovation. We believe that by converging efforts into unified and reusable components such as Velox, we can not only provide a more efficient platform, but also accelerate the pace of innovation. Our eventual goal is to commoditize execution and provide a common software layer that enables hardware accelerators to be more transparently (and pervasively) leveraged by data management. Collaborations with large-scale cloud vendors such as Microsoft Fabric are crucial in that journey.❞ - Pedro Eugenio Rocha Pedreira, Head of Velox & Software Engineer at Meta Platforms ❝For the last 2 years the team at IBM has worked closely with Velox contributors from Meta, Microsoft, Intel, and others. We’re excited to see how far the project has come in those two years with advancements in both the Presto and Spark native engines. Microsoft Fabric’s Native execution engine with Velox is a testament to this cross-functional community work. While it’s just the beginning, we’re looking forward to continue collaborating closely with the Velox community to bring the native engine further along in the market.❞ - Ali LeClerc, Chair of the Presto Foundation Outreach Committee | Co-chair of VeloxCon | Open Source at IBM Learn more, and help us with your feedback We encourage you to share your feedback directly to our product team by using this form. We look forward to your valuable input and are eager to discuss your findings in detail.
