Krakowskie Spotkanie Grupy Power BI
Agenda Intro Łukasz Balcerzak - "Grupy obliczeniowe (calculation groups) w praktyce - wprowadzenie i przykłady zastosowania" (300) Pizza break/Networking Mateusz Motoła - "Power BI Overview" (100) Outro (i dojadanie pizzy ) Bio Łukasz Balcerzak Pracuję jako analityk biznesowy w środowisku Microsoft. Na co dzień zajmuję się modelami tabelarycznymi Power BI wykorzystujac narzędzia Tabular Editor, DAX Studio czy SQL Server Management Studio. Przygodę z danymi zacząłem w 2018 od Excela i SPSSa, pomału rozwijając kolejne kompetencje. W analizie danych największą satysfakcję sprawia mi wyciąganie informacji że zbiorów danych i prezentowanie ich na intuicyjnych wizualizacjach. Mateusz Motoła Jestem fascynatem dziedziny Business Intelligence z ponad dziesięcioletnim doświadczeniem w środowisku korporacyjnym. Obecnie pracuje jako Senior BI Developer w Coca-Cola Hellenic gdzie na co dzień wykorzystuję platformę danych Microsoft budując narzędzia analityczne. Dzięki pasji do ciągłego uczenia się z przyjemnością odkrywam granice możliwości ulubionego narzędzia jakim jest Power BI. W projektach cenię nieszablonowe myślenie oparte o szeroką perpspektywę, które pozwala na efektywne rozwiązywanie problemów.DAXdakan Ep3: Dashboard Design Workflows
This is the 1st week we asked you to determine the DAXdakan topic... Based on the survey, let's discuss real-life workflows during dashboard design! DAXdakan is a fortnightly (once every 2 weeks) Teams meeting with other Power BI-erists (Power BI Pilipinas members). Language is Filipino/English. Attendance is free.DAXdakan 1:many w/ PBI Pinas Moderators
This is it, DAXdakan with your PBI Pinas Community moderators! Hear their data journeys, ask them anything! Join us for another very special edition of DAXdakan where we meet: • Drizzle Pablo (TechBayanihan, Bizapps Pinoy Community) • Alex Gamboa (Chad GPT) • Dane Belarmino (DAXJutsu) • Arman Castroverde • Rico Balibalita • Dennis Demandaco • Jemar Lim • Adam Christian Amistad DAXdakan is a re-occuring Teams meeting with other Power BI Pilipinas members. Language is Filipino/English. Attendance is free. Join us online via Teams. Watch us live via Facebook Live. Or wait for recorded versions to be uploaded in Youtube, Spotify, and our other social media channels. While you're accepting the invite, please join our official Microsoft Community: https://community.fabric.microsoft.com/t5/Power-BI-Pilipinas/gh-p/pbi_philippines_usergroup?WT.mc_id=DP-MVP-5005302DP-800 & Microsoft Fabric
🚀 Live Community Session: DP-800 & Microsoft Fabric How does the new DP-800: SQL AI Developer Associate certification fit into the Microsoft Fabric ecosystem? Join us for a live community session with Kevin Chant as we explore the connection between DP-800 and Microsoft Fabric, what the certification covers, where Fabric fits into the picture, and what it means for SQL professionals and developers working with Microsoft's modern data platform. 📅 Tuesday, July 21 🕕 18:00 UTC 🎤 Speaker: Kevin Chant 📍 Microsoft Fabric Discord Server Whether you're curious about the new certification, working with Fabric today, or interested in where AI-powered SQL development is heading, we'd love to have you join the discussion. See you there! 🚀A new analytics frontier: GPU-accelerated Fabric Data Warehouse (Early Access Preview)
As data volumes grow, concurrency rises, and analytics workloads become more dynamic and AI-driven, performance becomes harder to predict and harder to scale. Every query sits in the critical path, adding pressure to the warehouse, and every second counts. This is the core tension in analytics today. The expectations have changed, but the underlying technology has not, leaving agents, applications, and AI systems waiting on data. To meet this moment, analytics needs a new kind of execution engine.
Microsoft Fabric September 2023 Update
Welcome to the September 2023 update. We have lots of features this month including updates to the monitoring hub, Fabric Metrics app, VS code integration for Data Engineering, Real-time data sharing and many more. Continue reading for more details on our new features! Contents Core Monitoring hub – column options OneLake OneLake file explorer (v.1.0.10) - Menu Option to View Workspaces and Items Online Power BI General Upgrade Power BI Desktop .NET Framework to 4.7.2 or newer Reporting Mobile layout interactive canvas Smart Narrative improvements Modeling Edit your data model in the Power BI Service - Updates Edit linguistic relationships in Q&A setup DAX function changes to MINX and MAXX Edit relationships in the properties pane (GA) Data Connectivity SAP HANA (Connector Update) Emplifi Metrics (New Connector) Service Row-level security test as role improvements Mobile Supporting AAD Shared device mode (preview) Developers Git integration Visualizations New visuals in AppSource Change Chart Beeswarm Chart Explore time-based data down to each millisecond with Drill Down TimeSeries PRO New Updates for accoPLANNING (Release 57) Zebra BI Charts on-visual settings Word Cloud by Powerviz Innofalls Charts: A Versatile and Interactive Visual Hierarchy Chart by MAQ Software Power BI JSON Report Theme Generator by BIBB Synapse Data Warehouse Data Warehouse Utilization Reporting in Fabric Metrics app Column-Level security is now available on Data Warehouse and SQL Endpoint Data Warehouse Row Level Security SQL Projects support for Fabric Data Warehouse Deployment Pipelines Default Dataset Improvements Data Engineering VS Code Integration Introducing Notebook File System support in Synapse VS Code extension for Data Engineering and Data Science workload in Microsoft Fabric Import Notebook in Workspace view Mssparkutils new API for fast data copy Support diff versions when saving notebook Notebook Sharing with Execute-only mode Notebook resources .whl file support Real-time Analytics OneLake shortcut to delta tables from KQL DB Real-Time Data Sharing: Introducing database shortcuts in Real-Time Analytics Sample Gallery – Explore what you can achieve in Real-Time analytics Model and Query data as graphs using Kusto Query Language (KQL) Easily connect to KQL Database from Power BI desktop Eventstream now supports AMQP format connection string for data ingestion Eventstream supports data ingestion from Azure IoT Hub Improved Eventstream Creation Speed Data Factory Data Pipeline Dataflow Gen2 New Service Principal authentication kind Bug fixes and reliability improvements Community New learning path: Implement a Lakehouse with Microsoft Fabric Core Monitoring hub – column options We have released a new feature inside the monitoring hub to better users customize their experience. Column options gives users more room to operate. Users can select and reorder the columns that meet the scene according to their customized needs. Regardless of whether the user switches to any other tool or scene in Fabric, these columns options are persisted along with the filter until the user returns again. You can now easily reorder the columns with drag and drop inside the option menu. You do not need to worry about the ones at the bottom. All the selected column options will be bubbled up to the top of the menu when you open it next time. Try it out yourself! OneLake OneLake file explorer (v.1.0.10) - Menu Option to View Workspaces and Items Online We are excited to announce the new release of OneLake file explorer for Windows! Now you can seamlessly transition between using the OneLake file explorer app and the Fabric web portal. When browsing data in OneLake file explorer, right click on a workspace and select “OneLake->View Workspace Online.” Selecting this will open the workspace browser on the Fabric web portal. Similarly for items within your workspace. In addition, you can now easily find your client-side logs, which you may need to troubleshoot issues. Right-click on the OneLake icon in the Windows notification area, select Diagnostic Operations, then Open logs directory. Get started by downloading the latest OneLake file explorer. Power BI General Upgrade Power BI Desktop .NET Framework to 4.7.2 or newer Power BI Desktop with .NET Framework 4.5 will no longer be supported after 9/30/2023. Please upgrade to .NET Framework 4.7.2 or newer to avoid any support issues. Reporting Mobile layout interactive canvas With this latest update we are happy to announce that we have made the mobile canvas interactive. This new capability provides the ability to test how buttons, slicers, and visuals will behavior on the app before publishing the report. But that's not all. With this canvas interactivity, users can now interact with visuals directly and adjust Table and Matrix column headers to align perfectly with mobile screens. Smart Narrative improvements The Smart Narrative visual can now display either text or numeric values when showing data for a column. Previously this was limited to only numeric values. Learn more about the Smart Narrative’s summarization capabilities at Create smart narrative summaries - Power BI | Microsoft Learn. Modeling Edit your data model in the Power BI Service - Updates The new data model editing in the Service feature was released to preview in April. We’ve been busy reacting to your feedback and enhancing the experience. Below are the improvements coming later this month: Improve layouts limitations Changes you make to layouts will now persist between Desktop and the Service, including: Changes made to data model layouts in Desktop will now be incorporated into the Service upon uploading the .PBIX file. Similarly, changes made to layouts in the data model within the Service will now be incorporated in the Desktop upon downloading the .PBIX file. Please continue to submit your feedback directly in the comments of this blog post or in our feedback forum. Edit linguistic relationships in Q&A setup The Q&A visual is an effective way to help users further understand their data by asking questions and receiving answers in the form of visuals. It offers users a way to explore their data in ways not covered by the rest of the report without requiring deeper knowledge of their data model or report authoring. However, while the Q&A engine is good at answering precise questions about data, it may not be able to associate every word or phrase a user inputs with data in the model. For example, answering “what are our best consoles this year?” may require connecting the term consoles to the name products in the model, and understanding that the term best corresponds to the highest sales values. These terms are contextual, however – users could mean something completely different asking for console and best in other industries, organizations, or even datasets. To help authors ensure that the Q&A visual provides consistent and accurate answers based on the unique language their report consumers actually use, we introduced Q&A setup tools with an emphasis on providing Q&A with synonyms for column and table names in the model. This way, authors can explicitly define console as referring to products, and users will always receive the correct answers when they ask similar questions in the future. However, synonyms (nouns) are only half of the picture. The other half of the terms (adjectives, verbs, prepositions, adverbs) can’t be defined with such straightforward mappings because they must be understood as a part of a phrase – they qualify other terms or relate other terms together. Best in the previous example is one; asking “who sold the most books” requires us to know that stores sell books, connecting stores to books. There are many types of these linguistic relationships, so we built a new tab entirely to help you create and manage linguistic relationships for your data. You can get into the Q&A setup menu using the gear icon on the Q&A visual or the Q&A setup option in the Modeling tab of the ribbon, then selecting the new Relationships tab. There, you’ll be able to define a variety of relationships, including verb, adjective, noun, preposition, and more. Choose a type which fits the term you’re trying to define (for example, “best” is an adjective), then follow the prompts to define what it means in the context of your data. Our investment into Q&A does not stop here. Even in a world where natural language capabilities are increasingly driven by large language models, there is value in the precision, consistency, and customizability of our sophisticated Q&A engine. In the other direction, defining synonyms and relationships can be a lengthy process just asking to be streamlined with the power of AI-generated suggestions. Keep an eye out in the future for the ways we’re bridging the two to bring out the best of both worlds! DAX function changes to MINX and MAXX We have added an optional variant parameter to MINX and MAXX DAX functions. These functions ignore text and Boolean values when there are variants or mixed data types, such as text and numeric. Now with the new optional variant parameter set to TRUE, the functions will consider the text values. Here is an example of a variant measure. In the table below the variant measure is ordered in ascending order: MINX and MAXX without the variant parameter set, or set to FALSE, will ignore text and Boolean data types. MINX and MAXX with the variant parameter set to TRUE will now include text values. Boolean values are still ignored. The default of the MINX and MAXX optional variant parameter is FALSE, so all existing DAX expressions using MINX and MAXX will not be impacted by this change. To allow text values to be included, you can update the DAX expression to include TRUE as the third parameter. Read more about these changes at Microsoft Learn: MINX function (DAX) - DAX | Microsoft Learn MAXX function (DAX) - DAX | Microsoft Learn Edit relationships in the properties pane (GA) We are excited to announce the general availability of the edit relationships in the properties pane! The edit relationships in the properties pane has been available for public preview since the October 2022 release. Now you can click on any relationship line in the modeling to edit your relationship in the properties pane. This is available in the Model view of Power BI Desktop and in web modeling of the Power BI service. Edit relationships without running queries to preview the data and only validating the relationship when you click apply changes. A welcome relief for those using DirectQuery storage mode, as these queries can take time and impact your data sources. You can learn more about how to utilize this feature at Create and manage relationships in Power BI Desktop - Power BI | Microsoft Learn. Data Connectivity SAP HANA (Connector Update) The update enhances the SAP HANA connector with the capability to consume HANA Calculation Views deployed in SAP Datasphere by taking into account SAP Datasphere’s additional security concepts. This enables consumption of Calculation Views in Datasphere and allows customers to connect to HANA Cloud views without the need for additional privileges on the _SYS_BI schema. Emplifi Metrics (New Connector) We are happy to announce the release of the new Emplifi Metrics connector. Please find release notes from the Emplifi team below: “Integrating social media insights alongside the rest of your marketing or business intelligence data gives you a holistic understanding of your entire digital strategy, all in one place. With Emplifi Power BI Connector, you’ll be able to include social media data from the Emplifi Platform in your charts and graphs and combine them with other data you own. The Power BI Connector is a layer between Emplifi Public API and Power BI itself. It helps you work with your data intuitively, directly in the Power BI tool. The majority of data and metrics available in the Emplifi Public API are also available in the Connector. Please visit the official documentation for more information about Emplifi Public API and a list of available metrics. You’ll find it here: https://api.emplifi.io/.” Service Row-level security test as role improvements We have listened to your feedback about the row-level security test as role experience in the Service and have made several improvements this month including: We have added a new dropdown, allowing you to choose and test any report connected to the dataset. You can now easily see important permissions details pertaining to a specific individual while role testing for that person. We have improved our warning and error messages. To learn more about validating row-level security roles in the Service read our documentation. Mobile Supporting AAD Shared device mode (preview) We, in Power BI continue our investments in empowering frontline workers with data and insights in their work. And in this month release, the Power BI mobile apps add support in Azure active directory shared device mode. Shared Device Mode enables you to configure devices to be shared by a group of employees. This is very common for frontline workers, who do not use their personal device for work related tasks but are getting a work-related mobile device from a shared pool of devices for these tasks. The Shared Device Mode enables single sign-on (SSO) and device-wide sign out for Microsoft Power BI and all other apps that support Shared Device Mode. Once a user signs in into a supported app, the user will be logged into any application compatible with Shared Device Mode as well. Once a user signs out of a supported app, all other applications integrated with shared device mode sign out, to to prevent unauthorized or unintended access by the next user. Back to the frontline worker scenario: when an employee picks a device from a pool at the start of their shift, they need only a single sign-in to one supported app (for example Power BI), and they’re automatically signed in to all other apps that support shared device mode on the device, so the device and the apps are ready for work with the user’s account. At the end of their shift, when they sign out of one app, they're signed out globally from all other apps that support shared device mode, having the device is ready for the next employee and can be safely handed off. Developers Git integration Paginated reports now supported with git integration Since Git integration has been launched few months ago, we supported only 2 items in Power BI- Power BI reports and Power BI datasets. We are now excited to add a new item- Paginated reports! After connecting your workspace to Azure DevOps, you can commit your paginated reports and have them versioned in your repository. After that, you can open the .rdl files directly from git in Power BI Report Builder, edit and push the changes into git. The workspace will identify the changes and will prompt developers to update the workspace with the changes to the paginated report. With this feature, we are adding paginated reports developers to enjoy the collaboration, versioning and modern developer workflows offered in Power BI and Fabric. Learn more about using git with paginated reports. Checkout and Switch Branches in a Workspace A workspace can connect to a single branch at a time. Once you’ve setup your separate workspace to work with git, there might be scenarios where you want to change just the connection of a branch. We have now added 2 new features to help you achieve this much faster: Checkout branch- checkout allows you to easily create a new branch, based on the current state of the workspace. After choosing to checkout, you can create a new branch that will be connected to the WS, while the uncommitted changes are retained. This is useful in cases of conflicts, that allows you to commit your changes to a backup branch, and then manage the merge conflict in the git repo. Switch branch- Workspace admins can decide to change just the connection of the branch very easily. It’s helpful in cases you want to start working on a new branch and wipe clean your WS content for that, or when you are moving from your regular project to small bug fixes (assuming they are on the same content), that should happen on a different branch. New E2E CI/CD tutorial now available Many users have been asking, since the release of git integration, how to work with deployment pipelines and git as part of a CI/CD process. These two features can help build a compelling e2e flow, when connected properly. Follow this step-by-step guide to learn how to make the most out of both tools together. Visualizations New visuals in AppSource Change Chart Pro Circle Card by Devlup Funnels Apex Milestone Trend Dot Chart Activity Gauge by Powerviz Bridger Visual by BI Samurai Number lines by BI-Champ Funnel Chart by Powerviz Date Picker by Powerviz Bar Chart with Export Data Feature Wordcloud by Powerviz PackedBubbleChart Box Ploty by Devlup Funnels LeapLytics - Markdown Viewer Aimplan Data Input Table Beeswarm Chart Overview of Beeswarm Chart Beeswarm chart shows the distribution of data along one axis or both axes while also showing individual points. It is like a one-dimensional or two-dimensional scatter plot but with closely-packed, non-overlapping points. How to use this Visual The visual is intuitive and easy to follow. Only 1 field is mandatory i.e. Bee Category. Tip: If you don’t have any category, you can simply add a calculated column with any name and use it in the category field Bee Size field is optional and is represented in the visual by bubble area or size. If this field is missing then all bubbles will be of same size Bee Label field is also optional. It is used to add labels to each individual bubbles. This chart can be used in 3 ways Y – Axis only X – Axis only Scatter Plot (Both Y and X Axes) 1) Y-Axis Only As shown in picture below, if you only use a column/field in “Beeswarm Y-Axis” section while keeping “Beeswarm X-Axis” section empty, your data will be plotted on Y-Axis categorized according to the “Bee Category” section 2) X-Axis Only Similarly, if you only use a column/field in “Beeswarm X-Axis” section while keeping “Beeswarm Y-Axis” section empty, your data will be plotted on X-Axis categorized according to the “Bee Category” section 3) Scatter Plot (Both X and Y Axes) If you use columns/fields in both sections i.e “Beeswarm X-Axis” section and “Beeswarm Y-Axis” section, the visual will work like a scatter plot with data points plotted on X-Axis and Y-axis . This is depicted below You can show images inside bubbles as well using the “Images” field. For example, the report below shows top millionaires of the world with their net worth on Y-Axis and their Age on X-Axis. Its available in demo file. Formatting Options These options are available in the format pane of the visual under “Settings”. 1) Bubble size As the name implies, this setting simple increases / decrease the bubble size 2) Lower Upper and Left Space Beeswarm chart produces non colliding bubbles or in other words circles don’t overlap. This can often cause bubbles do go outside the chart area especially when there is a concentration of data around minimum and maximum data values. This is where these settings are very useful to bring data inside the chart area. These settings increase or decrease the Y-Axis and X-Axis range to achieve this objective. Try adjusting these settings on the demo file’s sheet “ScatterPlot Beeswarm” to see how they work. Purchase of Premium Features To use the visual without Watermark in Power BI Online, please purchase the Visual’s license for a 1 year period. The license is for unlimited number of users and viewers. If you have any questions, please send me email at [email protected] Download Download the demo file here. Download the custom visual from APPSOURCE Explore time-based data down to each millisecond with Drill Down TimeSeries PRO Drill Down TimeSeries PRO lets you create timeseries charts that are easy to explore on any device. Using its intuitive interactions, users can drill down to months, days or hours by clicking on the chart. Combine up to 12 series and choose between multiple charts - line, column, area. Learn more about Drill Down TimeSeries PRO! MAIN FEATURES: On-chart interactions – pan, zoom and drill down Full customization – set intervals, min/max values, colors, fonts, stacking and clustering Static and dynamic thresholds – set up to 4 thresholds to demonstrate KPIs Cross-chart filtering – select data points on multiple charts instead of using slicers Touch device friendly – explore your data anywhere POPULAR USE CASES: Finance – stock exchange indices, capital ratios, transaction volumes Sales – web traffic, audience reach, lead volume IT – network traffic, response times, syslog and error trends Logistics – inventory movement and turnover, loading time, fleet maintenance costs Get Drill Down TimeSeries PRO now! This visual comes with 30 days free access to paid features. ZoomCharts Drill Down Visuals are known for interactive drilldowns, smooth animations and rich customization options. They are mobile friendly and support: interactions, selections, custom and native tooltips, filtering, bookmarks, and context menu. New Updates for accoPLANNING (Release 57) accoPLANNING for Power BI empowers business users with writeback capabilities, advanced planning, forecasting, budgeting, project management and analysis solutions. We have some new and exciting features for you in this release: • Custom calculated rows and columns configuration. It allows you to customize your own client-side calculations in your already established accoPLANNING table. • End user Lock cells. This gives the flexibility for the end user to lock cells and make sure these will not be affected by splashing or use as an approval indicator. • Hide rows and columns. This Allows users to hide rows and columns - adding flexibility in making asymmetric column and row selections in the grid for better reporting or just for ad-hoc purpose to get a better overview. The latest update also includes a range of new features and improvements designed to increase your productivity: • Our navigation has undergone a major overhaul, with the most notable change being the revamped toolbar. • Better API error message handling in the grid. • Support for automatically expanding all rows/columns/both, eliminating the need to manually adjust the size of each cell. With the accoPLANNING visual, you combine the planning and reporting process in Power BI. For more information, visit our website. https://www.accotool.com/ https://appsource.microsoft.com/en-us/product/power-bi-visuals/WA200002600?tab=Overview https://youtu.be/kNIVC6rBJWA Zebra BI Charts on-visual settings Zebra BI visuals for Power BI are taking another step further to make the user experience as seamless and interactive as possible. Zebra BI Charts has received some important updates so that you save time while customizing your visuals. Thanks to the on-visual settings, you don’t need to go to the visualizations pane every time you want to make a change. COMMENTS SETTINGS can now be adjusted directly on the visual by simply clicking on the settings icon next to the comment box. You can adjust the title, which variances you display, the icon, gap between the comments, and padding. CATEGORY SETTINGS let you adjust several settings by just clicking on the category area. Customize the axis font (family and color), trim /rotate long labels, adjust the label density, and set the gap between the columns. LEGEND SETTINGS come in handy when you want to rename the entries and don’t want to search for this option in the formatting pane. Adjust margins, use aliases in tooltips and switch comparisons. STACKED CHARTS SETTINGS on the visual let you adjust the Top N feature, set color of the chart, and display labels as %. Try it on your data for free. Word Cloud by Powerviz We are excited to announce the new Word Cloud by Powerviz, it's an advanced visual which empowers you to create some of the most high-quality and creative word art in the Power BI. Key Features: Word Styling: Make your word clouds pop with personalized text styles. It offers font styling, direction & text editing features. Color Options: Choose from 30+ color palettes, including color-blind safe options. Shapes: Want to make a statement? Create captivating word clouds by choosing shapes from icons and images, or upload your own image. Exclude: No more hassles anymore! Easily remove unwanted words, symbols from the text to create a clean and focused word cloud. Ranking: Filter out Top/Bottom N Words. Conditional Formatting: Easily spot words with dynamic rules. Many other features included lasso/reverse lasso, grid view, show condition, and accessibility support. Business Use Cases: Marketing: Sentiment analysis & SEO keywords. Education: Brainstorming, Improve engagement. Market Research: Opinion poll, open-ended survey. Presentation: Capture attention & easy communication. Try Word Cloud Visual for FREE from AppSource Check out the visual features in demo file Step by Step instructions and documentation To learn more, visit Powerviz website. Introducing Word Cloud by Powerviz – A Powerful Power BI Custom Visual on YouTube Innofalls Charts: A Versatile and Interactive Visual Innofalls Charts is a powerful visual that offers a wide range of interactive features. Our bar and waterfalls charts come with a drill-down history, enabling orientation and facilitating comparisons. All charts can be stacked or grouped into columns. Waterfalls: Our special attention to waterfalls includes running totals and delta waterfalls, enabling effective comparisons of measures or members. The waterfalls are expandable, drillable, stackable, and offer intermediate sums. Drill Down History: Keep track of your drill downs with visual feedback. Change your drill downs retroactively and combine multiple drill downs for in-depth exploration. New Comparison Features: Effortlessly make comparisons while viewing your report and incorporate deviation charts with a single click. Interactivity: Enjoy various interactive features such as drill, expand, undo/redo, dynamic comparisons, and zoom for enhanced data exploration. IBCS Formatting: Our IBCS theme provides scenario formatting, equal scaling, highlights, deviations and more for consistent and standardized reports. Animations: Enhance data comprehension and reduce change blindness with animations. Promptly detect resorted or added data elements. Discover the full potential of Innofalls Charts and get started today. Innofalls Charts is free for Power BI Desktop! Get started today [ https://www.innofalls.com/blog-get-now ] Visit our website [ https://www.innofalls.com/ ] Hierarchy Chart by MAQ Software Struggling to visualize complex hierarchical relationships? Improve understanding and decision-making with the Hierarchy Chart by MAQ software. In today's data-driven environment, visualizing hierarchical relationships is key to business insights and decision-making. From sales structures to budget allocations, our Power BI-verified visual offers a customizable way to represent these structures easily, catering to various business needs. Figure 1: Sample visual showing a color-coded organization chart (with a tooltip). Key business uses: Organization/HR: Organize human resources by department and hierarchy. Operations: Illustrate manufacturing processes, breaking down components or ingredients hierarchically. Finance: Show budget allocations by division or project, with color-coded subcategories to highlight differences. Sales: Visualize sales structure, territory responsibilities, and targets with group field indicators showing performance status. IT: Visualize IT assets by category and availability with detail. Key features: Color-code cards using a legend. Adjust card (i.e. box) size, borders, and corners. Customize the appearance of the links connecting each card. Interact with cards to control levels and cross-filter visuals. Zoom and reposition the visual with ease. Try out the features of the Hierarchy Chart by MAQ Software today on the visual’s sample report. Learn more about our Power BI custom visuals on our website. Power BI JSON Report Theme Generator by BIBB https://powerbithemegenerator.bibb.pro/ BIBB recently released its take on the Power BI Theme Generator, aiming to streamline the theming process for users. This new tool offers a user-friendly interface, allowing for an easy selection of colours and generation of JSON themes. Within BIBB's generator, users can choose colours in various ways, from manual selection to trending combinations and even importing from images or external sources like Coolors. Synapse Data Warehouse Data Warehouse Utilization Reporting in Fabric Metrics app We are excited to announce that Compute utilization reporting for Data warehouse and SQL Endpoint are now available in Public Preview for Microsoft Fabric across all regions! In the capacity-based SaaS world of Microsoft Fabric, customers can purchase a Fabric SKU and get entitled to a set of Capacity Units (CU). Various workloads, including data warehouse, consume capacity units based on usage. Fabric metrics app provides visibility into capacity usage for all Fabric workloads, including data warehouse in one place. It is used by Capacity or SQL warehouse admins to identify CU usage trends across warehouse items within a capacity, monitor overload information, and understand the cost of running operations which leads to informed capacity sizing decisions. For more information, see the detailed blog Data Warehouse Utilization Reporting in Fabric Capacity Metrics App Column-Level security is now available on Data Warehouse and SQL Endpoint We are excited to announce that Column-Level security is now available on Data Warehouse and SQL Endpoint. Column-level security simplifies the design and coding of security in your application, allowing you to restrict column access to protect sensitive data. For example, ensuring that specific users can access only certain columns of a table pertinent to their department. The access restriction logic is located in the database tier rather than away from the data in another application tier. The database applies the access restrictions every time data access is attempted from any tier. This restriction makes your security more reliable and robust by reducing the surface area of your overall security system. In addition, column-level security also eliminates the need for introducing views to filter out columns for imposing access restrictions on the users. You can implement column-level security with the GRANT T-SQL statement. Only Azure Active Directory authentication is supported. Data Warehouse Row Level Security We are thrilled to announce that Row-Level security is now available in Fabric Warehouse and SQL Endpoint. Row-Level Security enables you to use group membership or execution context to control access to rows in a database table. Row-Level Security (RLS) simplifies the design and coding of security in your application. RLS helps you implement restrictions on data row access. For example, you can ensure that workers access only those data rows that are pertinent to their department. Another example is to restrict customers' data access to only the data relevant to their company. The access restriction logic is located in the database tier rather than away from the data in another application tier. The database system applies the access restrictions every time that data access is attempted from any tier. This makes your security system more reliable and robust by reducing the surface area of your security system. Implement RLS by using the CREATE SECURITY POLICY Transact-SQL statement, and predicates created as inline table-valued functions. SQL Projects support for Fabric Data Warehouse We are excited to announce the Microsoft Fabric Data Warehouse as a supported target platform in the SQL Database Projects extension available inside of Azure Data Studio! SQL Database Projects for Azure Data Studio provides a way to design, edit, and publish schemas for SQL databases from a source controlled project. For a complete development workflow, build and deploy your database projects in CI/CD pipelines, such as GitHub Actions or Azure DevOps. A SQL project is a local representation of SQL objects that comprise the schema for a single database, such as tables, stored procedures, or functions. Use the SQL Database Projects to extract and publish warehouse schemas directly from and to Fabric Data Warehouse. Other compatible databases include SQL Server, Azure SQL Database, Azure SQL Managed Instance, and Azure Synapse SQL (serverless and dedicated). Get started today with the insiders build of ADS. Deployment Pipelines In today’s world, analytics is a vital part of decision making in almost every organization. Fabric's deployment pipelines tool provides BI creators with a production environment where they can collaborate to manage the lifecycle of organizational content. Deployment pipelines enable creators to develop and test content in the service before it reaches the users. Supported content types include reports, paginated reports, dashboards, datasets, dataflows, and now warehouses! The deployment process lets you clone content from one stage in the deployment pipeline to another, typically from development to test, and from test to production. The connections between the copied items are kept during the copy process as well. In addition, Fabric applies the configured deployment rules to the updated content in the target stage. You can also deploy content programmatically, using the deployment pipelines REST APIs. You can learn more about this process in Automate your deployment pipeline using APIs and DevOps. Default Dataset Improvements We have released some new capabilities to enhance the default dataset experience: Turn off automatically adding new objects to the default dataset – navigate to the Warehouse artifact settings to toggle this on/off Updates to the manage default dataset dialog in the model view Filter by schema Filter by object (table/view) We’ve heard your feedback about the ability to turn off the default dataset. Thank you for the feedback, we’re planning on delivering this feature in the coming months! Data Engineering VS Code Integration Introducing Notebook File System support in Synapse VS Code extension for Data Engineering and Data Science workload in Microsoft Fabric The Synapse VS Code extension empowers users to develop their notebook artifacts directly within the VS Code environment. Once users download the .IPYNB file to their local desktop, they gain the flexibility to edit, debug, and execute their notebook code locally. Furthermore, they have the option to select the kernel provided by the extension to execute and debug their notebook code on the remote Fabric Spark compute. Python developers commonly structure reusable functions as modules in the form of .py files, a well-established software engineering best practice. Now, with the incorporation of the Notebook File System within the VS Code Synapse extension, developers can apply this principle to their Fabric notebook development. In addition to running and debugging notebook code, developers can also execute and debug code from imported modules. This integration enhances code modularity and facilitates efficient development workflows. When a user clicks "Open Notebook Folder," the VS Code Synapse extension not only opens the selected .IPYNB file but also downloads all associated files, including .PY modules, from the notebook file system to the local VS Code environment. This feature ensures that the entire set of necessary files and modules is readily available for seamless development and collaboration within the VS Code interface. In this example, there is a .PY module named US2Europe.py which contains a function to covert the datetime format from US fashion to Europe’s one. To invoke this function from the notebook side and execute it on the worker node, you need to import the module containing the function and apply into the data frame To synchronize changes between the local environment and the remote workspace in the VS Code Synapse extension, users can take advantage of two key actions: Publish Resource Folder: Clicking the "Publish Resource Folder" button enables users to upload their local changes to the remote workspace. This ensures that any modifications made locally are reflected in the remote environment. Update Resource Folder: The "Update Resource Folder" button allows users to pull changes from the remote workspace into their local VS Code environment. This ensures that any updates made by collaborators or on the remote side are mirrored locally. In the event that a file has been deleted in the remote workspace but still exists in the local environment, the Synapse extension will automatically handle this situation as follows: The deleted file will be moved to a backup folder named "__backup" under the root directory of the building folder. This approach preserves a copy of the deleted file for reference, ensuring that no data is lost during the synchronization process. These actions provide users with an efficient and robust means to manage the synchronization of their files and code between local and remote environments while maintaining data integrity. Import Notebook in Workspace view We are excited to announce that we now have the “Import Notebook” entry on the Workspace -> New -> Import item! You can easily import one or more files (support .ipynb, .py, .sql, .scala, .r file format) as Fabric Notebook items in the target workspace. Mssparkutils new API for fast data copy We now support a new method in mssparkutils that can enable large volume of data move/copy much faster, which is: Mssparkutils.fs.fastcp() You can use mssparkutils.fs.help("fastcp") to check the detailed usage. According to our benchmark testing, we achieved ~7x to ~180x speed up (varies as different source type, data size and regions) compares to the traditional file system copy method mssparkutils.fs.cp(). As the below example shows, it takes ~5s to copy a 10G file from ADLS Gen2 account to Fabric Lakehouse. Support diff versions when saving notebook We now support viewing and comparing the differences between 2 versions of the same notebook when there are saving conflicts. This is particularly useful when more than one users are working on a same notebook instance via different clients, like: Publish notebook from Fabric VSCode extension Some users are using Manual save mode while others are collaborating Collaborating on notebook with unstable network With the diff function the users in collaboration session can get notified and compare the changes from other user’s edits, and they can choose a version to keep, or save a copy of any version, to easily handle the save conflicts without losing the code accidently. Notebook Sharing with Execute-only mode Fabric Notebook now supports checking the “Run” operation separately when sharing a notebook, if you just selected the “Run” operation, the recipient would see a “Execution-only” notebook after opening the item. With the “Execution-only” mode you can easily spread your notebook without worrying about someone messing up the contents, while the recipients can still run the cells. Notebook resources .whl file support We now support uploading the .whl files in the Notebook Resources explorer, you can put your own compiled libs here, use Drag & Drop code snippet to install them in the session and import the libraries in code conveniently. Real-time Analytics OneLake shortcut to delta tables from KQL DB Now you can create a shortcut from KQL DB to delta tables in OneLake, allowing in-place data queries. With this enhancement, you can effortlessly query delta tables in your Lakehouse or Warehouse directly from KQL DB. Learn more. Real-Time Data Sharing: Introducing database shortcuts in Real-Time Analytics A database shortcut in Real-Time Analytics is an embedded reference within a KQL database to a source database in Azure Data Explorer allowing in-place data sharing. The behaviour exhibited by the database shortcut is similar to that of a follower database. If you are a data provider hosting data in Azure Data Explorer and want to share this data with consumers in Microsoft Fabric, you can create a database shortcut. Database shortcut enables sharing in real-time, with no need to build or maintain a data pipeline. All database changes, including schema and data, on the provider side are instantly available on the consumer side. You can then consume data in Microsoft Fabric by using a KQL Queryset, PBI, Notebooks etc. easily . While today, you can share data from an Azure Data Explorer database with a KQL Database, soon you will also be able to share data from a KQL Database with other KQL Database(s) in Microsoft Fabric. Check out the announcement blog to learn more: Real-Time Data Sharing in Microsoft Fabric blog Sample Gallery – Explore what you can achieve in Real-Time analytics Real-Time analytics now offers a comprehensive sample gallery with multiple datasets allowing you to explore, learn and get started quickly. Selecting a sample in Real-Time Analytics automatically creates a KQL Database with data and prewritten KQL Queryset from any of the 6 sample datasets. This will let users to get started with KQL Database and KQL Queryset without the hassle of creating database, finding the right real time dataset, ingesting data, and writing queries. Each sample lets you explore different streaming and real time scenarios. The samples include data and queries for Stock analytics, Weather analytics, IOT analytics, Log analytics, Metrics analytics and Automotive operations analytics. Each dataset is accompanied by a KQL Queryset which includes sample queries introducing you to KQL capabilities such as. aggregations, search , transformations using update policies, user defined functions, geospatial and time-series analytics. Model and Query data as graphs using Kusto Query Language (KQL) We are excited to share with you a new public preview feature of Kusto Query Language (KQL) that allows you to model and query data as graphs. Graphs are useful for representing complex and dynamic data that involve many-to-many, hierarchical, or networked relationships, such as social networks, recommendation systems, connected assets, or knowledge graphs. By using KQL with graph semantics, you get the speed and scale of KQL queries with the flexibility and expressiveness of graphs. You can query graphs using KQL graph semantics, which has a simple and intuitive syntax that works well with the existing KQL features. You can also mix graph queries with other KQL features, such as time-based, location-based, and machine-learning queries, to do more advanced and powerful data analysis. For example, you can use time-aware graphs to examine how the graph evolves over time, or use geospatial queries to analyze the spatial distribution or proximity of nodes and edges. To learn more about graph semantics in KQL, check out the following resources: - Introduction to graph semantics in KQL: https://aka.ms/kusto/graph-overview - Graph operators and functions reference: https://aka.ms/kusto/graph-operators We hope you enjoy this new feature and find it useful for your data analysis needs. As always, we welcome your feedback and suggestions on how to improve KQL and its graph semantics extension. Happy querying! Easily connect to KQL Database from Power BI desktop Power BI desktop just released new ways to easily connect to a KQL database. There are two ways to easily locate your KQL Database: Open the Get Data dialog and select the Microsoft Fabric menu item. Select KQL Databases from the list: Open the OneLake data hub dialog from the main ribbon. Select KQL Databases from the list: Eventstream now supports AMQP format connection string for data ingestion AMQP, stands for Advanced Message Queuing Protocol, is a protocol that supports a wide range of messaging patterns. It's a widely used communication protocol that facilitates efficient and reliable communication among IoT devices and systems. Now in Eventstream, you can create a Custom App source or destination and select AMQP format connection string for ingesting data into Fabric or consuming data from Fabric. Eventstream supports data ingestion from Azure IoT Hub Azure IoT Hub is a cloud-hosted solution that provides secure communication channels for sending and receiving data from IoT devices. With Eventstream, you can easily stream your Azure IoT Hub data into Fabric and perform real-time processing before storing it in Kusto Database or Lakehouse. To add an Azure IoT Hub source, follow these three simple steps: Choose "Azure IoT Hub" as your eventstream source. Give this source a name and enter your Azure IoT Hub details. You may need to enter the share access key from your Azure IoT Hub to establish the cloud connection. On the canvas, select the IoT source and click "Data Preview" to get a preview of the incoming data from your Azure IoT Hub. Improved Eventstream Creation Speed In response to user feedback, we've made significant improvements to the Eventstream creation, enabling you to create an Eventstream item in seconds. Setting up your Eventstream is now faster and smoother than ever before. These improvements save you valuable time and allow you to ingest and transform your data streams more efficiently. Data Factory Data Pipeline The Outlook email & Teams activities in Data Factory Pipelines can now be used in collaborative shared pipeline New pipeline activities are now available as well! If you’ve used the Custom activity type in ADF pipelines in the past, you’ll be very familiar with the new Azure Batch activity in Fabric data pipelines. This will allow you to execute custom executables from Azure Batch for powerful pipeline workflows. We’ve also enabled Azure Functions and KQL scripts as activities in Fabric pipelines. We've updated the Pipeline output monitoring view to focus on simplification, easily representing pipeline run status with customizable output layouts Pipeline lineage now includes lineage for Connections Dataflow Gen2 New Service Principal authentication kind Service Principal has been added as an authentication type for a set of data sources that can be used in Dataset, Dataflow, Dataflow Gen2 and Datamart. Azure service principal is a security identity that is application based and can be assigned permissions to access your data sources. Service principals are used to safely connect to data, without a user identity. Supported data sources include: Azure Synapse analytics Azure SQL database Azure data Lake gen 2 Azure data lake Azure blob storage Web Dataverse SharePoint online Note: service principal is not supported on the on-premises data gateway and virtual network data gateway. Support is planned by the end of 2023. Read more about Service Principal support from the official announcement post. Bug fixes and reliability improvements We continue to listen to your feedback and we’re actively working and making strategic fixes and quality improvements to our service. The following is a short list of the most impactful fixes and improvements delivered in the past couple of weeks. Revert changes for the limit of 50 queries per dataflow. In our previous monthly update we implemented a limit to the number of queries that a dataflow could have. We reverted this limit until further notice due to a number of undesired experiences created, but we will reintroduce this limit again in the near future. If you have dataflows with more than 50 queries currently, we encourage you to start splitting them into multiple dataflows with lower number of queries, as this will give you a head-start when the limit is reintroduced in the future. This documentation article describes several best practices when developing complex dataflows that you may find useful: Best practices for designing and developing complex dataflows - Power Query | Microsoft Learn Multiple fixes and improvements to common error messages and scenarios. Thanks to multiple customers who have reached out to us through our support system and our Community Forum, we’ve been able to address several common issues and improve the error messages that were quite frequent. Some of the error that have been fixed and/or modified to provide more descriptive information are: “The current row is too large to write.” “We cannot convert the value null to type Table.” “Dataflow refresh transaction failed with status Failed” Errors with Dataflow connector when creating linked entities. This issue has been fixed. We want to emphasize our commitment to our customers on improving the product and take your feedback into account in our decision making. We encourage you to engage directly with us through our support channel or through the Community forums. Community New learning path: Implement a Lakehouse with Microsoft Fabric To learn more and get started, check out the Implement a Lakehouse with Microsoft Fabric learning path. That is all for this month! Please continue sending us your feedback and as always, keep voting on Ideas to help us determine what to build next. We are looking forward to hearing from you!
Solved Fabric Community posts are now available in the Fabric Help Pane
The Fabric Help Pane provides Fabric users with a fast and efficient way to access self-help content, allowing them to resolve issues independently or direct them to create a support ticket if further help is required. Following its launch last year, we've heard your feedback to improve the relevance of our self-help resources. We are pleased to announce that now you can find solved posts from Fabric Community discussions relevant to your search in the Fabric Help Pane. To navigate to the help pane, start by clicking on “?” on the top right corner of the screen. Once you are on the help pane, you can search for any issues for which you are trying to find resolution. The search results will return relevant content from both Microsoft Fabric Docs and Community. _Community_posts_are_now_available_in_the_Fabric_Help_Pane If you are only interested in Community posts search results, you can also user the filter and select “Forum Topics” If you have any feedback on the self-content you would like to see in the help pane, please leave us a comment in the Join Discussion section below.
Fabric April 2025 Feature Summary
Welcome to the Fabric April 2025 Feature Summary! This update brings exciting advancements across various workloads, including Low-code AI tools to accelerate productivity in notebooks (Preview), session Scoped distributed #temp table in Fabric Data Warehouse (Generally Available) and the Migration assistant for Fabric Data Warehouse (Preview) to simplify your migration experience. Contents Community & Events Get certified in Fabric – for FREE. Free live learning sessions for Data Engineers General Fabric Copilot and AI Capabilities available on all paid SKUs Data Science Low-code AI tools to accelerate productivity in notebooks (Preview) Low-code AI capabilities in Data Wrangler (Preview) Data Warehouse ALTER Table Drop Column and sp_rename column support in Fabric Warehouse (Generally Available) Session Scoped distributed #temp table in Fabric Data Warehouse (Generally Available) Migration assistant for Fabric Data Warehouse (Preview) OPENROWSET function (Generally Available) BULK INSERT statement (Generally Available) Real-Time Intelligence Fabric special for Kusto Detective Agency: solve the Digibus real-time crisis Azure monitor data sources are now fully integrated with KQL Queryset Improvements to Data Exploration (low-code) experience Eventhouse system and KQL Database overview: in-item monitoring enhancements Eventstream's Real-time Weather Connector SQL database in Fabric New regions supported Backup billing Performance dashboard Terraform support, Rest API & CLI support Integrations Graph database support Data Factory Mirroring Mirroring for Snowflake protected by a firewall(Preview) Closing https://youtu.be/shz68BKiibg?si=iPbDBTTkMu6E59WB Community & Events Get certified in Fabric – for FREE. As part of the Microsoft AI Skills Fest, Microsoft is celebrating 50 years of innovation by giving away 50,000 FREE Microsoft Certification exam vouchers in weekly prize drawings. Enter the sweepstakes now to have the most chances to win a free exam voucher for DP-600 or DP-700. Free live learning sessions for Data Engineers Whether you're new to Microsoft Fabric or building on your existing skills, these sessions, hosted by Microsoft Fabric experts, give you the knowledge and confidence to get certified and take your data engineering career to the next level. Register now - live sessions in English start April 30 th . Available on-demand in Spanish and Portuguese. General Fabric Copilot and AI Capabilities available on all paid SKUs We are thrilled to announce a major update in Microsoft Fabric! Starting today, the SKU requirement for Copilot and AI features will be lowered to F2, making it much more accessible for you to explore, test, and utilize Fabric AI capabilities. This exciting change grants you full access to Copilot in Fabric, Fabric data agents, and Fabric AI Functions—all designed to enhance productivity, uncover insights swiftly, and seamlessly enrich your data. With this update, more teams will have the opportunity to experiment with AI-driven workflows within their existing capacity. It's worth noting that while the smallest SKUs provide full feature access, they will support a more limited number of AI requests due to their smaller capacity size. Nonetheless, this will allow more users to experience the benefits of AI capabilities and improve their workflows. We look forward to seeing the creative ways your team will utilize these AI capabilities to boost your projects and productivity! Data Science Low-code AI tools to accelerate productivity in notebooks (Preview) Fabric notebooks allow you to accelerate your productivity with native AI capabilities like Copilot and AI functions. A new notebook tab devoted to AI and ML tools now provides low-code shortcuts for transforming data with Data Wrangler, training custom models with AutoML, chatting with Copilot, and more. Among the updates is a low-code interface to apply AI functions for seamless LLM-powered data enrichment. Just select one of the functions, choose an input pandas or Spark DataFrame and a target column to transform, and fill in any required parameters. Fabric will produce the code for you. To learn more, refer to the Transform and enrich data seamlessly with AI functions documentation. Low-code AI capabilities in Data Wrangler (Preview) All Fabric notebook users have access to Data Wrangler, a low-code tool with an immersive interface for exploring and transforming pandas or Spark DataFrames. Data Wrangler provides a library of common data-cleaning operations that you can browse and apply seamlessly getting real-time previews and generating reusable code. We have new AI-powered capabilities coming to Data Wrangler later this month: Automated suggestions with rule-based AI: A new set of automated suggestions will analyze your data and use rule-based AI from the Microsoft PROSE team to highlight the most relevant Data Wrangler operations for you. Convert natural language to code with Copilot: Need an operation that you don’t see in Data Wrangler? You can now use Copilot to generate custom code. As with any Data Wrangler operation, you’ll get a preview before applying or discarding it. Use AI to translate custom code from pandas to PySpark: Data Wrangler automatically converts Spark DataFrames to pandas for performance reasons, then translates your applied code back to PySpark when you export it. With GenAI in Data Wrangler, custom code operations will also be translated to PySpark—whether you type them in yourself or generate them with Copilot. Data Warehouse ALTER Table Drop Column and sp_rename column support in Fabric Warehouse (Generally Available) There are two powerful new features in Fabric Warehouse that we are happy to introduce: ALTER TABLE DROP COLUMN and SP_RENAME COLUMN. ALTER TABLE DROP COLUMN effortlessly removes unnecessary columns to streamline storage, boost performance, and improve query efficiency. Cloning a table as of a point in time & time travel to a point in time that is before the table was dropped is not supported. Dropping columns from Lakehouse tables is not a supported scenario. SP_RENAME COLUMN easily renames columns without downtime, making schema adjustments faster and reducing the risk of errors.Columns and Tables are not renamable in Lakehouse. These new features make it easier to maintain a clean and efficient data model, allowing your team to quickly adapt to evolving business needs with minimal disruption. Check sp_rename Microsoft learn and ALTER TABLE (Transact-SQL) - SQL Server for additional details and syntax. Session Scoped distributed #temp table in Fabric Data Warehouse (Generally Available) Are you unable to manage intermediate query results efficiently in your batch jobs? Fabric Data Warehouse users can now create session-scoped #temp tables to handle these results seamlessly. These temp tables can be backed by either Parquet (distributed) or mdf (non-distributed), offering flexible options to cater to different needs. Users can create two types of #temp tables: Non-Distributed Temp Tables (mdf-backed) - These are created using syntax like user tables in Fabric DW, with the key difference being the need to prefix the table name with ‘#’. CREATE TABLE #table_name ( Col1 data_type1, Col2 data_type2 ); Distributed Temp Tables (parquet-backed) - These tables are distributed and created using the following syntax: CREATE TABLE #table_name ( Col1 data_type1, Col2 data_type2 ) WITH (DISTRIBUTION=ROUND_ROBIN); Note: data_type1 and data_type2 are placeholders for the supported data types in Fabric Data Warehouse Data types. For additional details on why we offer two types of temp tables, the scenarios they support, and their limitations, refer to the Session-scoped distributed #temp tables in Fabric Data Warehouse documentation. Start leveraging session-scoped temp tables in Fabric Warehouse to streamline your data processing tasks and enhance your workflow efficiency. Happy querying! Migration assistant for Fabric Data Warehouse (Preview) The Migration Assistant for Fabric Data Warehouse is now in preview. The migration experience is built natively into Fabric and enables Azure Synapse Analytics (Data Warehouse) customers to transition seamlessly to Microsoft Fabric. This new DW migration experience allows users to easily migrate both metadata and data from the source database, automatically converting the source schema to Fabric Data Warehouse, helping with data migration, and providing AI powered assistance. With integrated assessment tools and guided support, this capability simplifies migration, enabling customers to leverage Fabric’s capabilities without the complexity of traditional migrations. The Migration Assistant for Fabric Warehouse streamlines the migration process into four steps: Metadata migration Problem resolution Data copying Connection rerouting Each of these steps are explored in detail in this Migration Assistant for Fabric Data Warehouse (Preview) blog post. For a more comprehensive guide, you can also review the migration assistant how-to article for step-by-step instructions and the Fabric Migration Assistant documentation for more in-depth information. OPENROWSET function (Generally Available) The OPENROWSET function in Fabric Data Warehouse and Fabric SQL endpoint are generally available. The OPENROWSET function enables you to seamlessly read Parquet and CSV files stored in Azure Data Lake Storage and Azure Blob Storage, as it is shown in the following example: SELECT TOP 10 * FROM OPENROWSET( BULK 'https://<storage>.blob.core.windows.net/container/file.parquet' ) With OPENROWSET, you can easily browse files before loading them into the Fabric Data Warehouse, allowing you to inspect the schema before creating the target table. This function provides several valuable features that significantly enhance the data ingestion experience: Referencing custom folder structures – the OPENROWSET function can reference URI patterns by using * (wildcard) and /** (recursive child wildcard) that can match multiple source files that match the same pattern or return all files that are recursively placed under the URI. Reading partitioned data sets – the OPENROWSET function can retrieve partition values from the folder names, which is crucial if you are reading data from hive-style partition structures and return these values in the result set. Reading Parquet complex types – the OPENROWSET function supports complex types such as struct, array, and map, returning them as JSON text for easier manipulation and analysis. Customizing result set scheme – the OPENROWSET function allows you to map the result set columns to the source columns and define the optimal column types for all columns using the WITH clause, providing flexibility in how data is presented and utilized. OPENROWSET supports most of the options available in SQL Server, Azure SQL, and Synapse, facilitating seamless migration and code reuse between these platforms. Ingesting data with CTAS or INSERT SELECT statements – the OPENROWSET function enables you to ingest data using Create Table As Select (CTAS) or INSERT SELECT statements by using the OPENROWSET as a source and allowing you to modify the values from the source values at ingestion time. This is crucial in scenario where you need to modify source data that is not in expected format. The OPENROWSET function will significantly improve your data ingestion experience by enabling you to browse files, transform data during ingestion, and facilitate easier migrations from Synapse, SQL Server, and Azure SQL Database to Fabric Data Warehouse. This powerful functionality ensures that you can handle complex data types, manage partitioned data efficiently, and customize your result set schema to meet your specific requirements, all while maintaining compatibility with existing SQL options. BULK INSERT statement (Generally Available) The BULK INSERT statement in Fabric Data Warehouse is generally available, it enables you to ingest data into a table from the specified file path: BULK INSERT table_name FROM file_url_path The BULK INSERT statement is very similar to the COPY INTO statement and enables you to load data from external storage. The key value of BULK INSERT is that it supports traditional SQL Server and Azure SQL syntax, thus facilitating an easy migration of SQL Server databases to the Fabric Data Warehouse without the need for code changes. Additionally, BULK INSERT supports several traditional options used in SQL Server, such as text/xml format files that are used by bcp tool and importing non-Unicode files with the custom code pages. This compatibility ensures that you can migrate your databases to the Fabric Data Warehouse with minimal code changes to your ingestion code while retaining your existing ingestion logic without altering the input files. By leveraging the BULK INSERT statement, you can maintain your data ingestion workflows and schemas, ensuring a seamless transition to the Fabric Data Warehouse. This feature not only preserves the integrity and structure of the data but also enhances the efficiency of the migration process, reducing the potential for errors and downtime. As a result, businesses can continue to operate smoothly while taking advantage of the advanced capabilities and scalability offered by the Fabric Data Warehouse. Real-Time Intelligence OpenAI plugins for Eventhouse Two new OpenAI plugins are available to generate embeddings and leverage the power of OpenAI models within the Eventhouse context. You can use the plugins to build Retrieval Augmented Graph (RAG) applications or augment your data analysis with OpenAI models. ai_embed_text : Integrates OpenAI embedding models to generate embeddings within KQL. ai_chat_completion: Leverages the power of ChatGPT and other OpenAI models to augment data analysis within the Eventhouse context. Fabric special for Kusto Detective Agency: solve the Digibus real-time crisis This challenge is specifically optimized for onboarding to the RTI platform. You can use it to enhance onboarding for customers and create CTF (Capture the Flag) tournaments to learn with hands-on experience. During this quest, you'll learn KQL and utilize powerful tools including: Eventstream: Process and transform real-time data Eventhouse: Store and query massive datasets Real-Time Dashboard: Visualize critical metrics Activator: Trigger automated responses Put your detective skills to work helping the Digibus Digitown transit company solve its crisis. Follow the clues, analyze the data, and uncover the solution with the power of Real-Time Intelligence. Lucky finishers will win prizes! Don't miss this opportunity to enhance your RTI skills while solving an engaging mystery. Get started with Kusto Detective Agency. ure_Summary Azure monitor data sources are now fully integrated with KQL Queryset KQL Queryset has always supported cross-service queries, but now we’re making Azure data sources even more accessible. Application Insights (AI) and Log Analytics (LA) are now first-class citizens, just like Eventhouse and ADX, providing a more seamless and intuitive experience. With this update, you can: Run cross-service queries between Log Analytics, Application Insights, Eventhouses, and Azure Data Explorer (ADX) native clusters, all connected to the same KQL Queryset. Directly query your Log Analytics workspace or Application Insights resources from KQL Queryset. This makes it easier than ever to explore and analyze data across services without extra configuration. Improvements to Data Exploration (low-code) experience We’re continuing to improve the low-code data exploration experience, making it even easier to analyze data in Real-Time Dashboard tiles and KQL Database tables (in Eventhouse and Real-Time Hub). Here’s what’s new: Hierarchal Columns Pane – easy to use summary of key data characteristics to guide exploration and data manipulation includes: List of participating columns Data types Statistical info (avg, min, max, cardinality etc.) Focus mode – work more efficiently by focusing on either the data visualization or results grid. Right-click actions on grid – quickly copy, export, and perform actions directly from the data grid. Filter builder with OR conditions –create WHERE statements with more flexible filtering options. Datetime picker control – select exact or relative dates effortlessly when filtering datetime columns. Learn more about data exploration experience here Explore data in Real-Time Dashboard tiles. Eventhouse system and KQL Database overview: in-item monitoring enhancements As part of our ongoing commitment to improving system visibility, performance monitoring, and user experience, we've introduced several enhancements across both the Eventhouse System Overview and KQL Database in-item monitoring pages. Eventhouse system overview enhancements New - Eventhouse ingested rows overtime One of the improvements to the Eventhouse System Overview is the ability to view ingested row metrics directly within the interface. Users can now see the number of rows ingested into each database, offering immediate visibility into data volume and ingestion activity. The feature includes time-based filtering, enabling users to analyze ingestion trends over specific periods. Feature_Summary New tab for Top ingested databases Another significant update is the introduction of a dedicated tab for Top Ingested Databases. This new section offers a detailed view of ingestion metrics for each database, including the total number of ingested rows and any ingestion failures (currently only partial failures are reported). The addition of time-based filtering makes it easier to identify patterns and anomalies across specific timeframes. Top 10 ingested databases- improvements The new multi-tab interface displays both the most queried databases and the top ingested databases, improving navigability and offering a clearer view of different performance indicators. Users can also benefit from added cache miss rates that were added to the queries database metrics. Eventhouse details moved to the menu bar To streamline access and maintain design consistency, Eventhouse details have been relocated to the main menu bar. Selecting this option opens a side panel that mirrors the familiar layout used in the Database and Table overview panes. To learn more, refer to the Manage and monitor an Eventhouse documentation. Eventstream's Real-time Weather Connector Last month, we introduced several powerful new connectors for Eventstream in Fabric Real-Time Intelligence. Now, we’re taking it a step further with a hands-on video that shows you how to use one of them: the Real-time Weather connector. In the demo video, we walk through how to easily add the Weather connector to an Eventstream and start streaming live weather data—like temperature, humidity, and wind speed—into Fabric. Whether you’re building real-time dashboards, alert systems, or enriching other streams, weather data adds valuable real-world context to your applications. What you’ll see in the video: A quick overview of the new Eventstream connectors and where to find them Step-by-step demo on adding and configuring the Weather connector Live preview of streaming weather data in action This video is a great starting point if you’re exploring how live data sources can enrich your streaming solutions in Fabric. The Weather connector is especially useful for industries like logistics, agriculture, and retail—anywhere environmental conditions influence operations. Watch the demo to see how easy it is to add real-time weather feed into an Eventstream. Can’t find your data sources, let us know! Send us an email at [email protected] or fill out our survey. h2> Databases SQL database in Fabric We have several new advances to share in SQL Database within Fabric. Continuous innovation is at the heart of our development, outlined are several key enhancements. New regions supported Australia Southeast Italy North Japan East Poland Central WestUS3 are new regions that supports SQL databases in Fabric workloads Backup billing SQL database in Microsoft Fabric offers automatic backups from the moment of database creation, ensuring data protection and recovery. The system makes full backups every week, differential backups every 12 hours and transaction log backups every 10 minutes, providing point-in-time restore capability up to 7 days. While compute and data storage are already included in the Fabric capacity billing model, starting April 1, 2025, backup storage will also be billed. Customers will only be billed for backup storage that exceeds the allocated database size. To learn more, check out the Automatic backups in SQL database in Microsoft Fabric documentation. Performance dashboard The SQL in Fabric dashboard now shows the lead blocking query to allow developers to quickly identify SQL queries that are blocking and impacting other queries thereby disrupting their operational workloads. Link to Video. Terraform support, Rest API & CLI support This capability enables customers to automate, scale, integrate, and govern their SQL databases within Microsoft Fabric, using a declarative approach with Terraform. HashiCorp Terraform, an open-source tool that offers a secure, predictable, and consistent method for deploying and managing infrastructure across multiple cloud environments. This functionality extends the capabilities of Fabric through Infrastructure-as-Code (IaC). To learn more about fabric-terraform-quickstart refer to the documentation. Be sure to check out the blog post Terraform support for Fabric GA for a more information. Integrations Fabric data pipelines will now support Fabric SQL database as a data source for Stored Procedure and Script activities, allowing users to just pick a database rather than having to enter connection information. Graph database support The query editor in SQL database in Fabric now have T-SQL support for graph databases. This feature enables the modeling of many-to-many relationships. You can create a graph database with nodes and edges and utilize the new MATCH clause to identify patterns and navigate through the graph. Learn more about how to Create a graph database and run some pattern matching queries using T-SQL from our documentation. Data Factory Mirroring Mirroring for Snowflake protected by a firewall (Preview) You now can mirror Snowflake protected by a firewall. Using either the VNet data gateway or the on-premises data gateway for mirroring is available. The data gateway facilitates secure connections to your source databases through a private endpoint or from a specific private network. Learn more about Mirroring for Snowflake from the Microsoft Fabric Mirrored Databases from Snowflake documentation. Closing We hope that you enjoy the update! Be sure to join the conversation in the Fabric Community and check out the Fabric documentation to get deeper into the technical details. As always, keep voting on Ideas to help us determine what to build next. We are looking forward to hearing from you!Fabric May 2025 Feature Summary
Today kicks off Microsoft Build and we have a lot of new features in store for you. Some highlights are the Fabric Roadmap tool, a way to get glimpse of that is coming soon to Fabric. Chat with your data, powerful AI capabilities that make Power BI even easier. Cosmos DB in Fabric, give you the power of Cosmos DB that's AI-ready. To get a taste of the Build excitement be sure to check out Arun Ulag's Arun Ulag's announcement blog and Kim Manis' announcement blog. Contents Events & Announcements New Fabric Roadmap tool Fabric Platform Additional REST APIs for Fabric Deployment pipelines New capabilities for Fabric Git integration Shortcut transformations (Preview) Data Engineering New regions supported in User Data Functions SPN support for User data functions SPN support for the Livy API Private libraries support for User data functions Data Science Copilot in Power BI now supports Fabric data agents Fabric Data Agent Integration with Microsoft Copilot Studio (Preview) Data Warehouse Warehouse Snapshots (Preview) Real-Time Intelligence Call of the Cyber Duty: a new season of Kusto Detective Agency begins Continuous Ingestion from Azure Storage to Eventhouse (Preview) Fabric Eventhouse now supports Eventstream Derived Streams in Direct Ingestion mode (Preview) Get Data in Fabric Eventhouse from Lakehouse using OneLake Catalog Eventhouse Accelerated OneLake Table Shortcuts (Generally Available) Databases Introducing Cosmos DB in Microsoft Fabric (Preview) Native change data capture (CDC) support in Copy Job (Preview) Semantic Model Refresh Activity (Generally Available) Copilot for Data pipeline - boost your productivity in understanding and updating pipeline with Copilot Mirroring Mirroring for SQL Server On-Premises (Preview) Mirroring for SQL Server 2025 (Preview) New features for Mirroring for Azure SQL Managed Instance Customize retention period for mirrored data Mirroring region expansion Mirroring for Azure PostgreSQL region expansion Fabric Mirroring for Azure Cosmos DB: public preview refresh live with new features Dataflow Gen2 Dataflow Gen2 (CI/CD) (Generally Available) Dataflow Gen2 Public APIs (Preview) Dataflow Gen2 parameterization (Preview) Lakehouse as an incremental refresh destination in Dataflow Gen2 (Preview) SharePoint files as a destination in Dataflow Gen2 (Preview) Natural language to custom column Community Power Designer - unleash your inner report wizard (Generally Available) Closing https://youtu.be/5qbIn80JrqY?si=BxhW9e_0Ck7QN-1M Events & Announcements New Fabric Roadmap tool We’ve heard from you that it’s critical to know when key Fabric features will land, especially those that directly impact your use cases or unblock your organization’s adoption. For example, if you're waiting on Private Link support for Workspaces due to internal security requirements, you need a clear view of when that capability is planned and when it becomes available. Until now, this information was spread across Release Plan documentation pages. Today, we’re making that experience better. The new Fabric Roadmap page brings it all together in one place, with a cleaner interface, real-time updates, and direct integration with the internal planning tool used by the Fabric team. Check it out at https://roadmap.fabric.microsoft.com and tell us what you think in the comments. Power BI Some of the highlights include Chat with your data, a revolutionary new way to use AI in PowerBI. And Translytical task flows, enabling users to automate action directly within the report—streamlining decision-making and operational follow-through. To learn about all of the latest updates to Power Bi head over to the Power BI May 2025 Feature Summary Fabric Platform Additional REST APIs for Fabric Deployment pipelines An additional batch of Fabric public APIs for Deployment pipelines have been released, following our initial release of Deploy APIs a few months ago. With this new release, the full list of available Fabric APIs now matches the APIs available in Power BI, excluding Admin APIs, which will be added later. This marks a significant milestone in our ongoing efforts to enhance the Fabric platform and provide our users with powerful tools to manage their deployment processes more efficiently. Overview of the new APIs The new APIs offer a range of functionalities that streamline the deployment process, making it easier for teams to manage their content across different environments: Pipeline management: Create, update, and delete deployment pipelines with the new APIs. Stage management: Get and update the deployment pipeline stages. Deployments management: List deployment pipeline operations and get details of specific deployment pipeline operations. Workspace assignment management: Assign and unassign workspaces to and from stages. Roles assignment management: List deployment pipeline role assignments and get or delete specific role assignments. Support for Service Principal (SPN) All fabric Deployment pipelines REST APIs are now having the support for Service Principal (SP). This allows for more secure and automated deployments, enabling teams to integrate Fabric into their existing DevOps workflows seamlessly. Getting started To start using the new Fabric public APIs for Deployment pipelines, you can refer to the Automate your deployment pipeline with Fabric APIs documentation. This provides comprehensive guides and examples to help you integrate these APIs into your deployment processes effectively. New capabilities for Fabric Git integration Service Principal (SPN) support for Azure DevOps A few weeks ago, we announced the capability to use Service Principal when working with Fabric Git API and your Git Provider is GitHub. Soon, we will support Azure DevOps as your Git Provider as well. Cross-Tenant support for Azure DevOps Previously, connecting your workspaces using your identity to an Azure DevOps repository required both Fabric and your Azure DevOps organization to reside within the same tenant. However, we're thrilled to announce that this limitation will soon be a thing of the past. With our upcoming update, you'll be able to connect to an Azure DevOps organization even if it belongs to a different tenant than your Fabric tenant. Shortcut transformations (Preview) The preview of shortcut transformations introduces the ability to transform data as it’s shortcut into Fabric including changing the data format into Delta tables or applying AI transformations to unstructured data—such as summarizing text, translating content, or classifying documents. Data Engineering New regions supported in User Data Functions After our preview launch, we have been working on increasing the number of regions supported for this feature. We have recently added the following 14 new regions where you can use this feature from: Australia Southeast Brazil South Canada Central Central India France Central Korea Central North Central US Norway East South Africa North South India UAE North UK West West Europe West US You can find the entire list of supported regions in this article: Fabric Region Availability. This article will be frequently updated to reflect the latest region support. SPN support for User data functions Fabric User data functions now support Service Principal Names (SPN) to run functions. This feature allows organizations to ensure compatibility with enterprise identity and access management systems. By using SPNs, it is possible to implement applications that can call a user data function without requiring user credentials. This aligns with the zero-trust security model, providing a secure way where user data functions are the glue between your application and your data in Fabric. To learn more, refer to the SPN support for user data functions documentation. SPN support for the Livy API The Fabric Livy API for Data Engineering now supports Service Principal Names (SPN) to submit and execute Spark code. This added SPN authentication method allows organizations to ensure compatibility with enterprise identity and access management systems. By using SPNs, it is possible to implement applications that can call a user data function without requiring user credentials. This aligns with the zero-trust security model, providing a secure way where user data functions are the glue between your application and your data in Fabric. To learn more, refer to the Create and run Spark Session jobs using the Livy API documentation. Private libraries support for User data functions A new feature has been introduced: Private libraries support for Fabric user data functions. These private libraries are code created by you or your organization. Data engineering can be challenging, especially with data quality and complex analytics. Private libraries help streamline work and enable proprietary code use within a team securely. Fabric User data functions now allow custom library uploads in .whl format, containing scripts or modules for internal business logic. This can improve developer productivity across your organization allowing you to reuse these libraries for automating various process across different teams or departments in your organization. To learn more refer to the documentation on How to manage libraries for your Fabric User Data Functions. Data Science Copilot in Power BI now supports Fabric data agents Fabric data agents can be used in the new chat with your data experience in Power BI to get answers to your questions and explore your data more effectively. This integration enables users in Copilot in Power BI to not only connect Power BI semantic models, but also to a wider range of data sources in OneLake—such as lakehouses, warehouses, and KQL databases—retrieving insights seamlessly through Fabric data agents. When you ask a question in the new full-screen Copilot in Power BI experience Copilot first searches for relevant Fabric data agents you have access to. If you have the necessary permissions, it uses those data agents to retrieve answers based on your access rights. This helps you discover content, ask questions, perform quick analyses, and refine insights—all without switching tools or leaving Copilot. You can also manually add a data agent to your Copilot session and chat with it directly from Copilot in Power BI, enabling seamless access to your OneLake data. Fabric Data Agent Integration with Microsoft Copilot Studio (Preview) Fabric data agent will be available in preview and can be added as an agent to your custom setup in Microsoft Copilot Studio. With this integration, your custom agent can access data stored in Microsoft OneLake—including lakehouses, warehouses, Power BI semantic models, and KQL databases—and retrieve insights seamlessly through the Fabric data agent. Once you add the Fabric data agent to your custom agent, you can publish your custom agent to various consumption channels, including Microsoft Teams and Microsoft 365 Copilot, and share it with specific users or your entire organization. When a user asks a question from the custom agent in any of these channels, the Fabric data agent is used to retrieve answers—provided the user has the necessary permissions. Responses are always scoped to the user’s access rights, making it easier to discover relevant content, perform quick analyses, and refine insights within the same channel. To extend functionality, you can define actions for your custom agent. Actions such as sending emails or initiating other tasks allow the agent to automate processes on behalf of users, helping streamline workflows and improve productivity without leaving the custom agent experience. Data Warehouse Warehouse Snapshots (Preview) Ensuring data consistency during ETL (Extract, Transform, Load) processes has long been a challenge for data engineers. We are pleased to announce the preview of Warehouse Snapshots, a new feature in Microsoft Fabric designed to offer a stable, read-only view of your data warehouse at a specific point in time. This capability facilitates uninterrupted analytics and reporting. A warehouse snapshot is a read-only representation of a data warehouse at a designated moment, retained for up to 30 days (until configurable retention is available). Warehouse snapshots can be seamlessly ‘rolled forward’ on demand, enabling consumers to connect to the same snapshot (or use a consistent warehouse connection string from third-party tools) to access a curated version of data. This ensures that data engineers can provide analytical users with a consistent dataset, even as real-time updates occur. Analysts can run SELECT queries based on the snapshot without any ETL interference. For more information on CRUD for warehouse snapshots and understanding their considerations and limitations, please refer to Warehouse Snapshot in Microsoft Fabric (Preview). https://youtu.be/cUGGrdpswLk?si=03UTJCbjvTD8WwjU Real-Time Intelligence Call of the Cyber Duty: a new season of Kusto Detective Agency begins Are you ready to put your sleuthing skills to the test? The Kusto Detective Agency is back - and this time, it’s bigger, bolder, and packed with adrenaline. Introducing ‘Call of the Cyber Duty’, a brand-new season of the Kusto Detective Agency challenge designed for the sharpest minds in data. Whether you're a seasoned Kusto veteran or a curious newcomer, this is your chance to dive into a thrilling online race where speed, smarts, and strategy collide. Challenge begins June 8, 2025 Register by June 7, 2025 Why should you care? Because this isn’t just a challenge - it’s a competition. And the stakes, monumental. $10,000 for 1st place Bragging rights across the Fabric community Team up or go solo - form a squad of up to six detectives or take on the mission alone. Who should join? If you're using Microsoft Fabric Real-Time Intelligence and working with Eventhouse, this is your moment. The challenge is built to stretch your KQL muscles, sharpen your investigative instincts, and connect you with a vibrant community of data detectives. How to get started: Watch the trailer: Kusto Detective Agency - Call of the Cyber Duty Visit https://detective.kusto.io to register Rally your team or fly solo Prepare for a season of puzzles, plots, and powerful insights. This is more than a game. It’s a celebration of what’s possible with Kusto and Microsoft Fabric. So, gear up, detectives—the cyber world needs you. Disclaimer: No Purchase Necessary. Must be 14+ to participate. Registration period closes on June 7th, 2025, end of day. Prizes are awarded as digital gift cards to the team leader. Continuous Ingestion from Azure Storage to Eventhouse (Preview) Get Data in Real-Time Intelligence Eventhouse offers a step-by-step process to guide you through importing or inspecting the incoming data, creating or editing the destination table schema, to exploration of the ingested result from multiple sources. One of the sources from which users can bring data into an Eventhouse table using Get Data wizard is Azure Storage, which allows users to ingest one or more blobs/files from the storage account. This capability is now being enhanced with the feature of continuous ingestion, where once the connection between the Azure Storage Account and Eventhouse has been established, any new blob/file uploaded to the storage account will automatically be ingested to the destination table. Continuous Ingestion from Azure Storage to Eventhouse, utilizes Azure Events in Fabric to listen to Azure Storage Account Events. Based on the subscribed events from Azure Events, Eventhouse pulls the corresponding newly created/renamed file from the connected Azure Storage. This simplifies the process of bringing data from your Azure Storage account as it is being generated and eliminates the need for creating and maintaining long complicated ETL pipelines. It also eliminates the need of defining time-based triggers for fetching new data from Azure storage and makes ingestion to Eventhouse near real-time. Continuous ingestion from Azure Storage to Eventhouse is now offered in preview in Microsoft Fabric. To learn more, refer to the Get data from Azure storage documentation. https://youtu.be/RlZnGloBvSA?si=IU5YbxGG8Fombygu Fabric Eventhouse now supports Eventstream Derived Streams in Direct Ingestion mode (Preview) The Eventstreams feature in Microsoft Fabric Real-Time Intelligence allows you to bring real-time events into Fabric, transform them, and then route them to various destinations such as Eventhouse, without writing any code (no-code). You can ingest data from an Eventstream to Eventhouse seamlessly either from Eventstream artifact or using Eventhouse Get Data Wizard. This capability is now being extended to support Eventstream Derived streams in direct ingestion mode. Derived stream is a specialized type of destination that you can create after adding stream operations, such as Filter or Manage Fields, to an Eventstream. The derived stream represents the transformed default stream following stream processing. You can route the derived stream to multiple destinations in Fabric and view the derived stream in the Real-Time hub. Direct ingestion from Derived stream allows you to ingest your event data directly into the Eventhouse without any processing. This can be configured from Eventstream, as well as from Eventhouse Get Data Wizard, including embedded Real-Time Hub in Eventhouse Get Data Wizard. Please refer to Get data from Eventstream to learn more and get started today. Get Data in Fabric Eventhouse from Lakehouse using OneLake Catalog OneLake catalog is the central hub for discovering and managing Fabric content. One of the artifacts that OneLake catalog enables the discovery of is Microsoft Fabric Lakehouse, which is a data architecture platform for storing, managing, and analyzing structured and unstructured data in a single location. Get Data in Eventhouse now embeds OneLake catalog which allows an easy discovery and navigation experience for ingesting data from Lakehouse to Eventhouse. Using OneLake catalog, you can easily look for a Lakehouse through multiple workspaces and identify the Lakehouse you recently used, your favorites or endorsed by your organization. Once you select Lakehouse from the embedded OneLake catalog in Eventhouse Get Data, you can select and ingest a file from the Lakehouse seamlessly, including the files within sub folders. To learn more, refer to the Get data from OneLake documentation and get started! Eventhouse Accelerated OneLake Table Shortcuts (Generally Available) Shortcuts are embedded references within OneLake that point to other files’ store locations without moving the original data. Previously, you could create a shortcut to OneLake delta tables using Eventhouse and query the data, but performance lagged direct ingestion in Eventhouse, as shortcut queries lacked the powerful indexing and caching capabilities of Eventhouse. Accelerated shortcuts are powered by query acceleration which indexes and caches data landing in OneLake on the fly, allowing customers to run performant queries on large volumes of data. Customers can use this capability to analyze real-time streams coming directly into Eventhouse and combine it with data landing in OneLake either coming from mirrored databases, Warehouses, Lakehouses or Spark. Customers can expect significant improvements by enabling this capability, in some cases up to 50x and beyond. How to enable Query Acceleration? You will now see an option to enable Acceleration while creating a new shortcut from Eventhouse. To learn more, refer to the Query acceleration for OneLake shortcuts - overview (preview) documentation. Databases Introducing Cosmos DB in Microsoft Fabric (Preview) Cosmos DB is now available in preview as a new addition to the databases workload in Microsoft Fabric. Cosmos DB in Fabric is easy to set up, with automatic scale and secure by default, enabling you to build AI applications with less overhead. You can store and retrieve semi-structured data within milliseconds, without having to tweak the database settings manually. Equipped with built-in vector indexing and AI-ready full-text, hybrid search capabilities of Cosmos DB, you can now seamlessly build GenAI applications. Your existing or new applications can instantly benefit from deep integration with Fabric OneLake, bringing you databases, analytics, data science, real-time intelligence, and Copilot-powered BI in one place, rather than assembling them individually. You can seamlessly join Cosmos DB data with any other data in OneLake, such as SQL DB, truly unifying your data estate. To get started, please join our preview program by filling in this opt-in form. For more information, refer to Announcing Cosmos DB (Preview). Data pipelines Native change data capture (CDC) support in Copy Job (Preview) Change Data Capture (CDC) in Copy Job is a powerful capability in Data Factory that enables efficient and automated replication of changed data including inserted, updated and deleted records from a source to a destination. This ensures your destination data stays up to date without manual effort, improving efficiency in data integration while reducing the load on your source system. With CDC in Copy Job, you can enjoy the following benefits: Zero Manual Intervention: Automatically captures incremental changes (inserts, updates, deletes) directly from the source. Automatic Replication: Keeps destination data continuously synchronized with source changes. Flexible Incremental Copy Options: Automatically detects CDC-enabled tables, allowing you to choose between CDC-based or watermark-based incremental copy at the table level. Optimized Performance: Processes only changed data, reducing processing time and minimizing load on the source. Learn more in the What is Copy job in Data Factory documentation. Semantic Model Refresh Activity (Generally Available) Semantic Model Refresh activity for data pipelines is now generally available! With this activity, you will be able to create connections to your Power BI semantic model datasets and refresh them from your data pipeline! To learn more, refer to the Semantic model refresh activity in Data Factory for Microsoft Fabric documentation. Copilot for Data pipeline - boost your productivity in understanding and updating pipeline with Copilot Maintaining complex Data pipelines in your ETL project is not easy work, especially when you need to understand complicated Data pipelines created by others or you need to update some configurations for a set of pipelines. Copilot for Data pipeline helps users quickly understand the purpose of a pipeline and the details of its activities. With this new release, it also allows users to update descriptions for pipelines and activities based on its summary. After updating, users can hover over any activity to see a simple explanation of its function. Copilot for Data pipeline empowers users to efficiently update the settings of multiple activities within seconds which is much faster than manual update that may take hours. For example, users can update the timeout of more than ten activities inside pipeline from 12 hours to 1 hour. To learn more, refer to the AI-powered development with Data pipeline documentation. Mirroring Mirroring for SQL Server On-Premises (Preview) Mirroring for SQL Server in Fabric for on-premises versions of SQL Server 2016-2022 is now in Preview! Mirroring in Fabric allows users to enjoy a highly integrated, end-to-end, and easy-to-use product that is designed to simplify your analytics needs. Built for openness and collaboration between Microsoft, and technology solutions that can read the open-source Delta Lake table format, Mirroring is a low-cost and low-latency turnkey solution that allows you to create a replica of your SQL Server data in OneLake which can be used for all your analytical needs. By leveraging Change Data Capture (CDC) technology available in SQL Server, mirroring service in Fabric uses on-premises data gateway (OPDG) to connect to SQL Server and read the initial snapshot as well as subsequent changes to data at the source. OPDG then pulls the data into OneLake and converts into an analytics-ready format in Fabric. To learn more, refer to the Mirroring for SQL Server in Microsoft Fabric (Preview) blog post. Mirroring for SQL Server 2025 (Preview) With the announcement of Microsoft SQL Server 2025 at Build, customers can also leverage mirroring from this version in Fabric. The overall user experience is like mirroring from other SQL Server versions and Azure SQL. Mirroring for SQL Server 2025 uses change feed instead of Change Data Capture and SQL Server keeps track and replicates the initial snapshot and changes to the landing zone in OneLake which is then converted to an analytics-ready format by the mirroring engine. To learn more, refer to the Mirroring for SQL Server in Microsoft Fabric (Preview) blog post. New features for Mirroring for Azure SQL Managed Instance We have made substantial updates to Mirroring for Azure SQL Managed Instance in Fabric. Based on user feedback, new features have been developed to address data replication needs: Mirror Azure SQL Managed Instance via private endpoint: VNet data gateway or on-premises data gateway can be used as a way to connect to your Azure SQL Managed Instance database for mirroring, removing the necessity of opening public access. The data gateway ensures secure connections to the source databases via private endpoint. Mirror tables without Primary Keys: We’ve relaxed the limitation to let you mirror tables even if they don’t have a primary key sometimes referred to as heap tables, offering increased flexibility. Support for expanded Data Definition Language (DDL): In addition to Alter/Drop/Rename tables/column, now you can Truncate tables in the source databases while mirroring is active. To learn more, refer to the Mirrored Databases from Azure SQL Managed Instance documentation. Customize retention period for mirrored data Mirroring in Fabric continuously replicates your existing data estate from various databases into OneLake in Delta Lake table format. To keep the mirrored data efficiently stored and always ready for analytics, mirroring automatically runs vacuum to remove old files no longer referenced by a Delta log. We now offer you the flexibility to customize the retention setting according to your requirements. For instance, you may choose a shorter retention period to reduce mirroring storage consumption or extend the retention period to utilize Delta’s time travel capabilities for analytics. Currently, this value can be set via API. To learn more, refer to the retention for mirrored data documentation. Mirroring region expansion Mirroring now supports all regions that are available for workloads in Microsoft Fabric. We have recently added a new region support for West US 3 to meet the growing customer demand. For detailed information about the Fabric regions that support mirroring, please refer to the supported regions documentation. Mirroring for Azure PostgreSQL region expansion Alongside the region expansion for all Mirroring in Fabric, Mirroring for Azure PostgreSQL will also expand region support from the initial 4 regions: Canada Central, West Central US, East Asia, and North Europe to all regions supported by Mirroring in Microsoft Fabric to ensure that customers have the best performance when replicating data from Azure PostgreSQL flexible server. To learn more, refer to the Simplify Your Data Strategy: Mirroring for Azure Database for PostgreSQL in Microsoft Fabric for Effortless Analytics on Transactional Data. Fabric Mirroring for Azure Cosmos DB: public preview refresh live with new features We’re thrilled to announce the latest refresh of Fabric Mirroring for Azure Cosmos DB! This update introduces key enhancements like Microsoft Entra ID authentication, container selection, support for special characters in column names, and even vector search compatibility for AI workloads. With features like auto schema inference and full CRUD API support, this release makes it easier than ever to build secure, scalable, and real-time analytics pipelines with Cosmos DB data in OneLake. To learn more, refer to the Fabric Mirroring for Azure Cosmos DB with new features blog post. Dataflow Gen2 Dataflow Gen2 (CI/CD) (Generally Available) With this new set of features now generally available, you can seamlessly integrate your Dataflow with your existing CI/CD pipelines and version control of your workspace in Fabric. This integration allows for better collaboration, versioning, and automation of your deployment process across dev, test, and production environments. New Dataflow Gen2 item experience with the option to enable Git integration, deployment pipelines and Public API scenarios. Key benefits Automated deployments: streamline your deployment process by integrating Dataflow with your CI/CD processes in Fabric. Version control: use GIT to manage and version your Dataflow Gen2, ensuring you have a history of changes and can easily roll back if needed. Collaboration: enhance team collaboration by leveraging GIT’s branching and merging capabilities. Multitasking support: you can now have multiple Dataflows open at the same time as other Microsoft Fabric experiences. These new features will significantly improve your workflow and productivity when working with Dataflows Gen2 in Fabric. We look forward to hearing your feedback and suggestions as we continue to enhance this feature. To learn more, refer to the Dataflow Gen2 with CI/CD and Git integration support (Preview) documentation. Dataflow Gen2 Public APIs (Preview) Data Factory in Fabric now provides a robust set of APIs that enable users to automate and manage their dataflows efficiently. These APIs allow for seamless integration with various data sources and services, enabling users to create, update, and monitor their data workflows programmatically. The APIs support a wide range of operations -- including dataflows CRUD (Create, Read, Update, and Delete), scheduling, and monitoring -- making it easier for users to manage their data integration processes. The APIs for dataflows in Fabric Data Factory can be used in various scenarios: Automated deployment: Automate the deployment of dataflows across different environments (development, testing, production) using CI/CD practices. Monitoring and alerts: Set up automated monitoring and alerting systems to track the status of dataflows and receive notifications in case of failures or performance issues. Data integration: Integrate data from multiple sources, such as databases, data lakes, and cloud services, into a unified dataflow for processing and analysis. Error handling: Implement custom error handling and retry mechanisms to ensure dataflows run smoothly and recover from failures. Learn more about Dataflow APIs in the documentation. Dataflow Gen2 parameterization (Preview) Parameters in Dataflow Gen2 enhance flexibility by allowing dynamic adjustments without altering the dataflow itself. They simplify organization, reduce redundancy, and centralize control, making workflows more efficient and adaptable to varying inputs and scenarios. Leveraging query parameters while authoring Dataflows Gen2 has been possible for a long time, however, it was not possible to override the parameter values when refreshing the dataflow. The ability to pass values from a pipeline into a Dataflow parameter for refresh has been one of the top ideas in the Fabric ideas portal since Dataflow Gen2 was released. We are happy to announce the preview of the public parameters capability for Dataflow Gen2 with CI/CD support as well as the support for this new mode within the Dataflow refresh activity in Data pipelines. Public parameters in Dataflow Gen2 with CI/CD support allow users to refresh their Dataflows by passing parameter values outside of the Power Query editor through the Fabric REST API or native Fabric experiences. This enables a dynamic experience with Dataflows, where each refresh can be run with different parameters that affect how the Dataflow is refreshed. To learn more about this new feature, refer to the Use public parameters in Dataflow Gen2 (Preview) documentation. Lakehouse as an incremental refresh destination in Dataflow Gen2 (Preview) Incremental refresh for Lakehouse destinations in Dataflow Gen2 is now in preview! This feature introduces a powerful way to optimize performance and ETL pipeline for one of the most popular data destinations to date. With incremental refresh, users can ensure faster refresh cycles, improve system efficiency, and reduce resource consumption, making it an ideal solution for large-scale analytics and operational data scenarios. This functionality is particularly valuable for businesses leveraging Lakehouse centric solutions to consolidate structured and unstructured data into a unified data model. To use this capability, configure your Dataflow Gen2 with a Lakehouse destination and enable incremental refresh settings within your dataflow editor as usual. Make sure to check out our documentation here to learn more about the considerations when you are using Lakehouse as a destination. To learn more, refer to the Incremental refresh in Dataflow Gen2 documentation. SharePoint files as a destination in Dataflow Gen2 (Preview) SharePoint data destinations in Dataflows Gen2 is now in preview! This innovative feature empowers users to seamlessly write CSV files directly into their designated SharePoint sites, streamlining data integration and enhancing team collaboration within Office 365. Using this new capability, users can effortlessly configure their dataflow queries to output data into specific folders within SharePoint, facilitating smoother workflows and ensuring that your data remains accessible and actionable in your operational processes. We encourage you to explore the possibilities of this feature and provide valuable feedback to help us refine and expand its functionality. Stay tuned for more updates and improvements as we continue to evolve data destinations for Dataflows Gen2! To start using SharePoint data destinations in Dataflows Gen2, follow these simple steps: Create a Dataflow Gen2 Get some data from one of your data sources In the destination settings, choose SharePoint as your output location. Provide the URL of the specific SharePoint site where you want your CSV files to be saved. Make sure you select the correct authentication method. Execute the dataflow to generate and store your CSV files in the selected SharePoint destination. To learn more, refer to the Dataflow Gen2 data destinations and managed settings documentation. Natural language to custom column Copilot is now available within the Custom column dialog of Dataflow Gen2. You can leverage a new Copilot experience where you can have Copilot write a custom column formula based on a prompt that you provide. For example, for a table that has the fields **OrderID**, **Quantity**, **Category**, **Total** you can pass a prompt like the following: If the total order is more than 2000 and the category is B, then provide a discount of 10%. If the total is more than 200 and the category is A, then provide a discount of 25% but only if the quantity is more than 10 otherwise just provide a 10% discount. After submitting this prompt, Copilot will process it and modify the custom column formula for you and adding a name and a data type if necessary. Be sure to give this new Copilot experience inside Dataflow Gen2 a try and share your feedback with us. Community Power Designer - unleash your inner report wizard (Generally Available) PowerBI.tips, in collaboration with Microsoft Fabric, is thrilled to announce that Power Designer is now Generally Available! This is a real time-saving application you won’t want to miss! Transition from generic, standard reports to sophisticated and highly customized presentations. What is Power Designer about? Power Designer is sleek, intuitive, and fun, making designing reports feel less like work and more like unleashing your inner artist. Craft themes like a pro: create detailed theme files for your Power BI reports with ease. Customize colors, fonts, and styles to match your brand. Real-Time visuals: watch your Power BI visuals update live as you build your style. Multipage mastery: add background images to each page with a snap, transforming your reports into polished, magazine-worthy layouts. AI-Powered: let AI take the wheel with auto-placement of visuals in your multipage templates. Preview: test your new theme on reports already published in your workspaces with the preview feature. Now that Power Designer has been released, it’s time to jump in and start creating. Head to your Fabric Workspaces, fire up Power Designer, and let your imagination run wild. To learn more, refer to PowerBI.tips Designer Now in Fabric – Power Designer. Check out the YouTube Video: Introducing Power Designer: Unleash Your Inner Report Wizard! Closing We hope that you enjoy the update! Be sure to join the conversation in the Fabric Community and check out the Fabric documentation to get deeper into the technical details. As always, keep voting on Ideas to help us determine what to build next. We are looking forward to hearing from you!
Statsig Experimentation Analytics (Preview)
Accelerating Product Innovation with Statsig Analytics on Microsoft Fabric Experimentation Analytics from Statsig offers a powerful new capability that’s set to transform how product teams innovate, measure product performance and make data-driven decisions to accelerate product adoption and growth. Imagine being able to unify experimentation, feature rollout, and impact analysis in one seamlessly integrated experience. That’s exactly what this workload delivers, now available to you via the Microsoft Fabric Workload Hub. With Statsig’s Experimentation Analytics in Microsoft Fabric, you can analyze product and behavioral data stored in OneLake in a frictionless, performant, secure, and powerful way. There is no need to move data or rely on external tools to build complex ETL pipelines, and you still have access to a robust Experimentation Analytics tool directly within the Fabric ecosystem. Why this Matters Product teams often face a dilemma and question: Will this feature drive engagement or increase revenue? With Statsig’s Experimentation Analytics, you can now answer that with confidence. Design and analyze experiments directly on your Fabric data. Define custom metrics – user engagement, retention, funnel analysis and more. Connect exposure data without moving it outside Fabric. Run rigorous statistical tests with built-in support for analytical tools. Automatically refresh results as new data lands in One Lake. Discovery: How to access workload in Fabric Start in the Fabric Workload Hub, we discover both first-party workloads and third party (ISV) published workloads. This is where you will find Statsig Analytics - offering a unified solution for experimentation, feature rollout, and impact analytics. It connects natively to OneLake, keeping all data governed and secure inside Fabric. How to: Step-by-Step approach From Metrics to Meaningful Insights Begin by connecting to your Fabric Data warehouse in your Fabric workspace. If already set up, define the metrics source using Statsig’s Metrics Explorer. For example, you may choose to measure User Engagement, Total Revenue or Checkout CTR (Click-through-rate) for your product. Experiment Setup and Assignment With your Fabric Data Warehouse connections and metrics definitions setup you connect to assignment sources to pull in exposure data. This data represents the control and variant groups that were created during the experiment design, including their exposures to different experiments over time. Statsig’s powerful statistical engine offers advanced A/B/n testing capabilities that allow you to choose from a range of statistical methods, Frequentist testing, Confidence intervals, Sequential Testing, and CURE for variance control to name a few. Real-Time, Incremental Analysis With your Datasource, metrics source and assignment source setup you can run deep- analysis, drilldowns, funnel analysis hypothesis testing on the data. With your automatic and incremental data collection and experiment setup Statsig automatically collects and refreshes results daily. No manual updates are necessary. You can simply monitor the scorecard to track the winning variant and expedite decision making. From Insight to Action In Microsoft Fabric if you were tracking the performance of your different experiments in Power BI to manage feature rollout readiness of your product; with insights from Statsig’s experimentation analysis you will be empowered to make a data-driven decision with confidence. A KPI turns green on the Power BI dashboard signaling greenlighting the winning variant for a staged release. Ready to Experiment? Whether you're a product manager, data scientist, or analytics leader—Statsig Analytics in Fabric is your new superpower. Start experimenting today and turn insights into impact. Try it now in the Microsoft Fabric Workload Hub and see your data turn to insights. To learn more about how to purchase a license visit Statsig’s Product page on the Azure Marketplace.
