Microsoft Fabric July 2023 Update
Welcome to the July 2023 update. We have features in Core, Synapse, Data Factory, Data Activator, Community, and Power BI. Contents Core Help Pane Monitoring Hub improvements Admin Microsoft Public Preview Default Sharing On by Default Effective July 5 th , 2023 OneLake OneLake file explorer update with support for switching organizational accounts Item sharing Synapse Data Warehouse Automatic updates for statistics UPDATE FROM and DELETE FROM T-SQL Optimization in data movements during query execution Dataflows Gen2 from the Warehouse Zero Copy Table Clones Data warehouse sharing Dbt Support for Fabric Data Warehouse Data Engineering Load to Table Feature Updates Lakehouse sharing and permission management Notebook Resources Notebook: Show cell output after session reconnection Notebook: Sharing and permission management Notebook: New status bar Data Science 1 Microsoft Fabric and SynapseML Offer New Features for Large Language Models Real-time Analytics KQL Database Create a Continuous Data Connection from Event Hubs to KQL Database Query KQL Database from MATLAB Data Factory Dataflows Gen2 Power Query Editor Enhancements to output destination experience Improvements to existing experiences Connectors Google Analytics (Connector Update) Oracle Database (Connector Update) Azure Databricks, Databricks (Connector Update) Denodo (Connector Update) EQuIS (Connector Update) Snowflake (Connector Update) Anaplan (Connector Update) Gateway and Connections SSO option in cloud connections Flag in cloud connections to allow them to be with gateway Data pipelines Activities Teams Activity Productivity Artifact parameterization support Performance improvement for moving parquet files to Lakehouse tables “Save as” support now available from your workspace Support column mapping for Lakehouse destination in Copy activity Data Activator Sample data Adding multiple streams or properties Designing triggers and alerts Trigger lifecycle Testing actions Teams app Community Implementing a Lakehouse with Microsoft Fabric Courses Filter by product for User Groups & Events Data Activator Community now live Power BI Reporting Smoothed and Leader lines – Updates On-Object Interaction (Preview) – Updates Customize the Pane Switcher CTRL click to open multiple panes in the Pane Switcher Treemap sub-selections now supported: Noteworthy bug fixes Modeling Edit your data model in the Power BI Service– Updates (Preview) Relationship validation Service Dataset details page revamp Mobile Datasets are coming to Power BI Mobile apps! Developers Boost Your Custom Visuals Performance: Check out our Latest Article! Visualizations New visuals in AppSource Elevate Your Map Charts With Drill Down Map PRO We are ready to present our new visual: an "all-in-one" Multi Target KPI card Other WebView2 GA Core Help Pane Help pane is feature-aware and displays articles about the actions and features available on the current Fabric screen. It is also a search pane that quickly finds answers to questions in the Fabric documentation and community. Open the Help pane From the upper-right corner of the Fabric screen, select the ? Icon. Help pane is feature-aware The feature-aware state is the default view of the help pane when you open it without entering any search terms. It shows a list of recommended topics, resources that are relevant to your current context and location in Fabric, and a list of links for other resources. It has three sections: Feature-aware documents: This section groups the documents by the features that are available on the current screen. As you explore Fabric, the feature-aware documents update based on what you've selected and where you are in Fabric. This is a great way to learn how to use Fabric. Give yourself a guided tour by making selections in Fabric and reading the feature-aware documents. Forum topics: This section shows topics from the Community forums that are related to the features on the current screen. Select a topic to open it in a new separate browser tab. Other resources: This section has links for feedback and Support. Help pane is a search engine Enter a keyword to find relevant information and resources from Microsoft articles and Community forum topics. Use the dropdown to filter the results. Monitoring Hub improvements With the new “column options” feature in Monitoring Hub, you can now customize the columns in the job list to check the information you are interested in. The columns selections would be memorized automatically. Feel free to go to other places, we will keep the columns and the filters for you once you are back at the monitoring hub. Admin Microsoft Public Preview Default Sharing On by Default Effective July 5 th , 2023 On July 5th, 2023, Microsoft began a staged rollout of an update to the Fabric Public preview setting. After the update, Fabric Public preview will be ON by Default unless customers explicitly opt out. With Fabric Public preview set to ON, users in your organization will be able to create Fabric items in workspaces attached to a Power BI Premium or Fabric capacity. When the Microsoft Fabric Preview was launched on May 23, 2023, Fabric was disabled by default, using a new tenant setting named "Users can create Fabric items (public preview)". Starting July 5th, 2023, Microsoft started activating Fabric preview by default for tenants who have not explicitly opted out. Any capacity level overrides of this setting will remain unaffected. If you are comfortable with your organization using the Fabric preview features, no further action is required. However, if you want to restrict usage of the Fabric preview in your P-SKU and F-SKU capacities, you have two options: Use the security group configuration of this setting at the tenant and capacity levels to limit Fabric to a smaller set of users. Turn off the setting at both the tenant and capacity levels to prevent all users from accessing the Fabric preview features. If you make these updates before the automatic update rolls out your tenant, this tenant setting update will not impact your organization. You may also make these changes at any time after the update. For more information about controlling Fabric use, visit Microsoft Learn. OneLake OneLake file explorer update with support for switching organizational accounts With the latest OneLake file explorer v1.0.9.0, it’s simple to choose and switch between different Microsoft Azure Active Directory (AAD) accounts. This was a highly requested feature from users who work with multiple AAD tenants, across organizations or even within a single organization. To switch accounts, right click the OneLake icon in the Windows notification area, select “Account” and then “Sign Out”. Signing out will exit OneLake file explorer and pause the sync. To sign in with another account, start OneLake file explorer again by searching for "OneLake" using Windows search (Windows + S) and select the OneLake application. Item sharing Starting July 2023, you can share Fabric items like data warehouses, lakehouses, Spark job definitions, Kusto databases, and KQL query-sets with users or groups. This enables collaboration with users that are not in workspace roles. Admins, Members, or users that have been granted reshare permission on a specific item, can share it with additional users outside of the workspace. You can share an item by clicking on Share in the item list or within the item. When an item is shared, the recipients can discover the item in Data Hub and may also receive a link to the item via email (if the option is selected while sharing). While sharing, a user can choose the level of access that the recipient can have. For example, when sharing a lakehouse, you can choose to grant the recipient with Read All SQL endpoint data permission, in addition to the read permission. This enables the recipient to read the default dataset associated with the lakehouse and access lakehouse data through the SQL endpoint. Here’s an example of sharing a lakehouse: Depending on the item being shared, you may find a different set of permissions that you can grant to recipients when you share. Read permission is always granted during sharing, so the recipient can always discover the shared item in Data Hub and open it. Here’s an example of sharing a warehouse: Sharing a Kusto database You can also grant or revoke permissions on specific items to users by selecting manage permission from the context menu. For example, you can grant the reshare permission on a Lakehouse to a user in the Workspace Contributor role. This would allow the contributor to share the lakehouse with users outside of the workspace. Note that you cannot modify the permissions inherited from a workspace role. Select manage permission from the context menu 2. Select Add user on Direct access tab and enter the names of users or groups that you want to provide access to. 3. Select grant To learn more about sharing Fabric items, read: Share items in Fabric. For more information on sharing of specific items, please read Share your Warehouse and manage permissions and How lakehouse sharing works? Synapse Data Warehouse Automatic updates for statistics SQL statistics are now automatically updated by the query engine! In the Fabric Data Warehouse and Lakehouse SQL Endpoint, statistics are a critical tool for helping your query run quickly and efficiently. When a query is executed, the engine will try to collect existing statistics for certain columns in the query and use that information to assist in choosing an optimal execution plan. Today, column statistics are automatically generated when the query engine requires statistics on columns that don’t yet have any. When data in your table’s columns changes significantly, it’s important that those statistics objects also update to accurately reflect the new data. Previously, this meant users would need to manually update on a regular basis. Now with automatic statistic updates, any statistics required during a user query are automatically assessed and refreshed if determined as outdated, allowing your query to use the most precise plan for execution and ensuring your workload is positioned for the best performance possible – all with zero user intervention. For more information on automatic statistic updates, Statistics - Microsoft Fabric | Microsoft Learn UPDATE FROM and DELETE FROM T-SQL You can now update and delete data in your target table from an existing source table using the FROM argument in your UPDATE and DELETE scripts in Fabric SQL respectively! These T-SQL commands allow you to perform MERGE T-SQL like operations. For more information, see UPDATE (Transact-SQL) - SQL Server | Microsoft Learn and DELETE (Transact-SQL) - SQL Server | Microsoft Learn Optimization in data movements during query execution The Fabric SQL engine creates a query plan that comprises of execution steps and data move operations. Data move operations ensure that the data required for a query step is ready in the location of its execution. With this new optimization in Fabric SQL, the engine makes data movement in intermediate steps of query execution more balanced and improves overall query performance. No user intervention or code change is required to use this optimization; it works out of the box! Dataflows Gen2 from the Warehouse You can now create a new Dataflow Gen2 directly from your warehouse to ingest data. Simply click Get Data --> new Dataflow Gen2. The newly created Dataflow Gen2 artifact will already prepopulate the destination to the warehouse. For more information about creating, ingesting, and transforming data with Dataflows Gen2, see Create your first Microsoft Fabric dataflow - Microsoft Fabric | Microsoft Learn. Zero Copy Table Clones We are pleased to announce that zero copy Table Clones are now available in Public Preview! Zero copy clones are a near-instantaneous metadata only operation that enables you to easily create a copy of your Warehouse table(s) with no additional cost and minimal overhead. Table clones contain a reference to the source table that the clone was created from. Underlying parquet files are not duplicated when a clone is created – under the hood, a fork is created, and the clone behaves as an independent table which can be modified as needed. Any changes made to the source table after it was cloned are not reflected in the clone; similarly, any changes made to the cloned table are not reflected in the source. They are independent of one another. As of today, customers can create a Table Clone within the same schema or to a different schema within the same Warehouse. Customers looking to create a Zero Copy Table clone would typically do so for a variety of test, development and production use cases and experimentation. For example, a customer may want to stage a production release in a table clone prior to going live with the changes. Once they are happy with the changes, they can be merged to the production table. For more information on Table Clones, see Clone table in Microsoft Fabric Data warehouse sharing We are excited to announce that Data warehouse sharing is now available in Public Preview! Data sharing is essential to fostering a data-driven culture within an organization. Sharing a Warehouse allows you to easily provide read access to enable downstream users to consume this data, without making copies of data. With this new capability, an Admin or Member within a Fabric workspace can share a Warehouse with another recipient (AAD user or AAD groups) within your organization. The following are the permissions that are provided: [Default] Connect permissions to the warehouse – This option is provided by default and it provides permissions to connect to the warehouse (equivalent of Connect permissions in SQL) but not query any table or view. You can grant granular object access using GRANT in T-SQL. [Default] Build reports on the default dataset - This option is provided by default and provides “build” permissions on the default dataset that is connected to your Warehouse. This option can be useful for your Power BI developers who want to create reports on this default dataset. [Optional] Read all data using SQL – This option provides “readData” (equivalent of db_datareader) permissions which allows for read access to all tables and views within the Data warehouse. This option can be useful for users who want to read using SQL. [Optional] Read all data using Apache Spark – This option provides “readAll” permissions which allows read access to the Warehouse’s underlying files in One Lake that you can read through Spark. This option can be useful for your data scientists who want to read using Spark. For more information, see the [detailed sharing blog]. Dbt Support for Fabric Data Warehouse We are excited to announce the preview of dbt plugin adapter for Synapse Data Warehouse in Microsoft Fabric (preview). This data platform-specific adapter plugin allows you to connect and transform data in Synapse Data Warehouse in Microsoft Fabric. For more information, see the Introducing the dbt adapter for Synapse Data Warehouse in Microsoft Fabric, Microsoft Fabric Synapse Data Warehouse dbt adapter setup and Microsoft Fabric Synapse Data Warehouse dbt adapter configuration. Data Engineering Load to Table Feature Updates Previously the "Load to Table" feature allowed users to load a single file to a new table, which was very well met with data engineers due to the added productivity value of quickly using simple right click actions to enable table loading on Files and as well as because of the no-code experience, which lowers the entry bar for all personas. This new release brings improvements to this experience with several new functionalities: Folder-level load: Users can now load all files under a folder and its subfolders at once by selecting "Load to Delta Table" after clicking on a folder. This feature automatically traverses all files and loads them to a Delta Table. Load to existing table: User can now choose to load their files and folders to a new or an existing table of their choice. If they decide to choose to load to an existing table, they have the option to either append or overwrite their data in the table. Source file option: User can specify if their source file includes the column names as the first row of data and the separator used in it. For more detailed information on this feature, visit the documentation here. Lakehouse sharing and permission management We are announcing a capability for users with Admin and Member roles to share an individual Lakehouse with users without providing them access to the workspace. That will grant access to a specific lakehouse without exposing other items in the workspace. The users will get access to shared items through Data Hub or the link included in the sharing notification email. With access to a lakehouse, users can access the SQL endpoint and default dataset. That provides features like querying data using T-SQL and building Power BI reports on top of the lakehouse data. Permission management also allows users in Viewer role to get additional permissions to access Lakehouse data using Spark. These features will push data democratization in Fabric even further, enabling more collaborative work. Notebook Resources We are excited to announce the "Notebook resources" on Fabric notebook. This feature offers notebook users a writeable file system space where you can store small-sized files, such as code modules, datasets, and images. You can access them with code in the notebook as if you were working with your local file system. The Notebook Resource explorer provides a Unix-like file system to help you manage your folders and files. You can use common operations such as create/delete, upload/download, rename, duplicate, and search through the UI, and rich built-in snippets are provided through “Drag & Drop”. For more details please see How to use notebooks - Microsoft Fabric | Microsoft Learn. Notebook: Show cell output after session reconnection Fabric notebook now supports displaying the running cell output after reconnecting to the original session. This feature allows you to easily recover your ongoing work after accidentally closing the browser or leaving the live session. You don't need to take any additional action to enable it, as this feature is available by default on the Notebook. Notebook: Sharing and permission management Starting now, sharing a single notebook with your colleagues is easier than ever before, without having to grant workspace permissions. With the Notebook sharing feature, you can collaborate with team members and share your work conveniently. Additionally, we now support managing permissions for each Notebook instance. You can easily check and update the permissions of notebooks after they have been shared, ensuring proper access to the notebook. Notebook: New status bar Notebook recently upgraded the design of status bar. In this upgrade, we have styling refinement as well as functional enhancements. Now you can easily discover the “Save options” status, you can easily navigate to the failed cell, and you can find more useful information in the floating info card - the diagnostic information is especially helpful when you encounter issue and need service support! In the next release we’ll add another series of quick access entries on the status bar, so stay tuned for the upcoming new features! Data Science Microsoft Fabric and SynapseML Offer New Features for Large Language Models Microsoft Fabric and SynapseML has featured a variety of new features that have simplified the use of large language models. In our latest release, we have integrated the Azure OpenAI service with the distributed machine learning library SynapseML, which makes it easy to use the Spark distributed computing framework to process millions of prompts with the OpenAI service on Microsoft Fabric. The new OpenAI APIs that have been introduced into SynapseML include “OpenAICompletion”, “OpenAIEmbeddings”, “OpenAIChatCompletion”, and “OpenAIPrompt”. The following helps to better understand these new OpenAI APIs. Leveraging the new OpenAI APIs in SynapseML and Microsoft Fabric, we have also demonstrated how to perform Q&A on PDF Documents. You can read more about it here. Please note that native access to the Azure OpenAI service will be coming to Microsoft Fabric later this year. Real-time Analytics KQL Database Create a Continuous Data Connection from Event Hubs to KQL Database Azure Event Hubs is a big data streaming platform and event ingestion service that can process and direct millions of events per second. Now you can easily stream your Azure Events Hubs data directly into your Fabric KQL Database. There are 2 main steps required to stream the Event Hubs data to the KQL Database: Create a Microsoft Fabric platform-based data connection to a specific event hub instance. This data connection can be used across all Microsoft Fabric workspaces and is managed centrally. Connect this Microsoft Fabric-based data connection to a KQL database. This process creates a database-specific Event Hubs data connection. The connection streams data into the table you specified during setup, and the data will then be available to query using a KQL queryset. A pre-requisite for creating a cloud connection in Microsoft Fabric is to set a share access policy (SAS) on the event hub and collect information to be used later in the setting up the cloud connection. This step is performed in the Azure portal. Go to your event hubs instance and under Settings select the Shared access policies. Add a new SAS Policy or select an existing one: To Create the Cloud Connection from Azure Event Hubs and Fabric, go to the menu bar of your Fabric workspace and go to Manage Connection and gateways. In the New Connection form, you will need to enter details taken from the defined SAS policy. The final step is to connect a table in your KQL database to the event hub cloud connection defined above. This can be done by selecting the Event Hubs option in the Get Data menu bar: For more details, go to the docs: Get data from Azure Event Hubs. Query KQL Database from MATLAB In manufacturing and energy verticals, MATLAB is still heavily used. MATLAB is a programming and numeric computing platform used to analyze data, develop algorithms, and create models. We are happy to announce that Fabric now supports the querying of KQL database data directly from MATLAB. Now engineers that are familiar with MATLAB do not need to learn KQL to query the high performant RTA KQL database, but rather reuse their MATLAB skills to query the data. This functionality is supported by a very lightweight MATLAB Connector that securely connects MATLAB to a Fabric KQL Database. For more details, go to the docs page: Query data using MATLAB. Data Factory Dataflows Gen2 Power Query Editor Enhancements to output destination experience Auto-fix column names during mapping of columns dialog Auto-fix data types during mapping of columns dialog Improvements to existing experiences Rename a dataflow inside of the Power Query Editor A similar experience to other artifacts inside of Microsoft Fabric, you can now change the name of a Dataflow Gen2 inside of the Power Query Editor. Connectors Google Analytics (Connector Update) The Google Analytics connector has been updated to support Google Analytics Data API (Google Analytics 4). To use this new functionality, use “Implementation 2.0” when connecting. Existing connections will not be affected. Oracle Database (Connector Update) The Oracle connector has been updated to enable Azure AD-based Single Sign-On functionality through the on-premises data gateway. This will require the July release of the on-premises data gateway. Azure Databricks, Databricks (Connector Update) The Azure Databricks and Databricks connectors have been updated. Please find notes from the Databricks team below. Add a new DSRHandler to databricks-multicloud Fix UC_NOT_ENABLED and Catalog 'spark' not found error in legacy code path using Databricks.Contents Denodo (Connector Update) The Denodo connector has been updated. Please find notes from the Denodo team below. This new version adds graphical support for the specification of native SQL queries at data source creation time EQuIS (Connector Update) The EQuIS connector has been updated. Please find notes from the EQuIS team below. Remove "Beta" attribute Retrieve report content as .csv to remove the row limitation of .xlsx files Optimize handling of facility groups in navigation tree Show report and/or location folders in navigation tree even if one or the other is empty Snowflake (Connector Update) The Snowflake connector has been updated to include various performance improvements, such as usage of SQLBindCol. Users should experience better performance when running queries. Anaplan (Connector Update) The Anaplan connector has been updated. Please find notes from the Anaplan team below. This version of Power BI connector for Anaplan includes backend changes for compatibility with ongoing Anaplan infrastructure updates. There is no change to user facing connector features. Gateway and Connections SSO option in cloud connections Added support for Single Sign-On (SSO) via Azure AD in cloud connections. Currently this feature is only applied for DirectQuery mode in datasets. However, we plan to progressively extend SSO capabilities to other Fabric workloads in the future. Flag in cloud connections to allow them to be with gateway We have enhanced our security measures to allow users to disable cloud connections to be used with gateway connections. This precaution prevents the decryption and logging of cloud connection credentials within on-premises systems. Data pipelines Activities Teams Activity We’re excited to announce that the Teams activity is now available to use in your Data Factory data pipelines. In your pipeline, you can use the Team activity to customize a message to send to a Teams channel or a Teams group chat. For example, you can use the Teams activity to send a notification if a pipeline has failed, helping you to better monitor your data integration pipelines. Productivity Artifact parameterization support We’re excited to share that you can now parameterize Fabric artifacts in your data pipeline. This will allow you to use expressions, functions, parameters, and variables to dynamically refer to your Lakehouse or Data Warehouse, allowing more flexibility when designing your data pipelines. Performance improvement for moving parquet files to Lakehouse tables We’ve made performance improvements when copying parquet files to your Lakehouse! “Save as” support now available from your workspace “Save as” is now supported by data pipelines from your Data Factory workspace. When you are in your workspace, you can use the selection menu to save a new copy of your pipeline. This will allow you to build upon or edit existing pipelines without having to completely rebuild a data pipeline. Support column mapping for Lakehouse destination in Copy activity Data pipelines now supports column mapping when a Lakehouse is selected as a data destination. In the Mapping tab, you can now add, edit, or delete column mappings from your data source to your data destination. Data Activator Sample data The simulator that generates sample data now includes multiple event streams that you can use to try out building objects and alerts. If you want to try the simulator, check out our tutorial: End-to-end tutorial using simulated data. Adding multiple streams or properties The Data screen now gives you options to create objects directly from columns in your data streams, and gives you shortcuts to quickly create properties directly rather than having to create them and manually select the column. We’ve also planned a new UX to let you create multiple properties or objects across multiple streams in one go – stay tuned for that in the coming months! Designing triggers and alerts We’ve combined the first two steps of triggers (Reference to a property and selecting the value from an event) into one step. This means the first thing you’ll see is a chart that plots the values you select helping you understand the data more quickly. It’s currently called “Property field reference”, which will be updated to be more user-friendly soon! We’ve added many more functions to the library you can use to build triggers: Detect functions. Changes, Changes from/to Sends an event each time the field changes from/to a value, or changes at all. Is less than/greater than, Is false/true, Is equal/not equal Sends an event whenever the field meets that condition. Becomes less than/greater than, becomes true/false, Exits/Enters range Sends an event the first time the field meets that condition (if subsequent events also meet that condition, no further events are emitted) Summarize functions. Maximum, minimum, average, count over time. Calculates a aggregation of the values of all events in a time window. Filter functions Filter Filters individual events where the value meets a specified condition. Other events are dropped. This is useful for filtering out error values such as a sensor that returns -99 for errors. Property filter Filters out instances that match the criteria. This is useful if you only want a trigger to fire for certain instances (this will be more usable when the trigger changes described below are done). ‘Detect’ functions such as ‘Crosses above’ or ‘Changes to’ also have options to only trigger when the criteria is met a certain number of times, for example 3 times in an hour. This can help reduce noise in your triggers. Coming soon, we have a major change to the way triggers are defined. Early preview feedback showed that the detailed step-by-step setup for triggers was too complicated, so we’ve geared them around selecting 3 things: what you what to monitor, what condition you want to detect, and what action to take. This should make it easier to configure your triggers and make it clearer what’s needed to get alerts on your data. Trigger lifecycle Based on feedback to make the management of triggers simpler, we removed the Draft concept. You don’t need to Publish a trigger before you can start it. Triggers are either just Started or Stopped! Any time you make changes, you select Update to change the running trigger to use the new values. Testing actions The ‘Test action’ button confused early users so we’ve updated the messaging to clarify that it will send a test to the current user. It’s also only enabled if there is some data that met the criteria, which is used as sample data in the test message. Teams app Data Activator can now send alerts to you from our new Teams app. To install it, search the Teams store! A Teams administrator can set it up for everyone in your organization if you need to deliver messages broadly, rather than each user installing it themselves. Community Implementing a Lakehouse with Microsoft Fabric Courses This course is designed to build your foundational skills in data engineering on Microsoft Fabric, focusing on the Lakehouse concept. This course will explore the powerful capabilities of Apache Spark for distributed data processing and the essential techniques for efficient data management, versioning, and reliability by working with Delta Lake tables. This course will also explore data ingestion and orchestration using Dataflows Gen2 and Data Factory pipelines. This course includes a combination of lectures and hands-on exercises that will prepare you to work with lakehouses in Microsoft Fabric. Check out the course: Course DP-601T00: Implementing a Lakehouse with Microsoft Fabric Filter by product for User Groups & Events You can now filter by product for both Fabric User Groups and Events in the Fabric Community site. With the broad range of experiences within Fabric, product filtering allows you to easily find User Groups or events that match your interests! You can also quickly refilter your results by removing products either in the Product drop down or with the tiles below the search bar. Data Activator Community now live Data Activator now has its own community forum, alongside the other product forums across Fabric: https://community.fabric.microsoft.com/t5/Data-Activator-preview-Community/ct-p/dataactivator (or use the short URL https://aka.ms/dataActivatorCommunity!). If you’ve got any questions or feedback about Data Activator you can post there and the product team will be in touch. Power BI Reporting Smoothed and Leader lines – Updates We are excited to announce the launch of one of our most highly acclaimed features! Report creators can now create smoother line and area charts, providing a more polished look to their visualizations. To access this setting, go to Lines > Shape > Line Type. We've recently added leader lines for both line and area charts. This new feature creates a visual connection between each data point and its corresponding label. To access this feature, simply navigate to the Data labels > Options > Leader lines. These features are just the beginning of the many improvements we have in store for graphs, charts, plots, and markers in the coming months. Get ready for even more exciting updates! On-Object Interaction (Preview) – Updates The new on-object interaction feature released to preview back in March. This month we bring more improvements and bug fixes. Customize the Pane Switcher We’ve now added a new “+” button on the pane switcher to quickly add new panes directly from the pane switcher without having to go to the View ribbon. This menu also gives a brief description of what panes are available and what their functions are. Even better, the panes added to the switcher are saved across reports. Configure once and you’re done! You can also access the 2 preference settings released last month for “always show the pane switcher” and re-attaching the build menu as a pane by using the gear icon. CTRL click to open multiple panes in the Pane Switcher In addition to the right click option “open in new pane”, it is now even easier to open multiple panes from the pane switcher by simply holding down the CTRL key and clicking the pane you wish to open. Treemap sub-selections now supported: Noteworthy bug fixes Overlap of the on-object buttons on the formula bar has finally been resolved! We appreciate your patience as this bug was a bit trickier to fix the right way. Visual tooltip showing automatically when opening the build menu, blocking the formatting on-object button is now fixed. Selected visual type is now reflected in the ribbon visual gallery accordingly. Mini-toolbar’s fill color icon now reflects conditional formatting gradient as well. Thanks for continuing to try out the new preview and provide feedback. We’re working hard to react to your suggestions and add the necessary changes to make on-object work for you. Please continue to provide your comments directly in this blog post or in our community forum via the “Share feedback” button next to the preview switch. Modeling Edit your data model in the Power BI Service– Updates (Preview) The new data model editing in the Service feature released to preview in April. We've been busy reacting to your feedback and enhancing the experience. Below are the improvements we are adding this month: Relationship validation We are adding relationship validation in the Service, making it easier to create and edit relationships in the web! Like Power BI Desktop, as you define the properties of your relationship, the system will automatically validate it and offer appropriate choices for cardinality and cross filter selections. Please continue to submit your feedback directly in the comments of this blog post or in our feedback forum. Service Dataset details page revamp We are happy to announce the revamp of our dataset details page! Now, when you click on a dataset item in the OneLake data hub and workspace view, you will be directed to the redesigned page that not only enhances the look and feel but also introduces new capabilities for an improved user experience. Here's what you can expect to find on the dataset details page: Actions: You will find various actions that can be performed on the dataset, such as creating a report and refreshing the dataset. With this release, we have added the option to view the refresh history under the refresh menu. Dataset Metadata: Gain insights into the dataset through its description and last refresh time. Related Items: Explore existing related items associated with the dataset. Dataset Schema: Get a comprehensive view of the dataset's tables and columns. Clicking on a table provides a table preview, with export capabilities available using paginated reports behind the scenes. Additionally, we have made significant improvements to the related items list. It now showcases all the downstream and upstream dependencies for the dataset. This enhancement allows you to easily identify the sources of the dataset, composite model relations, reports, and dashboards associated with it. We believe that these updates will greatly enhance your experience with the dataset details page, providing you with a more intuitive and comprehensive understanding of your data. We look forward to your feedback as you explore these new features!" Mobile Datasets are coming to Power BI Mobile apps! In the next Power BI Mobile app release, we are adding a long-waited feature that will help dataset owners and report creators to manage their dataset directly from their phone. That means that you will be able to see in your mobile device datasets. Go to a workspace, make sure to select the “dataset” pill at the top and get the list of datasets, that you have access to in that workspace. When tapping on a dataset, you will get the dataset metadata pane, which includes the name, owner, sensitivity label and also the latest refresh status. From this pane you can also trigger a dataset refresh – all directly from your mobile app! Dataset owners will also get push notifications when schedule refresh fails. They will be able to view the failure details and be able to re-try the refresh while they are on-the-go. Developers Boost Your Custom Visuals Performance: Check out our Latest Article! We have recently published an article that focuses on techniques to improve the performance of custom visuals. In this article, we discuss the performance improvements we have made in visual rendering and load times. We identified and addressed certain bottlenecks in the code, and these improvements are available for any visual that has been updated to API version 4.2 and onwards. Along with these fixes, we also provide code practices and techniques that can greatly enhance the performance of rendering custom visuals. I encourage you to check out the article here. We believe that these techniques can make a significant impact on the performance of your custom visuals. Visualizations New visuals in AppSource Spider Chart for Power BI by VisioChart HTML Content (lite) Stacked Lipstick Bar Chart (Standard) Stacked Lipstick Column Chart (Standard) 100% Stacked Column Chart with Values instead of % (Standard) Dual Axis Scatter Chart (Standard) Category Comparison Bar Chart Stacked Column with Percentage and Total in Label (Standard) Likert Scale Chart for Power BI by ChartExpo Elevate Your Map Charts With Drill Down Map PRO Drill Down Map PRO by ZoomCharts is a custom map visual for Power BI that lets you show your data on an interactive map and give it location-based context. Features include: Built-in shape layers – use preset shapes for easy filtering of countries. Custom shape layer support – provide custom shapes through KML and GeoJSON files. Lasso tool – draw and save your own filter shapes on top of the map. Node clustering capabilities – clusters can be turned into donut or pie charts for category display. Map base layer lets you choose from 4 options – Azure maps, Custom (OpenStreetMaps, Google, CartoDB etc.), Image (e.g., floor plans), None (visualize shapes without a background). Aura, image, and custom label support. Popular use cases: Production – monitoring production data by location. Sales and marketing – mapping sales results by region. Public sector – visualizing environmental and sociodemographic data. Get Drill Down Map PRO now! Learn More about Drill Down Map PRO. ZoomCharts Drill Down PRO Visuals are known for their interactive drilldowns, smooth animations, rich customization options. All Drill Down PRO Visuals support: touch input devices, interactions, custom and native tooltips, filtering, bookmarks, and context menu. We are ready to present our new visual: an "all-in-one" Multi Target KPI card The Multi Target KPI card works with a single query and includes three additional indications, multiple categories, a pixel perfect alignment setting, and built-in conditional formatting. You can change the settings of layout type and color conditional formatting for additional measures in our visual, and it is fairly simple for non-designers to use! Just select the desired measure and category, if necessary. Add up to three additional indicators to provide the context you need for your metric. It will help you improve reporting performance and save you time when designing and developing supplemental measures. Start a new level of business dashboarding! Link to AppSource: https://appsource.microsoft.com/en-us/product/power-bi-visuals/ndividualentrepreneuralekseikolokolov1654522457293.80416480?tab=Overview Link to our website: (delete if not needed) https://data-2-speak.com/cardkpi Other WebView2 GA WebView2 is now generally available. Thanks to everyone who reported issues during the preview phase! Your input helped us improve reliability to higher than it was before we introduced WebView2. Please continue to report any issues using the “There was a problem with WebView2” dialog.Microsoft Fabric August 2023 update
Welcome to the August 2023 update. We have lots of features this month including the new layout switcher for Power BI, SSD caching in Synapse Data Warehouse, in-line Python support for KQL in Synapse Real-time Analytics, lookup activity for Data Factory Dataflows, and much more. Continue reading for more details on our new features! Contents Core Multitasking navigation improvement Monitoring Hub support for personalized column options Power BI Reporting New layout switcher New bubble range scaling setting On-Object Interaction (Preview) – Updates Resizing and positioning to the on-object menus On-object format subselections now supported in spotlight and focus mode Modeling Updates to ORDERBY function Data Connectivity New modern data connectivity and discovery experience in Dataflows Lakehouses (Connector Update) Service XMLA Write support for Direct Lake datasets Automatic replica synchronization for Dataset Scale-Out (preview) Mobile Choose your start-up content Developers Published AppSource visuals now generate leads in Partner center and CRM Publish a Power BI Project (PBIP) directly from Power BI Desktop Visualizations New visuals in AppSource Editor's pick of the quarter Drill Down Combo Bar PRO by ZoomCharts Sunburst Chart by Powerviz xViz Performance Flow by Lumel Synapse Data Warehouse SSD Caching Data Engineering High Concurrency Mode for Notebooks Data Science Model & Experiment Usability Improvements New Data Science Sample are Coming to Microsoft Fabric Real-time Analytics KQL Database support for inline Python KQL Database provisioning in a few seconds Data Factory Dataflows Gen2 Power Query editor Edit connection in Manage Connections Enable Staging labels Publish dataflow without staged queries Limit to number of queries in a Dataflow when publishing Refresh history Direct navigation to last failed refresh Statistics for tables Connectors Lakehouse connector update Other improvements Data pipelines Connectors FTP Connector Productivity Lookup activity for Lakehouse, Data Warehouse, and KQL Database Get Metadata activity for Lakehouse and Data Warehouse Pipeline run status now available Secure input and output options for Set Variable activity Data Activator Trigger and property design template Quickly assign columns as properties in an object Data Activator now supports Power BI visuals with a time axis Trigger Power Automate flows from Data Activator Core Multitasking navigation improvement In our latest update, we've introduced an enhancement that preserves all Fabric items opened in a single browser tab on the left navigation bar, even in the event of a page refresh. This ensures you can refresh the page without the concern of losing context. Monitoring Hub support for personalized column options We have updated Monitoring Hub to allow users to personalize activity-specific columns. You now have the flexibility to display columns that are relevant to the activities you're focused on. Power BI Reporting New layout switcher We’ve added new buttons that make it easy for you to quickly switch between web and mobile layouts while you’re developing your reports. You'll find the new switcher buttons down at the bottom of the screen, right next to the page navigator. New bubble range scaling setting We are introducing the new bubble range scaling setting for scatter chart and map visuals! This setting gives report creators more control over how the bubble (marker) sizes respond to the data, making it more accurate or distinctive based on preference. With the magnitude option, the bubble areas closely follow the data proportions. With the data-range option, the bubble size limits are mapped to data minimum and maximum. The auto option, which is the default setting for new reports, selects the appropriate option based on data characteristics. For more information, visit our docs. This setting can be adjusted in formatting panel, Markers > Shape > Range scaling for scatter charts or Bubbles > Size > Range scaling for maps. For reports authored in earlier Power BI versions, these settings default to (Deprecated) for scatter charts (which differs in handling negative values), and Data range for map charts. Azure Maps charts will also include this feature in a coming product update. In the figure above, the size of each country represents Urban Population, which is also shown on the y-axis. On-Object Interaction (Preview) – Updates The new on-object interaction feature released to preview back in March. We’ve been busy adding even more improvements to the preview, here’s what’s part of the Aug release: Resizing and positioning to the on-object menus We’ve now added the ability to resize the on-object menus horizontally, this is especially helpful when you’re working with long field names. We’ve also improved the positioning of the on-object menus to make better use of the canvas space. Previously, when a visual was near the bottom of the canvas, the on-object menu was super small and required scrolling to be able to see and use the field wells. Now, the on-object menu moves up and stretches into the canvas to bring the field wells into view without needing to scroll. Before After On-object format subselections now supported in spotlight and focus mode When spotlighting a visual or expanding the visual in focus mode, you can now use on-object formatting to subselect and format styles. In focus mode, it can be hard to tell when you’ve entered format mode with just the subtle border. To address this, we’ve added a button to the header to better indicate when you’re in format mode and how to exit format mode while staying in focus mode. Thanks for continuing to try out the new preview! We’re working hard to react to your suggestions and add the necessary changes to make on-object work for you. Please continue to provide your comments directly in this blog post or in our community forum via the “Share feedback” button next to the preview switch. Modeling Updates to ORDERBY function If there are blanks in the data, you can specify where to order them by adding ‘BLANKS LAST’ or ‘BLANKS FIRST’. For example, this is a perfectly valid expression: BiggestSpender = SELECTCOLUMNS ( INDEX ( 1, ALLSELECTED ( 'DimCustomer' ), ORDERBY ( SUM ( 'FactInternetSales'[SalesAmount] ), DESC BLANKS LAST) ), 'DimCustomer'[FullName] ) Specifying how blanks are handled is optional and can be combined with specifying the order direction (DESC/ASC). Valid values include: BLANKS DEFAULT. This is the default value. The behavior for numerical values is blank values are ordered between zero and negative values. The behavior for strings is blank values are ordered before all strings, including empty strings. BLANKS FIRST. Blanks are always ordered on the beginning, regardless of ascending or descending sorting order. BLANKS LAST. Blanks are always ordered on the end, regardless of ascending or descending sorting order. Read more in our documentation. Data Connectivity New modern data connectivity and discovery experience in Dataflows We are excited to introduce the new data connectivity and discovery experience in Dataflow, Dataflow Gen2, and Datamart. Today, users spend a lot of time finding the right data, the right connection info and credentials. With the new Get Data experience, we make it easy to browse different Fabric artifacts through the OneLake data hub. This improved experience aims to expedite this process and get you closer to the data that you’re looking for in the quickest way possible. Learn more in our blog post Announcing a new modern data connectivity and discovery experience in Dataflows Lakehouses (Connector Update) This update includes significant performance improvements to the Lakehouses connector. Be sure to update to the August version of Power BI Desktop and Gateway to experience these improvements! Service XMLA Write support for Direct Lake datasets We are excited to announce that Direct Lake datasets now support XMLA-Write operations. Now you can use your favorite BI Pro tools and scripts to create and manage Direct Lake datasets. Whether you prefer SQL Server Management Studio (SSMS), Tabular Editor, DAX Studio, or something else, you can connect to your Direct Lake datasets using XMLA endpoints and perform operations such as deploying, customizing, merging, scripting, debugging, and testing. You can use tools like Azure DevOps or GitHub to implement source control, versioning, and continuous integration for your data models. You can automate and streamline your development and deployment processes. You can also use PowerShell or REST APIs to automate tasks such as refreshing or applying changes to your Direct Lake datasets. XMLA Write is incredibly powerful and the key to data modelling efficiency and productivity. For more information about XMLA Write support in general, check out the article Dataset connectivity with the XMLA endpoint in the product documentation. Automatic replica synchronization for Dataset Scale-Out (preview) We are excited to announce that we have finalized Dataset Scale-Out configuration APIs and completed the replica synchronization feature. Specifically, you no longer need to enable Scale-Out at the workspace level by using a burdensome XMLA request. The XMLA command is deprecated and will no longer work. You can now enable Scale-Out on a dataset-by-dataset basis using the Power BI REST API for datasets. You also no longer need to synchronize read replicas manually if you want to take advantage of automatic replica synchronization. Automatic replica synchronization is enabled by default. However, it is also possible to disable automatic synchronization to synchronize the read/write and read replicas of a dataset manually for controlled refresh isolation. Learn more in the automatic replica synchronization announcement blog Mobile Choose your start-up content Power BI mobile app users can now choose which item they want to have open automatically whenever they launch the Power BI Mobile app. This feature saves time for users who mostly view a specific item on their mobile app, and don't want to waste time navigating from the app’s home page every time they open the app. To configure a launch item for yourself, open the item you want to see when you launch the app. This can be a specific report page, dashboard, scorecard, report in an app or entire app. When the item is open, open the More options (…) menu from the header and select Set as launch item. This will mark the item as the launch item. Only one item at a time can be marked as the launch item. Administrators can also use mobile device management (MDM) tools to remotely configure a launch item for a group of users (front-line workers, for example) to simplify their experience with the app. Developers Published AppSource visuals now generate leads in Partner center and CRM Until recently, your customer leads may have come solely from customers downloading them from AppSource.com. However, now you have access to even more leads through Power BI - both the desktop and web embed AppSource. To access these new leads, simply navigate to the Referrals workspace in Partner Center. Here, you can see all the leads you receive from Power BI, as well as those from AppSource.com. Plus, if you've connected your CRM, you'll see them there too. By utilizing these new leads from Power BI, you can potentially reach more customers and increase your business's growth. So be sure to check your Referrals workspace and CRM regularly to stay on top of your leads! Publish a Power BI Project (PBIP) directly from Power BI Desktop You are now able to directly publish your Power BI Project (PBIP) files directly from Power BI Desktop, eliminating the need to save it as PBIX in order to activate the Publish feature. From an opened Power BI Project, choose: File > Publish > Publish to Power BI or select Publish on the Home ribbon. Select the destination workspace: And that's it, your PBIP dataset and report (or just report if it's a Live Connect) will be published to the selected workspace: To learn more about Power BI Projects, visit: https://aka.ms/pbidesktopdevmode Visualizations New visuals in AppSource The following are new visuals this update: Performance Flow - xViz Time-lines by BI-Champ Composed Line Area Bar Chart by Devlup Funnels Galigeo For Power BI Radial chart by Devlup Funnels Editor's pick of the quarter accoPLANNING Enterprise - Planning Power BI Writeback Sunburst Chart by Powerviz Spider Chart for Power BI by VisioChart Advanced Trellis / Small Multiples - xViz Drill Down Combo PRO Zebra BI Cards Shielded HTML Viewer Drill Down Combo Bar PRO by ZoomCharts Drill Down Combo Bar PRO by ZoomCharts offers a wide selection of customization options, letting creators build everything from regular bar charts to box and whisker plots. This visual also offers powerful cross-chart filtering capabilities combined with intuitive on-chart interactions. MAIN FEATURES: Multiple chart types – choose between column, line, and area charts Full customization – customize X and Y axes, legend, outline and fill settings Stacking and clustering – choose normal, 100% proportional, or zero-based stacking Static and dynamic thresholds – set up to 4 thresholds Multi-touch device friendly – get the same experience on mouse and touch input devices POPULAR USE CASES: Sales and marketing – sales strategies, sales results, and campaign-by-campaign marketing metrics Human resources – hiring, overtimes and efficiency ratios by department Accounting and finance – financial performance by region, office, or business line Manufacturing – production efficiencies and quality metrics by product line ZoomCharts Drill Down Visuals are known for their interactive drilldowns, smooth animations, and rich customization options. They are mobile friendly and support: interactions, selections, custom and native tooltips, filtering, bookmarks, and context menu. Try Drill Down Combo Bar PRO now by downloading the visual from AppSource. Learn More about Drill Down Combo Bar PRO by ZoomCharts. Sunburst Chart by Powerviz We are thrilled to present the new Sunburst Chart by Powerviz, a powerful visualization designed to display hierarchical data in a user-friendly and intuitive format. With its concentric circle design, you can easily display part-to-whole relationships and gain valuable insights from your data. Key Features: Rich Customization: Control display style, labels, Center circle, fill patterns and dynamic images. Color Options: Choose from 30+ color palettes, including color-blind safe options. Ranking: Easily filter the Top/Bottom N by each level and show the remaining categories as "Others". Conditional Formatting: Easily identify outliers based on value or category rules. Interactive Features: Enjoy full interactivity with zoom, drill down, and cross-filtering for detailed exploration. Many other features included annotations, grid view, show condition, and accessibility support. Business Use Cases: Sales and Marketing: Market share analysis and customer segmentation. Finance: Department budgets and expenditures distribution. Operations: Supply chain management, identify inefficiencies in manufacturing process. Education: Course structure, curriculum creation. Human Resources: Organization structure, employee demographics. Check out our video Introducing Sunburst Visual by Powerviz – A Powerful Power BI Custom Visual Get Sunburst Visual for FREE from AppSource Check out the visual features in sample file. Step by Step instructions and documentation can be found here. To learn more, visit Powerviz website xViz Performance Flow by Lumel xViz Performance Flow by Lumel is an integrated business flow monitoring visual with an interactive KPI tree visualization for organizational performance management use cases. It offers insights into People, Places, Processes and Entities with Performance Indicators, Trendlines, and Advanced Alerting on Goals, Metrics and their Variances. Performance Flow is suitable for a wide range of use cases: Organization Performance - Unveil HR/Employee Performance insights through an interactive pictorial Org Tree Chart with performance attributes and trends. Process Flow - Visualize business flows in various stages using Swim lanes with connector lines, icons, KPI metrics, and trends in one single integrated view. Financial Performance – Cost Center analysis with Scorecards or KPI Trees. Sales Performance - Dive deep into Sales Performance at Regional, Departmental, and Salesperson Levels with Decomposition Trees. The visual is packed with analytical and interactive features like: Interactive hierarchy navigation including quick search, zooming, subtree analysis, etc. Streamline Processes using Swim Lanes like in Visio Uncover hidden/Dotted connections using Links Data-driven conditional formatting rules Custom Tabs for end-users Try xViz Performance Flow visual from Microsoft AppSource today. To know more about the visual refer to our website Performance Flow and our video xViz Performance Flow - Introduction Synapse Data Warehouse SSD Caching T-SQL queries targeting massive amounts of data not fitting in-memory cache suffer from cache misses and higher latency due to repetitive reads from remote storage. SSD caching stores frequently accessed data on local disks in highly optimized format, significantly reducing IO latency and accelerating query processing. For more information check https://learn.microsoft.com/en-us/fabric/data-warehouse/caching. Data Engineering High Concurrency Mode for Notebooks We are thrilled to announce session sharing in Fabric through High Concurrency mode for Data Engineering and Data Science workloads. You can run notebooks simultaneously on the same cluster without compromising performance or security when paying for a single session. Session sharing is strictly within a single user boundary offering enhanced security and isolation also allowing you to do more while paying less. To learn more about High Concurrency mode in Fabric for Data Engineering and Data Science check out the documentation : High Concurrency mode in Fabric Spark Data Science Model & Experiment Usability Improvements We’ve made several usability enhancements to our Model & Experiment tracking features. You can now stay informed with real-time notifications for model and experiment updates. Plus, users can now enjoy a more seamless browsing and comparison experience with improved Run List and Model List views. New Data Science Samples are Coming to Microsoft Fabric We're thrilled to announce that Microsoft Fabric is introducing two exciting new Data Science Samples that showcase the power of Microsoft Fabric capabilities. The first sample focuses on bank customer churn problem and aims to build a machine learning model to predict whether bank customers would churn or not. The second sample is about machine failure and revolves around the use of machine learning to have a more systematic approach to fault diagnosis to proactively identify issues and take actions before a machine's actual failure. Both these samples provide a comprehensive display of end-to-end data science workflow, demonstrating Microsoft Fabric versatility in addressing diverse real-world challenges with AI-driven solutions. We can't wait to see the incredible possibilities these samples will unlock for our customers! To check these new Data Science samples along with others, please visit Microsoft Fabric and then select Synapse Data Science. Click on Use a Sample to access all Data Science samples. Real-time Analytics KQL Database support for inline Python Fabric KQL Database supports running Python code embedded in Kusto Query Language (KQL) using the python() plugin . The plugin runtime is hosted in a sandbox, an isolated and secured environment hosted on KQL Database compute nodes. This sandbox contains the language engine as well as common mathematical and scientific packages. The plugin extends KQL native functionalities with a huge archive of OSS packages, enabling Fabric users to run advanced algorithms, such as machine learning, artificial intelligence, statistical tests, time series analysis and many more as part of the KQL query. The Python plugin runs a user-defined function (UDF) using a Python script. The Python script gets tabular data as its input, and produces tabular output. The plugin is disabled by default. Before you start, enable the Python plugin in your KQL database. To enable the plugin browse to your KQL Database, select Manage -> Plugins, and enabled the plugin by toggling the button to On. KQL Database provisioning in a few seconds The KQL Database provisioning process has been optimized. Now you can provision a KQL Database within a few seconds. All you need to do to Create a new KQL database, is to give it a name, and after a few seconds you will have a fully functional KQL Database, that you can now start ingesting and querying your data. Data Factory Dataflows Gen2 Power Query editor Edit connection in Manage Connections The Manage Connections feature was recently release with only the capability to view the linked connections to your dataflow and the ability to unlink a connection from your dataflow. We’ve now added the capability to Edit a connection credentials and gateway from within the dialog. We continue to work on improving this experience during this calendar year and we welcome you to give this feature a try and look forward to future improvements on this feature. Enable Staging labels The concept of staging data was introduced in Dataflows Gen2 for Microsoft Fabric and now you have the ability to define what queries within your Dataflow should use the staging mechanisms or not. Learn more about the staging mechanism used in Dataflows Gen2 from the Data Factory Spotlight blog post. Publish dataflow without staged queries With the introduction of the ability to set the behavior of a query to be staged or not, you can now set all your queries to be evaluated without any staging and load the data directly to a destination of your choice. Note that your Dataflow Gen2 must at least have a single query with a data destination defined. The main benefit of this pattern is that your data will not need to be staged first which could potentially save time if the evaluation of your Dataflow might be quick and doesn’t need the overhead of staging mechanisms or it simply goes more in line with how you want your solution to be designed for a number of reasons. Limit to number of queries in a Dataflow when publishing When creating a Dataflow Gen2, we are now modifying the maximum number of entities that can be part of a particular Dataflow. The new maximum number of entities is 50. If you have 51 or more entities in your Dataflow, you will receive a warning letting you know that you need to reduce the number of entities to a maximum of 50 before you can publish your Dataflow. You are still able to save your Dataflow as draft before you make any changes. Refresh history Direct navigation to last failed refresh If you are ever in a situation where your Dataflow refresh failed, you can now click the warning sign right next to the timestamp in the Refreshed column and get taken directly to the Refresh history dialog for that particular refresh attempt. The goal of this new improvement is to reduce the number of clicks to get you to the last failure. Statistics for tables Inside the refresh history dialog you are now able to drill down to a particular table and see the volume processed for it as well as the endpoint where the volume was processed. More information such as Duration, Start Time and End Time are still available to you in this dialog. Connectors Lakehouse connector update This update includes significant performance improvements to the Lakehouses connector. Be sure to update to the August version of Power BI Desktop and Gateway to experience these improvements! Other improvements We are actively listening to your feedback and the feedback of thousands of customers who are trying out Dataflow Gen2 today. Some of the feedback doesn’t directly translate to new features, but rather to fixes or quality improvements to our backend and how reliable our service can be. Our team has been able to triage and work on more than 600 fixes and improvements in the past month. The list below is a small list of some of the categories of these fixes and the impact that it’ll have on how you use Dataflow Gen2 today: Better error messages: We are actively improving the error messages and have improved some of the most common error messages in the past few weeks. There’ll be multiple improvements on the error messages and categorization front in the coming months but we’re happy to be making tactical changes today to improve the experience and make it clear to users on what could be happening to produce the error. Multitasking efforts: There have been some issues in regard to multitasking in Fabric. A few of those have been addressed, but we’re actively working on a much better experience for Dataflow Gen2 with multitasking in mind much similar to how other artifacts leverage multitasking capabilities. Reliability and performance: We’re continuously working towards improving the reliability and performance of Dataflow Gen2 in Microsoft Fabric. This will translate into much faster refreshes times and with more reliability. Data pipelines Connectors FTP Connector We’re excited to announce that the FTP connector is now available to use in your Data Factory data pipelines. In your data pipeline, you can create a new connection to your FTP data source to copy, extract, and transform your data. Productivity Lookup activity for Lakehouse, Data Warehouse, and KQL Database The Lookup activity now connects to Fabric Lakehouse, Data Warehouse, and KQL Database. This now makes it easier for you to read or look up records, table names, and other values from your Fabric artifacts to use in downstream activities in your data pipeline. Get Metadata activity for Lakehouse and Data Warehouse The Get Metadata activity now connects to Fabric Lakehouse and Data Warehouse, making it easy to retrieve metadata from your data in your Fabric artifacts to use downstream in your data pipeline. Pipeline run status now available We’ve recently added Pipeline run status so that developers can easily see the status of the pipeline run. You can now view your Pipeline run status from the Output panel. Secure input and output options for Set Variable activity We’ve added advanced settings for the Set Variable activity called Secure input and Secure output. When you enable secure input or output, you can hide sensitive information from being captured in logs. Data Activator Trigger and property design template As mentioned last month, we have been working on a new experience for designing triggers and it’s now available in our preview! You now see 3 cards in every trigger: Select, Detect, and Act: ‘Select’ is where you choose the value you want to monitor. It can be a direct reference to a column from an event stream, or to an existing property. Once you’ve selected an input, you’ll see a preview and can add grouping/smoothing or filters to get the right value: The Detect card is where you specify the conditions and thresholds that you want to take action on. You select the type of threshold, enter the values and can optionally choose things like firing the trigger every time the condition is met, or only when it’s met a certain number of times over a longer period. The top chart shows, for the instances you have selected in the preview, when this trigger would have fired. The bottom charts shows the overall nubmer of times the trigger would have fired for all instances. This gives you an idea of how many emails or Teams messages would have been generated! Finally the Act card lets you set the action you want Data Activator to take. You can choose the recipient, optional information etc. as you build it out. Quickly assign columns as properties in an object From the data view, you can quickly assign an event stream to a new or existing object and make multiple properties from one UI. You need to choose the key column that identifies the individual object instances that you care about (e.g. a package ID, employee ID, location name etc.). Then, in the ‘Assign properties’ dropdown, select the columns from your event stream that you want to use as properties in the object. You can use the ‘Assign to existing’ option to map a second event stream onto an existing object, combining data from two events. Data Activator now supports Power BI visuals with a time axis Data Activator now supports Power BI visuals with a time axis. In the screenshot below, we create an alert if the “occupancy” measure on the visual goes above 60%. Note that Data Activator has detected the presence of the time axis and is highlighting this in the alert pane. Trigger Power Automate flows from Data Activator When a data activator trigger fires, you can now trigger a Power Automate flow. This means that you can use Data Activator to drive actions in any system that Power Automate can connect to. You can send an alert in a 3rd party alerting system, log a ticket in a ticketing system, or call a REST API to trigger actions in an operational system. The list is (almost) endless! To connect Data Activator to Power Automate, you create a custom action. A custom action is a reusable action template that triggers a flow. Once you have made a custom action, you can use it in any trigger, in any Data Activator workspace or item. Here, we create a custom action to send an SMS message via a 3rd party connector: After creating the custom action you can use it in any trigger, by selecting it in the “Act” card in the trigger designer. The act card prompts you for the input fields that need to go to the flow. Here, we see the action card for our SMS action. It is prompting for the phone number and message: For the trigger creator, the action works just like natively-supported email and Teams actions. This means that you can get a Power Automate expert in your organization to create the custom action, then roll it out to all Data Activator users, even they don’t have any Power Automate experience. That is all for this month! Please continue sending us your feedback! If you haven’t already head over to the Fabric Community to join in the conversation!Microsoft Fabric October 2023 update
Microsoft Fabric October 2023 update We have a lot of features this month including updates to the Item type icons, Keyword-Based Filtering of Tenant Settings, On-object Interaction updates, Eventstream Kafka Endpoints and many more. Continue reading for more details on our new and updated features! Core Item type icons Admin Keyword-Based Filtering of Tenant Settings Power BI Reporting Power BI Desktop OneDrive and SharePoint integration On-object Interaction Updates (preview) New! Date Hierarchy on data flyout New! Placeholder text for direct text editing New! Ribbon and Funnel charts now support on-object formatting. Power BI Home in Desktop Deduplication rules for composite models on Power BI Datasets and Analysis Services Modeling Edit your data model in the Power BI Service – Updates Model explorer public preview with calculation group authoring and creating relationships in the properties pane. Data connectivity Snowflake (Connector Update) Planview OKR (New Connector) BitSight Security Ratings (Connector Update) Starburst Enterprise (Connector Update) Service OneLake data hub – Quick access to your data Explorer pane - Quick access Favorite items Visualizations New visuals in AppSource ValQ Plan Date Picker by Powerviz Drill Down Network PRO: Show Categorical Relationships with Ease TMap 2.1 Inforiver Enterprise brings writeback to Fabric Lakehouse & Warehouse Synapse Data Warehouse V-Order write optimization SKU guardrails for burstable compute Data Science Semantic Link (Public Preview) Real-time Analytics KQL Database Capacity Reporting KQL Database Auto scale algorithm – improvements Filtering and visualizing Kusto data in local time with special PowerBI optimization Eventstream UX Improvement on Event Processor Eventstream Kafka Endpoints and Sample Code Data Factory Dataflow Gen2 Data connectivity SAP HANA (Connector Update) New certified connector: Emplifi Metrics Bug fixes and reliability improvements Pipelines Activities AML activity Deactivate/reactivate activity state (Preview) Productivity Category redesign of activities Copy runtime performance improvement Integer data type available for variables Pipeline name now supported in System variables. Support for Type editing in Copy activity Mappings Data Activator Data Activator is now in public preview Core Item type icons We’ve heard your feedback that the icons across Fabric are difficult to tell apart from large list views and other areas of the UI, and that the overall appearance of them was overly monochromatic and flat. Our design team has completed a rework of the item type icons across the platform to improve visual parsing and enrich the way users may form a habituated understanding of what icons operate within different semantic categories of working with data. To learn more about the thinking behind this update, see the detailed blog Admin Keyword-Based Filtering of Tenant Settings Microsoft Fabric has recently introduced keyword-based filtering for the tenant settings page in the admin portal. This new feature allows admins to locate the necessary tenant settings quickly and easily by filtering them based on keywords. To use keyword-based filtering, tenant admins simply enter the keywords they are looking for into the search bar at the top of the tenant settings page. The portal will then filter and display only those settings with matching keywords in the title or description. For instance, an admin could search for "preview" to access all tenant settings currently in preview, or "B2B" to view settings associated with the B2B data sharing feature. To learn more about the feature, check out this blog post. Power BI Reporting Power BI Desktop OneDrive and SharePoint integration OneDrive and SharePoint integrations are now easier than ever in Power BI Desktop! While the ability to open, save, and share reports in OneDrive and SharePoint was released to preview in May, the capabilities are now improved and on by default. You can: Open reports stored in OneDrive and SharePoint through the file menu. Save files directly to OneDrive and SharePoint. Share reports stored in OneDrive and SharePoint directly from Power BI Desktop. These are important changes because many Power BI authors use OneDrive and SharePoint to collaborate on their reports before publishing through the Power BI service. These new updates will streamline their workflow. Ad-hoc reporting is made easier and simpler, and new users starting out will be comfortable with the familiar Office interface. The new Desktop features are complimented by the ability to view Power BI reports stored in OneDrive and SharePoint directly in your browser. Previously, viewing a report stored in OneDrive or SharePoint required downloading both the Power BI file, and Power BI Desktop. The new capability allows users to interact with their reports in seconds. We understand that every organization has its unique needs and preferences. If you prefer not to have some of these features available in your organization, learn how to turn them off in the Fabric admin portal. Learn more about these features in our public preview announcement blog post. On-object Interaction Updates (preview) New! Date Hierarchy on data flyout When working with dates, you may choose to swap from a hierarchy to the raw date field. This is still available when right clicking directly on the date field. For greater discoverability, we’ve now added the ability to switch between date hierarchy and raw date on the data flyout as well. Option 1: Swap using right click. New! Option 2: Swap using data flyout. New! Placeholder text for direct text editing Previously, when turning on a text element such as a title for Tables, there was no reserved space to begin typing in characters directly on-object. Users had to go to the format pane to add a title. Now, when turning on a text element that does not have an “auto” value, we show a placeholder for users to begin typing directly on the visual. When using direct text edit, you’ll also see placeholders appear if you accidentally delete all the characters, but the text element is still ON. This again reserves the space for you to be able to come back and a text value using on-object. Placeholders only appear when the visual(s) are selected while editing, deselecting the visual will remove all placeholders so authors can preview what will be published. New! Ribbon and Funnel charts now support on-object formatting. Power BI Home in Desktop Deduplication rules for composite models on Power BI Datasets and Analysis Services Tables and measures in your model must have unique names. If you use composite models on Power BI Datasets and Analysis Services, it’s easy to get into a situation where tables and measure names have duplicate names and are not unique. Up to this point, when that happened one of the tables or measures would be renamed for you. For example, if you created a composite model from two sources and both sources defined a table called ‘Customers’”, one of the tables would be renamed ‘Customer 2’. This resulted in confusing situations as it was not clear which source the ‘Customer 2’s table came from. The same applies to measures: if you had two sources that both contained a measure called ‘Total Sales’ one would be renamed ‘Total Sales 2’ in the composite model. This month we are giving you more control! You can now apply a name disambiguation rule to a source in a composite model when you anticipate name conflicts with tables or measures from another source. You can set up a text to be added as a prefix or suffix to table names, measure names or both. Additionally, you can choose to add that text only when a duplication occurs or if you prefer to have it added all the time. Going back to the example above, let’s say that one of the sources you are combining is for marketing and the other is for sales. You can now set up a deduplication rule on the source connection so the ‘Customer’ table from the marketing source is named ‘Customer (marketing)’: You will find these options under Settings in the dialog that shows when you set up the composite model connection to a Power BI dataset or Analysis Services model: After you make the connections and set up the deduplication rule, your field list will show both ‘Customer’ and ‘Customer (marketing)’ according to the deduplication rule you set up: Note that you can: Specify whether you want the text to be added to the table or measure name as a prefix or a suffix. Apply the deduplication rule to tables, measures, or both. Choose to apply the deduplication rule only when a name conflict occurs or apply it all the time. The default is to apply the rule only when duplication occurs. In our example, any table or measure from the marketing source that does not have a duplicate in the sales source will not get a name change. If you do not specify a deduplication rule or the deduplication rules you specified do not resolve the name conflict the standard deduplication actions are still applied. Read more about composite models on Power BI datasets and Analysis Services in our documentation. Modeling Edit your data model in the Power BI Service – Updates The new data model editing in the Service feature was released to preview in April. We’ve been busy reacting to your feedback and enhancing the experience. Below are the improvements coming later this month: Manage relationships dialog. Now you can easily view and edit all the relationships within your data model in the Service! In the Home tab, simply select the ‘Manage relationships’ button. This will open the revamped ‘Manage relationships’ dialog, which provides a comprehensive view of all your relationships, along with their key properties, in one convenient location. From here you can then choose to create new relationships or edit an existing relationship. Additionally, you have the option to filter and focus on specific relationships in your model based on cardinality and cross filter direction. Mark as date table Within the Service, you can now mark a table in your data model as a date table. Marking a date table in your model allows you to use this table for various date-related elements including visuals, tables, quick measures, and more, with full Time Intelligence support. To set a date table in the Service, right-click on the desired table and choose ‘Mark as date table > Mark as date table’ in the menu that appears. Next, specify the date column by selecting it from the dropdown menu within the 'Mark as date table' dialog. Power BI will then perform validations on the selected column and its data to ensure it adheres to the 'date' data type and contains only unique values. Please continue to submit your feedback directly in the comments of this blog post or in our feedback forum. Model explorer public preview with calculation group authoring and creating relationships in the properties pane. We are excited to announce that the model explorer is now available for public preview in the model view. You can see all your dataset semantic modeling objects in one place and easily navigate between them. Finally, full visibility of the semantic model! An additional properties pane now shows for the dataset semantic model. And new icons are also showing on the Data pane! Additional UI changes will continue through December for the Data pane and properties pane. Also note that measure groups will always show at the top, followed by calculation groups, and then finally the other tables in the model. From the model explorer, not only can you now see the calculation groups with their calculation items, but you can also create and edit them in Desktop! Calculation groups are a powerful feature that allows you to apply dynamic calculations to your existing measures. For example, you can create a calculation group that applies time intelligence functions as calculation items, such as year-to-date, quarter-to-date, or month-to-date, to any measure in your model. Learn more at aka.ms/calculationgroups. To author these in Desktop, you go to the model view and click on the new “Model” tab of the Data pane. If you click on the Calculation groups node, you have three options to create a new one. (1) Ribbon button, (2) Context menu, (3) Properties pane. After clicking “New calculation group”, if you do not have the model property discourage implicit measures turned off, you will be told this setting needs to be on to create the calculation group. Once turned on, the calculation group is created, and you are landed directly in the first calculation item to define the DAX expression you want to apply to your existing measures. You can alter this DAX expression in the DAX formula bar. Optionally, you can also add a dynamic format string to the calculation item from the properties pane. New calculation items can be created from the context menu (right-click) of the Calculation items node or the calculation group, and in the properties pane of the Calculation items node. Controlling the order of the calculation items can be done in the Calculation items node properties pane or using the context menu (right-click) of the calculation items themselves! Known issues to be fixed before public release: Renaming calculation items will not reflect in the report view slicer until another calculation item added or deleted. Changing the order of calculation items in properties pane moves away from the properties pane. Changing precedence of the calculation groups in the properties pane moves focus away from the properties pane. Is hidden showing “Mixed values.” Cursor not being in the calculation item on creation. Another option also now available from the model explorer is the ability to create relationships in the properties pane. Just like edit relationships in the properties pane allows you to simply change table and columns without previewing data or validating until you click Apply change, this experience is now available to create relationships too. Simply choose new relationship from the model explorer Relationships node. This will show an empty relationship properties pane to fill out and then apply when ready! Learn more about adding and editing relationships at Create and manage relationships in Power BI Desktop - Power BI | Microsoft Learn. Known issues to be fixed: Setting the relationship as one too many even in a valid way will result in an error. This has been fixed but waiting for the changes to be included in Desktop, which may be after public release. The model explorer does show you other modeling features that do not yet have authoring paths in Desktop. These features are perspectives and cultures. These will still need to be authored outside of Desktop throughout XMLA write external tools or through XMLA directly. Learn more about XMLA write at Dataset connectivity and management with the XMLA endpoint in Power BI - Power BI | Microsoft Learn and external tools at External tools in Power BI Desktop - Power BI | Microsoft Learn. Finally, learn more about perspectives at Perspectives in Analysis Services tabular models | Microsoft Learn which work well with the personalized visuals feature of Power BI reports at Let users personalize visuals in a report - Power BI | Microsoft Learn. And cultures is the translations features of semantic models, learn more at Translations in Analysis Services tabular models | Microsoft Learn. These features are available in the latest version of Power BI Desktop. To use them please turn on the model explorer public preview switch. Go to File > Options and settings > Options > Preview features in the GLOBAL section. Data connectivity Snowflake (Connector Update) The Snowflake connector has been updated to support better implementation of “LIMIT 1” queries, resulting in performance improvements. Planview OKR (New Connector) We are excited to release the new Planview OKR connector. Here are the release notes from the Planview team. Planview Objectives and Key Results (OKRs) are an outcome-driven framework adopted by organizations who want to define key organizational goals and track progress toward achieving them. Defining OKRs creates organizational clarity by enabling organizations to answer the questions “Where do we want to go?” (objectives) and “How will we measure our efforts to get there?” (key results). OKRs can be created at different levels of an organizational structure – such as enterprise, portfolio, program, or team – and are connected using parent/child relationships. Linking organizational and team goals in a hierarchical way aligns work delivery to company strategy and provides a single line of sight into value delivered by the organization. Connect now to your OKR data with the custom connector Planview OKR. BitSight Security Ratings (Connector Update) The BitSight Security Ratings connector has been updated with minor bug fixes. Starburst Enterprise (Connector Update) The Starburst Enterprise connector has been updated. Here are the release notes from the Starburst team: Added optional Connection string (non-credential properties) in the Advanced Section to allow the use of other ODBC connection parameters. Changed data source display from Starburst Enterprise to a value based on the connection details ({“Host”:”sep.example.com”, “Port”:”443”}) to allow distinction between multiple Starburst Galaxy or clusters connected as separate data sources. Fixed an issue with port 8443 for OAuth 2.0 authentication. Fixed query folding with timestamp columns. Service OneLake data hub – Quick access to your data OneLake data hub is the central location for users to find and reuse existing organizational data. It allows users to browse through their data and discover insights that can help them make better decisions. We are happy to introduce some of the new features that we recently added to enhance the OneLake data hub discovery experience. Explorer pane - Quick access The Explorer pane enables users to navigate through the workspaces hierarchy and scope the data items to a specific workspace. With the recent enhancements, we added a Quick access section at the top of the Explorer pane. The Quick access section contains an active workspace, pinned workspaces, and recently used workspaces. Favorite items Another new feature is the favorite items. Users can now filter to view items that were marked previously as favorite in Power BI. Users can also find favorite items from within the data hub experience, and they will also show up across all the experiences including Home and Browse. This feature helps users to keep track of the data items that are most important for them. Visualizations New visuals in AppSource Step Filter ValQ Plan valQ Plan is a newer and re-architected Power BI-certified edition of valQ with numerous transformational updates and a simplified no-code experience. valQ helps users build complex business plans, what-if simulations, create & compare budgets, forecasts and scenarios – all within Microsoft Power BI. It can be used as your standalone and integrated business planning software tool or used in conjunction with your existing planning platforms. YT video URL - https://www.youtube.com/watch?v=w4AFQzr58j4 Feature highlights: Niche Use Cases such as Scenario Planning, Strategic Planning or Value Drive Trees and What-If Simulations. Typical Planning use cases: Financial planning & analysis (FP&A), 3-statement financial modelling, and Supply Chain Demand Planning etc. Excel-like grid input experience for your rolling forecasts with no-code auto forecasting algorithms, and goal seek. Integrated reporting: One-click IBCS templates for variance reporting, presentation mode with live charts and tables, enhanced export to Excel & PDF Intuitive modelling: Significantly enhanced formula editor experience, dynamic templating and more Time intelligence: Supports more than 12 periods allowing you to create 90-day rolling forecasts, 52-week forecasts, and even 10-year strategic plans. ValQ Plan can be purchased directly from Microsoft AppSource. Date Picker by Powerviz The Ultimate Date Slicer for Power BI. The Date Picker visual comes with a modern calendar view along with highly requested features like Presets, Pop-up mode, Default Selection, Themes, and more. This is a must-have date slicer for all the Power BI reports. It has rich formatting options to match your brand style guide and to meet your business needs. KEY FEATURES: Display Mode: Choose between Pop-up and Canvas modes. Presets: Many commonly used presets like Today, Last Week, YTD, MTD, or create your own preset using field. Default Selection: Enforce the selection of the desired period when a user opens the report without any custom DAX. Filter Type: Choose between Range and Start/End types. Multiple Date Ranges: Flexibility to select multiple date ranges. Themes: 15+ pre-built themes with full customization. Holidays and Weekends: Multiple formatting options. Import/Export JSON: Build templates and share your designs. Many more features and customizable options. Check out the video - Introducing Date Picker by Powerviz. Get Powerviz Date Picker for FREE from AppSource. Download the demo file here. Step-by-step guide and detailed documentation of all features. To learn more, visit Powerviz website. Drill Down Network PRO: Show Categorical Relationships with Ease Drill Down Network PRO by ZoomCharts is designed for effortlessly visualizing categorical data and automatically detects relationships based on category structure. Use your existing category-based data with few adjustments, and quickly create an interactive chart that makes relationships between each data category easy to read. Visit our site to learn more! Main features: Cross-chart filtering – eliminate slicers by selecting data points directly on the charts. Category-based customization – apply image to node and choose its type, shape, color, font. Link styling – configure 'from' and 'to' decorations and show link value. Display up to 9 data categories. Touch-input device friendly – explore data on any device. Popular use cases: Accounting & Finance – show cost attribution. Human Resources – analyze salary data by department. Production – map production volumes by product or factory. Sales & Marketing – visualize marketing campaigns. ZoomCharts Drill Down Visuals are known for interactive drilldowns, smooth animations, and rich customization options. They support interactions, selections, custom and native tooltips, filtering, bookmarks, and context menu. Use them to create visually appealing and intuitive reports that business users will love on every device. Try Drill Down Network PRO now from the AppSource! TMap 2.1 Newly released TMap 2.1 has been added new features for drilling down Donut Map, Choropleth Map, Bar Chart Map, Pie Chart Map and Stacked Bar Chart Map by georegion's names. Drilling down maps by georegion's names doesn't need to prepare a hierarchy of pre-built polygon layers and will save data professional's time and cost to extract deeper insights from the data on different levels of geographic regions. Screenshot 1 (Drilldown donut map for company A's sale in Asia by subregions) Realigning georegions will become much easier because you only need to change georegion's name in the lowest level and the polygon layer for upper level will be automatically generated during the drilling down process. Screenshot2 (Drilldown donut map for company A's sale in Asia by realigned subregions) You can go to Microsoft AppSource( https://appsource.microsoft.com/en-us/product/power-bi-visuals/mylocsinc1648311649136.tmap?exp=ubp8) to download and try it. To learn more on how to use it, please read tutorials (https://www.mylocs.ca/tutorials.html#drilldown-donut-map-name). Inforiver Enterprise brings writeback to Fabric Lakehouse & Warehouse Inforiver now supports three new data writeback destinations from Power BI: Microsoft Fabric Lakehouse, Fabric Datawarehouse, and Dataverse, along with other popular DWH, DL, and Databases. Here is our 2-minute demo and overview The latest Inforiver Enterprise release supports writeback to Fabric for these use cases. Create business projections by entering/editing directly in Power BI reports. Create & manage rolling forecasts by blending actual & forecast data series. Create dynamic simulations at the enterprise, BU, or department level and roll up projections (to the top) or distribute projections (to granular dimensions) based on chosen allocation/distribution methods. Create multiple scenarios, each containing its own set of simulations. Write back data, including comments & threaded conversations Create periodic report snapshots. Support multiple data input types and writeback from multiple Power BI users in reading mode. The above, combined with advanced audit, security & governance capabilities, make Inforiver the most advanced data input and write-back solution in the market. Inforiver supports writeback in the following deployment configurations: (a) Azure SAAS service managed by Inforiver or (b) managed by customers in their private tenant. Visit our FAQ page to learn more. For Fabric Writeback Proof of Concept (PoC), contact Inforiver here. Synapse Data Warehouse V-Order write optimization V-Order optimizes parquet files to enable lightning-fast reads under the Microsoft Fabric compute engines such as Power BI, SQL, Spark and others. Warehouse queries in general benefit from faster read times with this optimization, still ensuring the parquet files are 100% compliant to its open-source specification. Starting this month, all data ingested into Fabric Warehouses use V-Order optimization. SKU guardrails for burstable compute Synapse Data Warehouse on Microsoft Fabric has the desired flexibility to allow better performance under peak demand by providing burstable compute. SKU Guardrails ensure that customers are operating within the right boundaries for their capacity, preventing peak workloads from consuming all capacity units in a short duration. To learn more about SKU guardrails for burstable compute, check out the blog Data Warehouse SKU Guardrails for Burstable Capacity Data Science Semantic Link (Public Preview) We are pleased to introduce the Public Preview of Semantic Link, an innovative feature that seamlessly connects Power BI datasets with Synapse Data Science within Microsoft Fabric. As the gold layer in a medallion architecture, Power BI datasets contain the most refined and valuable data in your organization. With Semantic Link, we unlock this data's potential beyond traditional business intelligence by making it accessible to notebooks and Python in Microsoft Fabric. Real-time Analytics KQL Database Capacity Reporting A KQL Database will utilize capacity via Operations that can be monitored using the Microsoft Fabric Capacity Metrics. KQL Database Consumption - This is the number of seconds that your KQL database is active in relation to the number of virtual cores used by your database. For example, if my database uses 4 virtual cores and is active for 10 minutes then you will utilize 2,400 seconds of Capacity Units. An auto-scale mechanism is utilized to determine the size of your KQL database. This ensures the most cost-optimized and best performance based on your usage pattern. Read more in the blog: Understanding Fabric KQL DB Capacity | Microsoft Fabric Blog | Microsoft Fabric KQL Database Auto scale algorithm – improvements Users do not need to worry about how many resources are needed to support their workloads in a KQL database. KQL Database has a sophisticated in-built auto scaling algorithm. The algorithm ensures that the optimal number of resources are allocated to support the workloads, with minimum cost. The auto scaling algorithm is multi-dimensional, based on the following dimensions: Hot cache- how much data to store in hot cache for immediate response. This metric is impacted by the caching policy defined by the customer. Memory – how much memory required for meta data and data for optimal query performance. CPU usage- how much compute resources are needed to process queries, update policies, materialized views etc Ingestion - how much compute resources are needed to for data ingestion based on ingestion rates and load times. Filtering and visualizing Kusto data in local time with special PowerBI optimization Datetime values in Kusto (aka ADX/KQL database in Fabric) are assumed to be in UTC. There are good reasons why you should always keep it this way. On the other hand, in many cases you want to visualize the datetime values in a specific time zone and filter the data using values expressed in local time. This is correct but may lead to severe performance degradations if not done correctly. We recently implemented some optimizations that will make some such scenarios much more efficient. In addition, for the many users who are using PowerBI with Fabric KQL database for time series analysis, here is the optimal recommendation: Create a function that will receive the time range as parameters, shift it to UTC, filter the table and shift the filtered rows from UTC to the same time zone. For more details and examples refer to the following blog: Filtering and visualizing Kusto data in local time Eventstream UX Improvement on Event Processor The Event Processor within Eventstream is a powerful no-code editor, enabling you to process and manage your real-time data streams efficiently. You can easily aggregate and filter data streams using temporal functions before they reach your lakehouse or Kusto database. The recent UX improvements introduce a full-screen mode, providing a more spacious workspace for designing your data processing workflows. The insertion and deletion of data stream operations have been made more intuitive, making it easier to drag and drop and connect your data transformations. Eventstream Kafka Endpoints and Sample Code We've expanded the Custom App feature with a range of new endpoints in sources and destinations. Now, you can seamlessly connect your applications to Fabric Eventstream using protocols like EventHub, AMQP, and Kafka. To simplify your setup process, we've included sample Java code for your convenience. Simply add it to your application, and you'll be all set to stream your real-time event to Eventstream. Data Factory Dataflow Gen2 Data connectivity SAP HANA (Connector Update) The update enhances the SAP HANA connector with the capability to consume HANA Calculation Views deployed in SAP Datasphere by taking into account SAP Datasphere’s additional security concepts. This enables consumption of Calculation Views in Datasphere and allows customers to connect to HANA Cloud views without the need for additional privileges on the _SYS_BI schema. New certified connector: Emplifi Metrics We are happy to announce the release of the new Emplifi Metrics connector. Please find release notes from the Emplifi team below: “Integrating social media insights alongside the rest of your marketing or business intelligence data gives you a holistic understanding of your entire digital strategy, all in one place. With Emplifi Power BI Connector, you’ll be able to include social media data from the Emplifi Platform in your charts and graphs and combine them with other data you own. The Power BI Connector is a layer between Emplifi Public API and Power BI itself. It helps you work with your data intuitively, directly in the Power BI tool. Most data and metrics available in the Emplifi Public API are also available in the Connector. Please visit the official documentation for more information about Emplifi Public API and a list of available metrics. You will find it here: https://api.emplifi.io/.” Bug fixes and reliability improvements We would like to thank our community for reporting issues and providing feedback through the Data Factory Community Forum. We continue to work on improving the overall experience and reliability of Dataflow Gen2 in Microsoft Fabric. With over 400 work items closed in the last month, you should notice a better experience in the overall Dataflow Gen2 authoring and refresh experience. We encourage you to visit our community forum and provide any feedback or inquire about any possible issues that you might have with Dataflow Gen2. Your feedback is helping us make this a better product each day. Pipelines Activities AML activity We’re excited to announce that the Azure Machine Learning activity is now available to use in your Data Factory data pipelines. In your pipeline, you can use the Azure Machine Learning activity to connect to your Machine Learning pipelines or enable batch prediction scenarios such as identifying possible loan defaults, determining sentiment, and analyzing customer behavior patterns. Deactivate/reactivate activity state (Preview) We’re excited to share that you can now deactivate one or more activities from a pipeline, allowing you to skip activities during pipeline validation and during pipeline runs. This will help to improve developer efficiency, allowing you to comment out parts of your pipeline without deleting anything from your pipeline canvas. Deactivated activities can be reactivated at any time. Learn more here. Productivity Category redesign of activities We’ve redesigned the way activities are categorized to make it easier for you to find the activities you’re looking for with new categories like Control flow, Notifications, and more! Copy runtime performance improvement We've made improvements to the Copy runtime performance. According to our tests results, with the improvements users can expect to see the duration of copying from parquet/csv files into Lakehouse table to improve by ~25%-35%. Integer data type available for variables We now support variables as integers! When creating a new variable, you can now choose to set the variable type to Integer, making it easier to use arithmetic functions with your variables. Pipeline name now supported in System variables. We’ve added a new system variable called Pipeline Name so that you can inspect and pass the name of your pipeline inside of the pipeline expression editor, enabling a more powerful workflow in Fabric Data Factory. Support for Type editing in Copy activity Mappings You can now edit column types when you land data into your Lakehouse table(s). This makes it easier to customize the schema of your data in your destination. Simply navigate to the Mapping tab, import your schemas, if you don’t see any mappings, and use the drop-down list to make changes. Data Activator Data Activator is now in public preview Data Activator reached a big milestone this month, with its release to public preview. This means Data Activator is now available to all Fabric users, without the need to sign up to be a preview user. You can use Data Activator to drive alerts and actions from your Fabric data. Want to try it out right now? Open a Power BI visual and select the “trigger action” menu option, to create a Data Activator alert: Data Activator also works with real-time streaming data in EventStreams. To make a Data Activator alert on your EventStreams items, create a “reflex” destination: To learn more about Data Activator, check out the following: Data Activator demo video Data Activator documentation on Microsoft Learn
Announcing the public preview of task flows in Microsoft Fabric
Leverage the power of task flows to design and build your data solutions and manage workspace items in Microsoft Fabric. We're thrilled to announce that the task flows feature is now in public preview and is enabled for all existing Microsoft Fabric users. Fabric is unifying everything needed to deliver end-to-end data and analytics solutions within one complete and unified platform. Fabric offers a complete solution for an organization's data project, from start to finish. With Fabric task flows, when designing a data project, you no longer need to use a whiteboard to sketch out the different parts of the project and their interrelationships. Instead, you can use a task flow to build and bring this key information into the project itself. The task flow streamlines the design of your data solutions and ensure consistency between design and development efforts. It also allows you to navigate items and manage your workspace more easily, even as it becomes more complex over time. Hide and show the task flow in your workspace The new workspace page is divided into two sections: the task flow and the item list, each occupying half of the space by default. If desired, with just a click you can hide the task flow and only show the item list. You can also drag the bar up and down to resize the task flow. Design a data solution with a task flow A task flow is a visual guide for building your solution. It shows the relationships of tasks for completing a process or collection of processes. A task is a unit of process within the task flow and has items assigned to it. To set up a task flow, you can start by choosing one of the predesigned task flows that come built-in with Fabric. These predesigned task flows are based on what’s going on in the industry, and provide common, best-practice project designs that can help you get started quickly and efficiently. Task flows also feature a graphical tool that enables you to create professional diagrams with ease, helping you design your data project and communicate complex information in a clear and concise manner. If you decide not to use one of the predefined task flows, you can also get started by selecting task types and adding tasks to the task flow canvas one by one, gradually building your own task flow with the graphical tool. This enables you to create a tailored solution that meets your unique business needs and goals. Task flows make it easy for you to get started fast and efficiently, helping you to hit the ground running on your projects. Collaborate and follow task flow to build data solution A task flow shows your team members how the project works and helps them get started. It helps them understand the design and to build out the project task by task. For each task, it recommends the most relevant Fabric item types to use. You can select and create corresponding items from tasks directly. You don't have to wade through all the Fabric items to find the items you need. Rather, you'll discover the three or four items that truly fit the task. You can also assign existing items in the workspace to tasks that align with the project. The task flow facilitates a collaborative project and the ongoing development of living documentation as time progresses. Manage complexity by using the task flow With items assigned to tasks in the task flow, you can easily understand which items are doing what inside the project, even if you're revisiting it after a long period. You can navigate the workspace by using the task flow. When you select a task, the item list below shows only the items assigned to that task, making it easier to browse and understand what's going on. And you can focus on cleaning up one task at a time. The task flow assists in easy navigation and comprehension, regardless of when you return to the project. What early adopters are saying In the private preview, we observed many customers diving into task flows, discovering new insights, and providing valuable feedback. Below we’ve captured a few snippets: “The problem that Fabric faces is that workspaces can become cluttered and unmanageable and confusing. Someone new that accesses a workspace doesn’t know what to do. Task flows take this mess and make it into a cohesive, communicable, valuable story.” Simon Nuss, VP Data & Analytics, Hitachi Solutions “To document the flow in your architecture is extremely useful. and the transparency makes it easy to spot where you are in the flow.” Mads Steenstrup Hannibal, Principal Architect, Twoday kapacity “I have been quite active quite hands on because it’s a piece of functionality that interests me a lot because I do like things like the lineage view within fabric, power BI and of course this now gives us a way of grouping things together and giving a logical representation of a flow which I think is going to be a benefit.” Andy Cutler, Director Data, Datahai “The best beauty about task flow is like it makes my life much easier to understand (not a messy workspace). The whole flow where we start, what we are doing and everything. So that’s the one of the beauty I see. “ Matthias Eisenack, Corporate Controlling – Portfolio Manager Financial Performance, Bayer AG “Really helps abstract the complexity of individual items to see a higher-level view.” Ed Freeman, Data Engineer, Endjin “The task flow gives customers insights in which steps can be taken to realize a desired architecture and also what the possibilities are.” -Erwin de Kreuk, Principal Consultant Data & AI, InSpark Give it a try yourself To read more about task flows, check out the documentation. The task flow feature is still in its early stages, and we'd appreciate it if you could try it and let us know what you think. We can’t wait to see what you’ll build, and we’re always listening to your feedback. Help us prioritize what’s next by adding your ideas here: https://aka.ms/fabricideas
Alerting and acting on data from the Real-Time hub
Microsoft Fabric’s new Real-Time hub makes it easier than ever to discover, preview, and use streaming data from around your organization. But the data in these streams is only as useful as the actions and changes it can drive in your business. Data Activator provides a no-code experience for automatically taking actions when patterns or conditions are detected in changing data and is embedded around the Real-Time hub to make creating alerts accessible at all times. This can enable a host of scenarios such as: Monitor store sales and when they decline, alert the store managers. Alert retail managers to move food from failing grocery store freezers before it spoils. Retain customers who had a bad experience by tracking their journey through apps, websites etc. Help logistics operations find lost shipments proactively by starting an investigation workflow when package status isn’t updated for a certain length of time. Track data pipeline quality, either rerunning jobs or alerting when pipelines fail or anomalies are detected. You can find out more about the Real-Time hub in our blog post from earlier this week. Alerting from the Real-Time hub When you’re looking at the details page for streams or system events in the Real-Time hub, you will see an option to ‘Set alert’ This will open a side panel where you can create a Data Activator reflex that will monitor your events. You configure the events you want to monitor, the conditions you want to look for, and the action you want Data Activator to take. The Monitor section will be pre-filled with the stream you started from. The Condition area lets you choose what you want to look for in the data. You can choose to act on every event, compare values in the event to thresholds, or group the events for each business object they represent and track changes over time. Check out the Data Activator documentation for more details. Under Action you can choose how you want Data Activator to react once the conditions are met. Sending an email or Teams message will alert just you when those conditions are met. You can customize this later in the reflex item that is created. ‘Run a Fabric item’ allows you to run pipelines, notebooks and Spark Jobs. If there are other Fabric activities that you’d like to be able to start in response to events, please give us feedback at http://aka.ms/dataActivatorIdeas! Finally, you need to specify a Data Activator reflex item to save the alert in. Choose the workspace and item – this can be in the same workspace as the item hosting the stream or a different one. Alerting from Real-Time Dashboards Data Activator has supported alerting on Power BI reports for some time; you can now set up alerts from Real-Time Dashboards as well. From the … menu on Real-Time Dashboard tiles you can choose ‘Set alert’: This will open the same pane as before, connected to monitor the data in the dashboard tile. The data will be polled on a regular basis, pulling in the data visible in the tile and comparing to the condition you set. Anomaly detection triggers in Private Preview In the announcement of Real-Time Intelligence at Build, we also announced that we are bringing automated anomaly detection to Data Activator triggers. These will help find 'unknown unknowns' in your streaming and real-time data, going beyond basic rules to look for individual data points that lie outside the normal ranges. If you're interested in learning more, you can fill out https://aka.ms/anomalyDetectionPreview. We'll be accepting a small number of customers in the Private Preview phase based on region, use case etc. We'll reach out via email if we are able to enroll you in the preview and will let you know when it is more broadly available. Learn more, and help us with your feedback To find out more about Real-Time Intelligence, read Yitzhak Kesselman’s announcement. As we launch our preview, we’d love to hear what you think and how you’re using the product. The best way to get in touch with us is through our community forum or submit an idea. For detailed how-tos, tutorials and other resources, check out the documentation. This is part of a series of blog posts that dive into all the capabilities of Real-Time Intelligence. Stay tuned for more!Automating Fabric items with Real-Time Intelligence
Until today, you could schedule Fabric jobs to run data pipelines, notebooks and spark jobs to process data for example, loading data to OneLake or training ML models. But with the recently announced Real-Time Intelligence in Fabric, you can now make these items react in a more event-driven way. These operate in a similar way to triggers in Azure Data Factory pipelines, supporting the running of data pipelines in response to events from Azure Storage accounts. This can be used for scenarios such as: When new files are loaded to Azure storage accounts, start a data pipeline to load data to OneLake. Start a Fabric notebook when issues with data quality are found in Power BI reports. A number of events are now surfaced from the Fabric and Azure platforms in the Real-Time hub, meaning you can listen to changes from Azure Storage such as when a new file is created in a particular folder. You can find these by browsing to the ‘Fabric events’ pivot in the Real-Time hub. For more information about using the Real-Time hub and the Fabric events that you can find there please see our blog post. The 'Azure Blob Storage Events' category let you subscribe to changes in your Azure storage accounts, and 'Fabric workspace item events' are emitted when items are created, updated, read, or deleted in Fabric. In the details page for either set of events, you can choose the ‘Set alert’ option which opens a pane where you configure the alert. First you need to select the specific events and source you want to monitor. In the wizard that opens you can also scope to a particular workspace (for Fabric events) or specify filters such as a container or files of a particular type (for Azure events). At present the ‘Condition’ section only supports triggering on every event received. It will support checking for conditions in fields in the event when more sources are supported in future. We’d love your feedback on the conditions you want to look for in these events – please submit Ideas at https://aka.ms/rtiidea Finally, you can choose the ‘Run a Fabric item’ action and browse to the workspace and item that you want to run when the events are detected. You’ll also need to specify a workspace to save the reflex item that will manage the alert (this needs to be in a workspace on a Fabric capacity but doesn’t need to be in the same workspace as the item you are running). Triggering data pipelines with Azure storage events A very common scenario for data integration and ETL jobs is to invoke a data pipeline when a file arrives or is deleted in your Azure storage accounts. Now you can configure this directly from the data pipeline canvas. In the toolbar you’ll see a ‘Trigger (preview)’ button which will allow you to configure a storage trigger based on Reflex: That will open the pane with the current pipeline already selected as the item to run. When you select “Select events” you can connect to the appropriate storage account and select the events you want to listen for, and even filter to specific files or locations in your storage account. You’ll notice that the list of events from your storage account has many more options than ADF. In ADF, you can only configure blob created and blob deleted. Reflex also enables a large set of event options beyond Azure storage event sources that we’ll bring into data pipelines soon as well to make for even more exciting ETL and data integration scenarios. You will also have the option to filter based on the event subject which can allow you to listen for files of just certain names and types, similar to what ADF enables. When you’ve finished specifying the events, the trigger will be ready to run. Use the file and folder names in your pipeline The events created by Azure storage include information about the file that triggered the event, including the file type and the storage container it’s in. These are passed through to the pipeline as parameters that you can use, for example processing the specific file that started the pipeline. The ‘subject’ parameter contains the path to the file that you can use in your copy or processing tasks. Once your pipeline has been invoked from a storage event, you can now utilize the file name and folder name inside of your pipeline expression logic. Fabric Data Factory will automatically parse the event for you and store the file name and folder name in a new “Trigger parameters” areas of the expression builder. To make troubleshooting and testing easy, we honor the “?” syntax to allow your expressions to accept NULL values from these built-in parameters so you’re your pipelines will still execute even if no event has occurred or if the event did not supply that data: @pipeline()?.TriggerEvent?.FileName. Anomaly detection triggers in Private Preview In the announcement of Real-Time Intelligence at Build, we also announced that we are bringing automated anomaly detection to Data Activator triggers. These will help find 'unknown unknowns' in your streaming and real-time data, going beyond basic rules to look for individual data points that lie outside the normal ranges. If you're interested in learning more, you can fill out this form. We'll be accepting a small number of customers in the Private Preview phase based on region, use case etc. We'll reach out via email if we are able to enroll you in the preview and will let you know when it is more broadly available. Learn more and help us with your feedback To find out more about Real-Time Intelligence, read Yitzhak Kesselman’s announcement. As we launch our preview, we’d love to hear what you think and how you’re using the product. The best way to get in touch with us is through our community forum or submit an idea. For detailed how-tos, tutorials and other resources, check out the documentation. This is part of a series of blog posts that dive into all the capabilities of Real-Time Intelligence. Stay tuned for more!
Announcing Updates to Data Activator in Public Preview
Share Your Feedback We are committed to continuously improving Real-Time Intelligence and Data Activator. We'd love to hear what you think and how you’re using the product. The best way to get in touch with us is through our community forum or submit an idea. For detailed how-tos, tutorials and other resources, check out the documentation What’s Next? We’ll continue to make improvements as we approach General Availability of Data Activator. As we continue to make exciting, new updates, we will be making changes to the schema of reflex items to support application lifecycle management to help you manage the process of making your reflexes production ready. We also have some updates on Continuous Integration/Continuous Deployment to help with options to programmatically create rules in reflexes. With these upcoming updates, if you have existing reflex artifacts, please note they may require a migration process, and there may be some data loss. What is Data Activator? Data Activator, part of Real-Time Intelligence, facilitates the generation of automatic alerts and actions from your Fabric data. It is designed with a user-friendly visual interface that requires no technical expertise, making it accessible to anyone within your organization. Data Activator works over your streaming data in Eventstreams and Eventhouses, and Data Activator alerts can be seamlessly created directly from your Power BI reports, allowing business users to engage with it through the reports they are already utilizing. It aims to streamline data workflows, automate processes, and provide actionable insights that drive business growth but at the granular level of individual objects. How does Data Activator work? Monitors your Fabric data, including Power BI report visuals and real-time Eventstreams. Detects actionable conditions with a rule creation experience that allows you to set anything from simple thresholds to complex patterns, without coding. Acts automatically when conditions are met by sending alerts via email or Microsoft Teams, and activating Power Automate flows for various third-party systems and applications. What has changed since the Public Preview announcement in October? In October, we announced the public preview of Data Activator. In May, we announced Real-Time Intelligence bringing together Synapse Real-Time Analytics and Data Activator. Thank you all for trying it and giving us feedback. Based on your comments and feedback, we’ve made a number of updates to streamline the user experience. Data Activator's previous no code rule designer Data Activator's updated no code rule designer Simplified Navigation and Explorer Pane To provide a cleaner, simpler, and more organized interface, we have introduced unified the Data and Design tabs. This allows you to see your data while you build your objects and create rules on top of your data all in one view. Improved Rule Authoring Experience We’ve introduced a new definition pane that offers a view of what has been set in your rule, making it easier to fill out and navigate the controls available to fine tune your rules. This enhancement simplifies the process of editing and reviewing rule definitions. Event-Based Rules In addition to setting rules on properties, you can now set rules directly on event streams. This allows you to get alerts whenever a new event comes in on a stream. Actions Preview The actions preview feature allows you to get a preview of what a message will look like to your alert receivers. This will help you ensure that your actions look exactly as expected and that your alert receivers will receive all the associated information needed. Sample reflex We’ve made some updates to the sample reflex, so you can explore and discover the possibilities with Data Activator. In it, you’ll find a new tour that will walk you through the main modeling concepts in Data Activator, see new views that will help you discover more from your data, and show you a variety of rules you can create with added condition expressivity. Connect to data in a new reflex Sample reflex welcome screen Simplified Explorer Pane from Embedded Rule Creation Experiences Azure Data Lake Storage customers can create business rules when Fabric events occur, and PBI users can create rules on measures they want to monitor. We’ve improved the experience to help provide a seamless transition into Data Activator. ADLS When kicking off a data pipeline to set an alert on blob storage events when they occur, Data Activator previously created separate events, created an object, extracted the subject and type and turned them into properties. User feedback indicated it was hard to tell why the object was created, what event each subject and type property belonged to, and a rule had been created for each event. In the improved version, you see a simplified view that is easy to understand. To ensure the alert is easy to find and manage, the rule created in the embedded experience is captured and denoted in a group. All the storage events are simplified into one line, and a single rule is created on that event. Power BI Additional feedback indicated it was difficult to understand how creating a rule on a measure in a PBI visual translated over to Data Activator concepts. In the previous version of Data Activator, when multiple rules are created and added to the same reflex, all the entities are mixed in, and it’s difficult to navigate. In the new version of Data Activator, when each rule is created from an embedded rule creation experience, a group has been created and has a consistent structure. Each rule that has been created is kept in a separate, easy to read structure. Share Your Feedback We are committed to continuously improving Real-Time Intelligence and Data Activator. We'd love to hear what you think and how you’re using the product. The best way to get in touch with us is through our community forum or submit an idea. For detailed how-tos, tutorials and other resources, check out the documentation What’s Next? We’ll continue to make improvements as we approach General Availability of Data Activator. As we continue to make exciting, new updates, we will be making changes to the schema of reflex items to support application lifecycle management to help you manage the process of making your reflexes production ready. We also have some updates on Continuous Integration/Continuous Deployment to help with options to programmatically create rules in reflexes. With these upcoming updates, if you have existing reflex artifacts, please note they may require a migration process, and there may be some data loss.
AI Ready Apps: build RAG Data pipeline from Azure Blob Storage to SQL Database in Microsoft Fabric within minutes
Microsoft Fabric is a unified, secure, and user-friendly data platform equipped with features necessary for developing enterprise-grade applications with minimal or no coding required. Last year, the platform was enhanced by introducing SQL Database in Fabric, facilitating AI application development within Microsoft Fabric. In a previous blog post, we discussed how to build a chatbot using SQL Database in Fabric. This blog post will provide step-by-step instructions for creating a Retrieval Augmented Generation (RAG) pipeline to prepare your data for AI integration using SQL Database in Microsoft Fabric. For this blog post, we will consider a use case involving Contoso Recruiting Agency, which aims to make more sense of the stored candidates' resumes by creating a searchable database of candidates to match specific skill requirements. RAG pipeline activity workflow Each recruiter uploads the resume (in PDF format) to their designated folder in Azure Blob storage. Whenever a file is uploaded, the RAG pipeline is triggered. The pipeline processes the file by extracting the text content, chunking the text, redacting any Personally Identifiable Information (PII), generating embeddings, and finally storing the embeddings in a SQL database within Fabric as a vector store. These stored embeddings can subsequently be utilized to develop AI applications such as smart search, recommendation engines, chatbots, and other tools. Prerequisite This blog post requires users to bring their own key (BYOK) for AI services, which also means creating these services outside of the Microsoft Fabric platform. Download Git and clone the rag-pipeline repository. Azure subscription: Create a free account. Microsoft Fabric subscription: Create a free trial account. Azure OpenAI access: Apply for access in the desired Azure subscription as needed. Azure OpenAI resource: Deploy an embedding model (e.g. text-emebdding-3-small). Azure AI multi-service resource, specifically, we will be using Document Intelligence and Azure Language Services from this resource. Azure Portal: Create a storage account and assign storage blob data contributor role. Optionally, download Azure Storage Explorer to manage the storage account from your desktop. Optionally, download Visual Studio Code for free and install Azure Functions Core Tools if you plan to edit User Data Functions using Visual Studio Code. Dataset Considering the file formats supported by the Document Intelligence Service we utilize the PDF files from Resume Dataset from Kaggle. Steps Create a workspace named “IntelligentApp”. Create a Lakehouse named “blob_filestorage”. Create SQL Database in Fabric named “datamart”. Navigate to the workspace “IntelligentApp”, click Import - Notebook - From this computer and then import the notebook “cp_azblob_lakehouse.ipynb” from the cloned repository’s notebook folder. Import a Notebook 5. Attach the Lakehouse "blob_filestorage" to "cp_azblob_lakehouse" Notebook. Open the notebook, on the explorer click 'Add sources'. Select, 'Existing data sources'. Select, “blob_filestorage” from OneLake catalog and then click 'Connect'. Attach a Lakehouse to the Notebook 6. Create User Data Function, Navigate to the “IntelligentApp” workspace and click "+New item" Search for “Function” and Select “User data function(s)” Provide the name "file_processor". Click "New function". Add Lakehouse and SQL Database as managed connection(s). On the Functions Explorer screen click "Manage Connections" and then select "+Add Data connection". From the “OneLake catalog” select datamart (SQL database) and then click “Connect”. Repeat the previous step to add blob_filestorage (Lakehouse) as a managed connection. Click "Library management", add the following dependencies (Click “+ Add from PyPI” to add the dependencies). The dependencies are also listed in “/functions/requirements.txt” file of the cloned repository. Ensure you are using fabric-user-data-functions version 0.2.28rc0 or higher. Add User Data Function dependencies In the function editor, replace existing content with the contents of "function\function_app.py" from the cloned repository. Click “Publish” (on the top right) to deploy the function. After the functions are deployed, click “Refresh”. 7. Create a Data pipeline by navigating to the workspace and then clicking on “+ New Item” Search and select “Data pipeline”. Provide the name “blob_ingest_pipeline”. 8. Create a data pipeline storage trigger by clicking “Add trigger (preview)” button and provide the following configuration: Source: Select “Azure Blob Storage events” Storage account: “Connect to existing Azure Blob Storage account” Subscription: Select your Azure subscription Azure Blob Storage account: Select the blob storage account under your subscription. Eventstream name: blob_ingest_stream Create Event stream Click “Next”, to configure the event type and source Event Type(s): Select only the Microsoft.Storage.BlobCreated event. This will ensure that an event is generated each time a new blob object is uploaded. Click "Next" to review the configuration. Then, click "Connect" to connect to the blob storage. A successful connection will be indicated by the status "Successful". Finally, click "Save". On the Set alert screen, under Save location, configure the following settings. Select, “Create a new item”. New item name: blob_activator. Click “Create” to create and save the alert. Create Fabric Activator Now that we have setup the stream, it's time to define the “blob_ingest_pipeline”. Pipeline definition Pipeline can be defined in two ways as outlined below, Import template Templates offer a quick way to begin building data pipelines. Importing a template brings in all required activities for orchestrating a pipeline. To import a template: Navigate to the Home menu of the data pipeline. Click, “Use a template” From the “Pipeline templates” page click “Import template” Import the file “template/ AI-Develop RAG pipeline using SQL database in Fabric.zip” from the cloned repository. Import Data pipeline from template The imported data pipeline is preloaded with all necessary activities, variables, and connectors required for end-to-end orchestration. Important!!! If you are using a template, there is no need to manually add a variable or an activity. Instead, you can proceed directly to configuring values for the variables and each activity parameter in the pipeline, as detailed in the Blank Canvas section. Blank Canvas 1. Establish pipeline variables: Click on the pipeline canvas, select the “Variables” menu, and then click “+ New” to add and configure values for the following variables: Configure pipeline variables Name Type Value Comment fileName String @pipeline()?.TriggerEvent?.FileName container String @pipeline()?.TriggerEvent?.FolderPath source String @pipeline()?.TriggerEvent?.Source cognitiveServiceEndpoint String <<YOUR-MULTI-SERVICE-ACCOUNT-ENDPOINT>> Replace<< YOUR-MULTI-SERVICE-ACCOUNT-ENDPOINT>> with the cognitive Service Endpoint. For example, https://myservices.cognitiveservices.azure.com/ apiKey String <<YOUR-MULTI-SERVICE-ACCOUNT-APIKEY>> Replace <<YOUR-MULTI-SERVICE-ACCOUNT-APIKEY with the apikey of your multi-service account openAIEndpoint String <<YOUR-OPENAI-SERVICE-ENDPOINT >> Replace <<YOUR-OPENAI-SERVICE-ENDPOINT>> with the endpoint of your Azure OpenAI Account. For example, https://myopenaiservice.openai.azure.com/ openAIKey String <<YOUR-OPENAI-APIKEY>> Replace <<YOUR-OPENAI-APIKE>> with the apikey of your Azure OpenAI Account embeddingModel String text-embedding-3-small recepientEmailAddress String <<to-email-address>> recipients email address senderEmailAddress String <<from-email-address>> sender's email address Get multi-services account endpoint 2. Add a "Notebook" activity. The notebook associated with this activity utilizes NotebookUtils to manage file system. During the execution of the notebook, a folder corresponding to the container name will be created if it does not exist. Subsequently, the file will be copied from Azure Blob Storage to the Lakehouse folder. Configure this activity as outlined below: General tab, Name: azureblob_to_lakehouse Settings tab, Notebook: cp_azblob_lakehouse Base parameters Click “+New” to add the following parameters, Name: fileName Type: String Value: @variables('fileName') Name: container Type: String Value: @variables('container') Name: source Type: String Value: @variables('source') Use On Success connector of the activity to link to the subsequent function (Extract Text) activity. 3. Add a “Functions activity”. The function “extract_text” associated with this activity uses Azure AI Document Intelligence service to extract the "text" content from the file copied into the Lakehouse by the previous activity. Configure this activity as outlined below: General tab, Name: Extract Text Settings tab, Type: Fabric user data functions Connection: Sign-in (if not already) using your workspace credentials. Workspace: IntelligentApp (default selected) User data functions: file_processor Function: extract_text Parameters, Name: filePath Type: str Value: @activity('azureblob_to_lakehouse').output.result.exitValue Name: cognitiveServicesEndpoint Type: str Value: @variables('cognitiveServiceEndpoint') Name: apiKey Type: str Value: @variables('apiKey') Use On Completion connector of the activity to link to the subsequent If Condition (Text Extraction Results) activity. 4. Add an “If Conditions” activity to verify the success of the text extraction in the previous step. If the extraction was unsuccessful, an email would be sent to the configured recipient, and the pipeline would be terminated. Configure this activity as outlined below, General tab, Name: Text Extraction Results Activities tab, Expression: @empty(activity('Extract Text').error) Case: False, edit the “false” condition using the edit (pencil) icon, and add the following activities, “Office 365 Outlook” activity: To send alert emails. General Tab, Name: Text Extraction Failure Email Alert Settings Tab, Signed in as: Sign- in (if not already) using the credentials as those of your workspace. To: @variables('recepientEmailAddress') Subject: Text Extraction Error Body: <p>@{replace(string(activity('Extract Text').error.message), '\','')}</p> Advanced, From: @variables('senderEmailAddress') Importance: High Use On Success connector of the activity to link to the subsequent Fail activity. “Fail” activity: To terminate the pipeline General tab, Name: Text Extraction Process Failure Settings tab, Fail message: @{replace(string(activity('Extract Text').error), '\','')} Error code: @{activity('Extract Text').statuscode} Return to the main canvas by clicking the pipeline name (blob_ingest_pipeline) and use the On Success connector of the “If Condition” activity to link to the subsequent Function (generate chunks) activity. 5. Add a “Function” activity. The function “chunk_text” associated with this activity uses the tiktoken tokenizer to “generate chunks” for the text extracted by the previous activity. Configure this activity as outlined below: General tab, Name: Generate Chunks Settings tab, Type: Fabric user data functions Connection: If not already, sign in using the credentials that have complete access to your workspace. Workspace: IntelligentApp (default selected) User data functions: file_processor Function: chunk_text Parameters, Name: text Type: str Value: @activity('Extract Text').output.output Name: maxToken Type: int Value: 500 Name: encoding Type: str Value: cl100k_base Use On Success connector of the activity to link to the subsequent Function (Redact PII Data) activity. 6. Add a “Function” activity. The function “redact_text” associated with this activity uses the Azure AI Language service to “Redact PII Data” for the chunks generated by the preceding activity. The chunking of text is done prior to redaction to comply with the service limits requirements for the PII detection feature. Configure this activity as outlined below: General tab, Name: Redact PII Data Settings tab, Type: Fabric user data functions Connection: Sign-in (if not already) using the credentials that have complete access to your workspace. Workspace: IntelligentApp (default selected) User data functions: file_processor Function: redact_text Parameters, Name: text Type: list Value: @activity('Generate Chunks').output.output Name: cognitiveServicesEndpoint Type: str Value: @variables('cognitiveServiceEndpoint') Name: apiKey Type: str Value: @variables('apiKey') Configure a User Defined Function Use On Completion connector of the activity to link to the subsequent If Condition (PII Redaction Results) activity. 7. Add an “If Conditions” activity to verify the success of the PII redaction in the previous step. If the redaction was unsuccessful, an email would be sent to the configured recipient, and the pipeline would be terminated. Configure this activity as outlined below, General tab, Name: PII Reaction Results Activities tab, Expression: @empty(activity('Redact PII Data').error) Case: False, edit the “false” condition using the pencil icon, and add the following activities: “Office 365 Outlook” activity: To send alert emails. General tab, Name: Redaction Failure Email Alert Settings tab, Signed in as: Sign-in (if not already) using the credentials as those of your workspace. To: @variables('recepientEmailAddress') Subject: Text Redaction Error Body: <p>@{replace(string(activity('Redact PII Data').error.message), '\','')} </p> Advanced, From: @variables('senderEmailAddress') Importance: High Use On Success connector of the activity to link to the subsequent Fail activity. “Fail” activity: To terminate the pipeline General tab, Name: Text Redaction Process Failure Settings tab, Fail message: @{replace(string(activity('Redact PII Data').error), '\','')} Error code: @{activity('Redact PII Data').statuscode} Return to the main canvas and use the On Success connector of the “If Condition” activity to link to the subsequent Function (Generate Embeddings) activity. 8. Add a “Function” activity. The function “generate_embeddings” associated with this activity uses Azure Open AI Service embedding model to convert the redacted chunks into embeddings. Configure this activity as outlined below: General tab, Name: Generate Embeddings Settings tab, Type: Fabric user data functions Connection: Sign-in (if not already) using the credentials that have complete access to your workspace. Workspace: IntelligentApp (default selected) User data functions: file_processor Function: generate_embeddings Parameters, Name: text Type: list Value: @activity('Redact PII Data').output.output Name: openAIServiceEndpoint Type: str Value: @variables('openAIEndpoint') Name: embeddingModel Type: str Value: @variables('embeddingModel') Name: openAIKey Type: str Value: @variables('openAIKey') Name: fileName Type: str Value: @variables('fileName') Use On Completion connector of the activity to link to the subsequent If Condition (Generate Embeddings Results) activity. 9. Add an “If Conditions” activity to verify the success of the Generate Embeddings in the previous step. If the embeddings generation were unsuccessful, an email would be sent to the configured recipient, and the pipeline would be terminated. Configure this activity as outlined below: General tab, Name: Generate Embeddings Results Activities tab, Expression: @empty(activity('Generate Embeddings').error) Case: False, edit the “false” condition using the pencil icon, and add the following activities: “Office 365 Outlook” activity: To send alert emails. General tab, Name: Generate Embeddings Failure Email Alert Settings tab, Signed in as: Sign-in (if not already) using the credentials as those of your workspace. To: @variables('recepientEmailAddress') Subject: Generate Embeddings Error Body: <p>@{replace(string(activity('Generate Embeddings').error.message)</p> Advanced, From: @variables('senderEmailAddress') Importance: High Use On Success connector of the activity to link to the subsequent Fail activity. “Fail” activity: To terminate the pipeline General tab, Name: Generate Embeddings Processing Failure Settings tab, Fail message: @{replace(string(activity('Generate Embeddings').error), '\','')} Error code: @{activity('Generate Embeddings').statuscode} Return to the main canvas and use On Success connector of the “If Condition” activity to link to the subsequent Function (Create Database Objects) activity. 10. Add a “Function” activity. The function “create_table” associated with this activity executes a SQL command to create a documents table within the previously created “datamart”, SQL database. Configure this activity as outlined below: General tab, Name: Create Database Objects Settings tab, Type: Fabric user data functions Connection: Sign-in (if not already) using the credentials that have complete access to your workspace. Workspace: IntelligentApp (default selected) User data functions: file_processor Function: create_table Use On Success connector of the activity to link to the subsequent Function (Save Data) activity. 11. Add a “Function” activity. The function “insert_data” associated with this activity executes a SQL command to bulk insert rows in the documents table created in the previous activity. Configure this activity as outlined below: General tab, Name: Save Data Settings tab, Type: Fabric user data functions Connection: Sign-in (if not already) using the credentials that have complete access to your workspace. Workspace: IntelligentApp (default selected) User data functions: file_processor Function: insert_data Parameters, Name: data Type: list Value: @activity('Generate Embeddings').output.output Troubleshooting When adding a Python library from PyPI to User Data Functions, you might notice an error, such as a wiggly line under the library name (e.g., "azure-ai-textanalytics"), like a spelling mistake. Users should ensure the library name is spelled correctly and then ignore the error by tabbing out to the Version dropdown and selecting the correct version, this transient error should resolve itself. The imported pipeline reportedly doesn't seem to preload with the parameter values. For each activity in the pipeline, ensure that the parameter values are provided and correct. Refer to the Blank Canvas section for the required parameters and their values. Execute pipeline (pipeline in action) It’s now time to put everything we have done so far into perspective and see the pipeline in action. Upload a PDF file, Use the Azure Storage Explorer or alternatively Azure Portal and create a Blob container named “resume”. Upload a PDF file from the Kaggle dataset. Create a container and upload a PDF file Review pipeline execution From the pipeline’s “Run” menu, select “View run history” and select the recent pipeline run. In the details view, check to see if the status is “Succeeded”. In case of a Failure, try to “Rerun” the pipeline using the rerun option. Pipeline activity execution Review Lakehouse A folder with the same name as that of the container (resume) is created. The PDF file is copied from Azure Blob Storage to the Lakehouse files. PDF file copied to Lakehouse Review database The document table should be automatically created by the pipeline. Redacted chunk data and the embeddings stored in the documents table. Save data in SQL Database in Fabric Conclusion We hope this blog post has provided useful insights for preparing your data for AI applications. Your feedback is important, and we look forward to seeing how you use this information to develop innovative AI solutions. Would also like to express our sincere gratitude to Aaron Saidi for the invaluable contribution towards reviewing this blog post. Thank you, Aaron, for your support. Submit your feedback on Fabric Ideas and join the conversation on the Fabric Community.Microsoft Fabric Real-Time Intelligence: A Leader in the 2025 Forrester Streaming Data Wave
Businesses and organizations are entering a new operational era defined by immediacy, intelligence, and continuous adaptation. AI is shifting expectations across every industry. Organizations now need to sense what is happening across their business the moment it occurs, understand its significance, and respond with confidence. Real-time data has become the foundation for how resilient, competitive organizations run. Enterprises also realize that fragmented data stacks cannot support modern AI or operational agility. Microsoft anticipated this shift early on. We invested heavily in real-time services in Azure, including Event Hubs, Stream Analytics, and Data Explorer, powering mission-critical real-time workloads for years for both Microsoft and our customers and delivering proven reliability, performance, and planet scale. But the decisive step was building Real-Time Intelligence into Microsoft Fabric on top of that mature foundation in Azure. Real-Time Intelligence unifies streaming, analytics, and action in one governed platform, bringing batch and streaming together in OneLake and Fabric. Microsoft has been recognized as a Leader in The Forrester Wave™: Streaming Data Platforms, Q4 2025, which we view as a strong validation of our strategy and execution. This position as a leader is the result of our long-term conviction rather than short-term reaction. Microsoft invested early so organizations would have a mature, scalable real-time foundation exactly when the need became urgent, and that foresight is now paying off for customers. Forrester Wave Streaming Data Platforms, Q4 2025 A Leader for the Real-Time Enterprise Forrester’s 2025 evaluation confirms a clear market shift: enterprises are moving away from fragmented real-time architectures and toward unified platforms that can support AI-driven decisions at digital speed. As Forrester notes, “AI agents rely on seamless data flow across ingestion, transformation, and real-time insights to avoid bottlenecks and cascading errors. A robust platform unifies these workloads (messaging, processing, analytics), eliminating silos and latency that degrade decision quality.” In this environment, a fully integrated streaming platform has become essential rather than optional. This is exactly where Real-Time Intelligence in Microsoft Fabric stands out. Forrester notes that Microsoft’s strategy is to bring dozens of services together under a single umbrella, making real-time development and event-driven analytics “second nature” within Fabric. Real-time data becomes a first-class citizen in Fabric’s unified data estate. Streaming signals land directly into OneLake, the same foundation that powers the lakehouse, warehouse, semantic models, governance, Power BI, and AI agents. Forrester’s assessment is clear: “Microsoft excels at messaging, analytics, governance, developer experience, business user experience, and more, enabling robust performance for real-time analytics and event-driven applications. It provides seamless integration within the Fabric ecosystem to support enterprise use cases like predictive analytics and operational dashboards. It offers strong tooling for both technical and business users as well as deep integration with Azure services, empowering enterprises to build real-time solutions.” This recognition reflects a platform designed from the start to work as one coherent system, not a set of loosely assembled services. It is an end-to-end real-time platform that strengthens the entire data estate and positions organizations to run AI-driven operations with clarity, speed, and confidence. Why Enterprises Are Standardizing on Fabric Real-Time Intelligence Real-Time Intelligence delivers a complete end-to-end platform for understanding and acting on what is happening across the enterprise in the moment. It unifies signals across time, space, and relationships to provide a connected operational picture rather than isolated dashboards or fragmented telemetry. Every stage of the real-time lifecycle (streaming, analyzing, modeling, visualizing, and acting) is integrated into one governed, AI-ready system. This coherence is what enables teams and AI agents to work from one live, trusted view of the business and make high-quality decisions at digital speed. https://www.youtube.com/watch?v=bfXn21tqsAQ&t=2s The Real-Time Hub is the unified experience that makes every enterprise signal visible, governed, and ready for use. It brings all the capabilities of Real-Time Intelligence together into a coherent operational fabric, the live nervous system of the modern enterprise. Across five major areas, these capabilities form a complete real-time platform: Stream: Eventstream ingests, shapes, filters, and enriches data in motion, while Connectors pulls data from dozens of streaming sources, including Kafka, MQTT, IoT systems, SaaS apps, and CDC feeds. Event Schema Set standardizes events to keep signals consistent, interoperable, and easy to govern. Analyze: Real-time and historical insights converge in Eventhouse, a high-performance engine for interactive analytics over petabyte-scale data. Anomaly Detector highlights deviations and emerging risks the moment they appear. Model: Fabric’s modeling layer enables unified operational awareness. Graph links signals to entities and relationships, Fabric Map situates them in physical space, and Digital Twin Builder models assets and environments over time. Visualize: KQL Querysets enable fast, interactive analytics for exploring data and diagnosing problems, Graph Querysets for relational and causal patterns across the business, and Real-Time Dashboards for intuitive, no-code views of live conditions and trends. Act: Activator detects patterns over time on a per-instance basis and triggers alerts or workflows through visual no-code business rules. Operations Agent monitors your operations, reasons about your business, and takes automated actions based on natural language instructions. Fabric Real-Time Intelligence Components across Stream | Analyze | Model | Visualize | Act How Fabric IQ Completes the Intelligence Layer We recently introduced Fabric IQ (see blog post), the intelligence layer that transforms unified data into unified understanding for every team and every AI agent. IQ brings a semantic, reasoning-ready foundation to the entire Fabric platform, including Real-Time Intelligence, so organizations can interpret what is happening across their business, not just observe it. It augments human and AI workflows with natural language understanding, contextual reasoning, and a unified live view drawn from both streaming and historical data. IQ enables natural-language exploration across all your data in Fabric, allowing users to ask questions, investigate anomalies, and understand relationships without writing code. It synthesizes patterns across time, space, and relationships, surfaces anomalies, explains correlations, and identifies root causes. This shift is what elevates Fabric from a data platform to an enterprise intelligence platform, one where insights are generated, connected, and immediately actionable. https://www.youtube.com/watch?v=RjU0slwcZGs&t=3s IQ amplifies the power of Real-Time Intelligence. Streaming events stop being isolated signals and become part of a live coherent semantic picture: operators can ask why something happened, analysts can explore emerging risks and opportunities, and AI agents can reason over business meaning before taking action. Because IQ relies on the same governance, semantics, and security as the rest of Fabric, the insights it produces are consistent, trustworthy, and grounded in the organization’s shared data estate. Together, Real-Time Intelligence and IQ create a unified real-time decision system. Real-Time Intelligence senses what is happening in the moment; IQ interprets it and drives the next action. This seamless integration enables enterprises and their AI agents to operate with precision even as conditions change second by second. Fabric components across IQ - Act | Decide | Observe | Analyze Strategic Takeaways for Enterprise Leaders Real-time intelligence is becoming core to modern enterprise operations, no longer a specialist capability or an add-on component. AI requires up-to-date, contextualized data. Decision systems must unify batch and streaming data. And governance and semantics must extend across the entire data estate, from historical tables to real-time streams, to ensure trust, lineage, compliance, and safe AI behavior. Microsoft Fabric, with Real-Time Intelligence and IQ, offers a decisive foundation for this new era. It brings together data, meaning, and action in one governed platform. It unifies time, space, and relationships to give enterprises a complete operational picture. And it supports AI-driven decisions at digital speed while strengthening governance, reliability, and trust. As organizations modernize their data platforms and adopt AI at scale, Fabric provides the clarity, coherence, and confidence they need to run their business in real time and to thrive in the decade ahead. Statement from Forrester Forrester does not endorse any company, product, brand, or service included in its research publications and does not advise any person to select the products or services of any company or brand based on the ratings included in such publications. Information is based on the best available resources. Opinions reflect judgment at the time and are subject to change. For more information, read about Forrest's objectivity here. Learn More Presentations BRK222: Create a semantic foundation for your AI agents in Microsoft Fabric BRK226: Unlock the power of Real-Time Intelligence in the era of AI Conferences Join us at FabCon in Atlanta, Georgia, on March 16-20, 2026. Register and use code MSCATL for a $200 discount on top of the current Early Access pricing! Resources Explore the docs Find customer success stories Follow a tutorial Try an in-depth tutorial Complete the learning path Get certified Sign up for one-day trainingWorkspace Outbound Access Protection (OAP) for Real-Time Intelligence (Preview)
Workspace Outbound Access Protection (Preview) for key Real-Time Intelligence experiences in Microsoft Fabric, helping organizations reduce data exfiltration risk while keeping common real-time ingestion, analytics, dashboarding, and automation workflows running. Read on to learn which RTI items are supported today and how this preview helps admins apply stronger workspace-level security controls across real-time solutions.
