Fabric Runtime 2.0 (Generally Available)
Fabric Runtime 2.0 (Generally Available), marks a major milestone for Microsoft Fabric. As the execution foundation for Microsoft Fabric's Data Engineering and Data Science experiences, Runtime 2.0 delivers a modern, high-performance platform built on Apache Spark and deeply integrated across the Fabric ecosystem.Customer-managed key encryption now extends to Spark jobs in Microsoft Fabric
Security and compliance teams increasingly expect full control over the encryption keys that protect their data—not only while it sits in storage, but while it is actively being processed. In Microsoft Fabric, Spark powers some of the most data-intensive workloads, from large-scale data engineering pipelines to data science and machine learning. As these jobs run, they read, write, and temporarily stage data on the compute cluster. Customer-managed key support has been extended to that processing layer for Spark jobs.Job concurrency and queue monitoring for Spark, now with a capacity-level view (Generally Available)
Spark jobs don't always start immediately. When jobs are queued or throttled, it can be difficult to determine if the cause is workspace limits, capacity saturation, or contention from other workspaces sharing the same capacity. Job concurrency and queue monitoring for data engineering are now generally available, and it includes a new capacity-level view that shows exactly how each workspace consumes your capacity over time.Native Execution Engine now accelerates CSV workloads in Microsoft Fabric Spark
Accelerate CSV Workloads with Native Execution Engine in Microsoft Fabric Spark Comma-separated values (CSV) remain one of the most widely used file formats in data engineering. Whether ingesting log files, processing exports from legacy systems, or loading flat-file datasets, CSV workloads are a daily reality for Fabric Spark users. The Native Execution Engine in Microsoft Fabric Spark now natively accelerates CSV file reads, delivering up to 2× faster performance on benchmark workloads with no code changes required. New Native CSV Reader in Execution Engine Previously, when the Native Execution Engine processed CSV files, reads fell back to Spark's default CSV reader. As a result, CSV workloads didn't benefit from the vectorized columnar execution that accelerates Parquet and Delta workloads. The latest Native Execution Engine update includes a purpose-built CSV reader powered by the Velox engine with SIMD-optimized parsing. CSV file reads now execute directly within the vectorized pipeline, eliminating row-to-columnar conversion overhead and delivering significant performance improvements. What Performance Can You Expect? Benchmark Without NEE (Baseline) With NEE 20 GB CSV read/write workload 56 seconds 34–38 seconds (~1.5–1.65× faster) TPC-DS benchmark suite (CSV) Baseline Up to 2× faster TPC-H benchmark (SIMD optimized) Baseline ~35% improvement TPC-DS end-to-end Baseline ~20% improvement Note: Performance gains vary depending on workload characteristics, schema complexity, and data distribution. CSV workloads with larger datasets and simpler schemas typically see the largest improvements. Impact for Data Engineering Workflows Faster ETL Pipelines - CSV ingestion stages that previously bottlenecked pipelines can now complete more quickly, accelerating end-to-end data processing. Lower Compute Costs - Faster execution means workloads consume Fabric capacity for less time, potentially reducing compute costs. No Code Changes Required - If Native Execution Engine is enabled, CSV acceleration happens automatically. Existing notebooks, pipelines, and Spark SQL queries benefit immediately. Consistent Acceleration Across Formats - CSV joins involving Parquet, Delta, and other natively accelerated formats benefit from a unified performance model. How It Works Velox CSV Parser with SIMD Optimizations The parser leverages CPU-level vector instructions to process multiple bytes simultaneously, accelerating: Field parsing Delimiter detection Type conversion Mison-Based Structural Indexing For compatible workloads, the engine uses structural indexing to identify field boundaries in a single pass before parsing. This reduces CPU utilization and improves performance on large files. Together, these optimizations keep CSV data flowing through the vectorized columnar pipeline without falling back to Spark's row-based reader, preserving the performance characteristics of Native Execution Engine. Supported CSV Options The native CSV reader supports most commonly used Spark CSV options, including: Custom delimiters Custom quote characters Header inference Explicit schema specification Null value handling Multi-line records Encoding configuration Escape character configuration For a complete list of supported options and known limitations, see the Native Execution Engine documentation. Getting Started CSV acceleration is available automatically whenever Native Execution Engine is enabled. No additional configuration is required. spark.conf.set("spark.native.enabled", "true") You can also enable Native Execution Engine at the environment level for all Spark sessions. For setup instructions, see Enable Native Execution Engine. Prerequisites Microsoft Fabric workspace with Spark enabled Fabric Runtime 3.5 or later Native Execution Engine enabled at the environment or session level Next Steps Learn more about the Native Execution Engine Follow the step-by-step guide to Enable Native Execution Engine Explore supported data types and file formats in the Native Execution Engine documentation. Feedback Share your experience in the Fabric Community Forums Ask questions on Microsoft Q&A
Microsoft Fabric July 2023 Update
Welcome to the July 2023 update. We have features in Core, Synapse, Data Factory, Data Activator, Community, and Power BI. Contents Core Help Pane Monitoring Hub improvements Admin Microsoft Public Preview Default Sharing On by Default Effective July 5 th , 2023 OneLake OneLake file explorer update with support for switching organizational accounts Item sharing Synapse Data Warehouse Automatic updates for statistics UPDATE FROM and DELETE FROM T-SQL Optimization in data movements during query execution Dataflows Gen2 from the Warehouse Zero Copy Table Clones Data warehouse sharing Dbt Support for Fabric Data Warehouse Data Engineering Load to Table Feature Updates Lakehouse sharing and permission management Notebook Resources Notebook: Show cell output after session reconnection Notebook: Sharing and permission management Notebook: New status bar Data Science 1 Microsoft Fabric and SynapseML Offer New Features for Large Language Models Real-time Analytics KQL Database Create a Continuous Data Connection from Event Hubs to KQL Database Query KQL Database from MATLAB Data Factory Dataflows Gen2 Power Query Editor Enhancements to output destination experience Improvements to existing experiences Connectors Google Analytics (Connector Update) Oracle Database (Connector Update) Azure Databricks, Databricks (Connector Update) Denodo (Connector Update) EQuIS (Connector Update) Snowflake (Connector Update) Anaplan (Connector Update) Gateway and Connections SSO option in cloud connections Flag in cloud connections to allow them to be with gateway Data pipelines Activities Teams Activity Productivity Artifact parameterization support Performance improvement for moving parquet files to Lakehouse tables “Save as” support now available from your workspace Support column mapping for Lakehouse destination in Copy activity Data Activator Sample data Adding multiple streams or properties Designing triggers and alerts Trigger lifecycle Testing actions Teams app Community Implementing a Lakehouse with Microsoft Fabric Courses Filter by product for User Groups & Events Data Activator Community now live Power BI Reporting Smoothed and Leader lines – Updates On-Object Interaction (Preview) – Updates Customize the Pane Switcher CTRL click to open multiple panes in the Pane Switcher Treemap sub-selections now supported: Noteworthy bug fixes Modeling Edit your data model in the Power BI Service– Updates (Preview) Relationship validation Service Dataset details page revamp Mobile Datasets are coming to Power BI Mobile apps! Developers Boost Your Custom Visuals Performance: Check out our Latest Article! Visualizations New visuals in AppSource Elevate Your Map Charts With Drill Down Map PRO We are ready to present our new visual: an "all-in-one" Multi Target KPI card Other WebView2 GA Core Help Pane Help pane is feature-aware and displays articles about the actions and features available on the current Fabric screen. It is also a search pane that quickly finds answers to questions in the Fabric documentation and community. Open the Help pane From the upper-right corner of the Fabric screen, select the ? Icon. Help pane is feature-aware The feature-aware state is the default view of the help pane when you open it without entering any search terms. It shows a list of recommended topics, resources that are relevant to your current context and location in Fabric, and a list of links for other resources. It has three sections: Feature-aware documents: This section groups the documents by the features that are available on the current screen. As you explore Fabric, the feature-aware documents update based on what you've selected and where you are in Fabric. This is a great way to learn how to use Fabric. Give yourself a guided tour by making selections in Fabric and reading the feature-aware documents. Forum topics: This section shows topics from the Community forums that are related to the features on the current screen. Select a topic to open it in a new separate browser tab. Other resources: This section has links for feedback and Support. Help pane is a search engine Enter a keyword to find relevant information and resources from Microsoft articles and Community forum topics. Use the dropdown to filter the results. Monitoring Hub improvements With the new “column options” feature in Monitoring Hub, you can now customize the columns in the job list to check the information you are interested in. The columns selections would be memorized automatically. Feel free to go to other places, we will keep the columns and the filters for you once you are back at the monitoring hub. Admin Microsoft Public Preview Default Sharing On by Default Effective July 5 th , 2023 On July 5th, 2023, Microsoft began a staged rollout of an update to the Fabric Public preview setting. After the update, Fabric Public preview will be ON by Default unless customers explicitly opt out. With Fabric Public preview set to ON, users in your organization will be able to create Fabric items in workspaces attached to a Power BI Premium or Fabric capacity. When the Microsoft Fabric Preview was launched on May 23, 2023, Fabric was disabled by default, using a new tenant setting named "Users can create Fabric items (public preview)". Starting July 5th, 2023, Microsoft started activating Fabric preview by default for tenants who have not explicitly opted out. Any capacity level overrides of this setting will remain unaffected. If you are comfortable with your organization using the Fabric preview features, no further action is required. However, if you want to restrict usage of the Fabric preview in your P-SKU and F-SKU capacities, you have two options: Use the security group configuration of this setting at the tenant and capacity levels to limit Fabric to a smaller set of users. Turn off the setting at both the tenant and capacity levels to prevent all users from accessing the Fabric preview features. If you make these updates before the automatic update rolls out your tenant, this tenant setting update will not impact your organization. You may also make these changes at any time after the update. For more information about controlling Fabric use, visit Microsoft Learn. OneLake OneLake file explorer update with support for switching organizational accounts With the latest OneLake file explorer v1.0.9.0, it’s simple to choose and switch between different Microsoft Azure Active Directory (AAD) accounts. This was a highly requested feature from users who work with multiple AAD tenants, across organizations or even within a single organization. To switch accounts, right click the OneLake icon in the Windows notification area, select “Account” and then “Sign Out”. Signing out will exit OneLake file explorer and pause the sync. To sign in with another account, start OneLake file explorer again by searching for "OneLake" using Windows search (Windows + S) and select the OneLake application. Item sharing Fabric_July_2023_Update Starting July 2023, you can share Fabric items like data warehouses, lakehouses, Spark job definitions, Kusto databases, and KQL query-sets with users or groups. This enables collaboration with users that are not in workspace roles. Admins, Members, or users that have been granted reshare permission on a specific item, can share it with additional users outside of the workspace. You can share an item by clicking on Share in the item list or within the item. When an item is shared, the recipients can discover the item in Data Hub and may also receive a link to the item via email (if the option is selected while sharing). While sharing, a user can choose the level of access that the recipient can have. For example, when sharing a lakehouse, you can choose to grant the recipient with Read All SQL endpoint data permission, in addition to the read permission. This enables the recipient to read the default dataset associated with the lakehouse and access lakehouse data through the SQL endpoint. Here’s an example of sharing a lakehouse: Depending on the item being shared, you may find a different set of permissions that you can grant to recipients when you share. Read permission is always granted during sharing, so the recipient can always discover the shared item in Data Hub and open it. Here’s an example of sharing a warehouse: Sharing a Kusto database You can also grant or revoke permissions on specific items to users by selecting manage permission from the context menu. For example, you can grant the reshare permission on a Lakehouse to a user in the Workspace Contributor role. This would allow the contributor to share the lakehouse with users outside of the workspace. Note that you cannot modify the permissions inherited from a workspace role. Select manage permission from the context menu 2. Select Add user on Direct access tab and enter the names of users or groups that you want to provide access to. 3. Select grant To learn more about sharing Fabric items, read: Share items in Fabric. For more information on sharing of specific items, please read Share your Warehouse and manage permissions and How lakehouse sharing works? Synapse Data Warehouse Automatic updates for statistics SQL statistics are now automatically updated by the query engine! In the Fabric Data Warehouse and Lakehouse SQL Endpoint, statistics are a critical tool for helping your query run quickly and efficiently. When a query is executed, the engine will try to collect existing statistics for certain columns in the query and use that information to assist in choosing an optimal execution plan. Today, column statistics are automatically generated when the query engine requires statistics on columns that don’t yet have any. When data in your table’s columns changes significantly, it’s important that those statistics objects also update to accurately reflect the new data. Previously, this meant users would need to manually update on a regular basis. Now with automatic statistic updates, any statistics required during a user query are automatically assessed and refreshed if determined as outdated, allowing your query to use the most precise plan for execution and ensuring your workload is positioned for the best performance possible – all with zero user intervention. For more information on automatic statistic updates, Statistics - Microsoft Fabric | Microsoft Learn UPDATE FROM and DELETE FROM T-SQL You can now update and delete data in your target table from an existing source table using the FROM argument in your UPDATE and DELETE scripts in Fabric SQL respectively! These T-SQL commands allow you to perform MERGE T-SQL like operations. For more information, see UPDATE (Transact-SQL) - SQL Server | Microsoft Learn and DELETE (Transact-SQL) - SQL Server | Microsoft Learn Optimization in data movements during query execution The Fabric SQL engine creates a query plan that comprises of execution steps and data move operations. Data move operations ensure that the data required for a query step is ready in the location of its execution. With this new optimization in Fabric SQL, the engine makes data movement in intermediate steps of query execution more balanced and improves overall query performance. No user intervention or code change is required to use this optimization; it works out of the box! Dataflows Gen2 from the Warehouse You can now create a new Dataflow Gen2 directly from your warehouse to ingest data. Simply click Get Data --> new Dataflow Gen2. The newly created Dataflow Gen2 artifact will already prepopulate the destination to the warehouse. For more information about creating, ingesting, and transforming data with Dataflows Gen2, see Create your first Microsoft Fabric dataflow - Microsoft Fabric | Microsoft Learn. Zero Copy Table Clones We are pleased to announce that zero copy Table Clones are now available in Public Preview! Zero copy clones are a near-instantaneous metadata only operation that enables you to easily create a copy of your Warehouse table(s) with no additional cost and minimal overhead. Table clones contain a reference to the source table that the clone was created from. Underlying parquet files are not duplicated when a clone is created – under the hood, a fork is created, and the clone behaves as an independent table which can be modified as needed. Any changes made to the source table after it was cloned are not reflected in the clone; similarly, any changes made to the cloned table are not reflected in the source. They are independent of one another. As of today, customers can create a Table Clone within the same schema or to a different schema within the same Warehouse. Customers looking to create a Zero Copy Table clone would typically do so for a variety of test, development and production use cases and experimentation. For example, a customer may want to stage a production release in a table clone prior to going live with the changes. Once they are happy with the changes, they can be merged to the production table. For more information on Table Clones, see Clone table in Microsoft Fabric Data warehouse sharing We are excited to announce that Data warehouse sharing is now available in Public Preview! Data sharing is essential to fostering a data-driven culture within an organization. Sharing a Warehouse allows you to easily provide read access to enable downstream users to consume this data, without making copies of data. With this new capability, an Admin or Member within a Fabric workspace can share a Warehouse with another recipient (AAD user or AAD groups) within your organization. The following are the permissions that are provided: [Default] Connect permissions to the warehouse – This option is provided by default and it provides permissions to connect to the warehouse (equivalent of Connect permissions in SQL) but not query any table or view. You can grant granular object access using GRANT in T-SQL. [Default] Build reports on the default dataset - This option is provided by default and provides “build” permissions on the default dataset that is connected to your Warehouse. This option can be useful for your Power BI developers who want to create reports on this default dataset. [Optional] Read all data using SQL – This option provides “readData” (equivalent of db_datareader) permissions which allows for read access to all tables and views within the Data warehouse. This option can be useful for users who want to read using SQL. [Optional] Read all data using Apache Spark – This option provides “readAll” permissions which allows read access to the Warehouse’s underlying files in One Lake that you can read through Spark. This option can be useful for your data scientists who want to read using Spark. For more information, see the [detailed sharing blog]. Dbt Support for Fabric Data Warehouse We are excited to announce the preview of dbt plugin adapter for Synapse Data Warehouse in Microsoft Fabric (preview). This data platform-specific adapter plugin allows you to connect and transform data in Synapse Data Warehouse in Microsoft Fabric. For more information, see the Introducing the dbt adapter for Synapse Data Warehouse in Microsoft Fabric, Microsoft Fabric Synapse Data Warehouse dbt adapter setup and Microsoft Fabric Synapse Data Warehouse dbt adapter configuration. Data Engineering Load to Table Feature Updates Previously the "Load to Table" feature allowed users to load a single file to a new table, which was very well met with data engineers due to the added productivity value of quickly using simple right click actions to enable table loading on Files and as well as because of the no-code experience, which lowers the entry bar for all personas. This new release brings improvements to this experience with several new functionalities: Folder-level load: Users can now load all files under a folder and its subfolders at once by selecting "Load to Delta Table" after clicking on a folder. This feature automatically traverses all files and loads them to a Delta Table. Load to existing table: User can now choose to load their files and folders to a new or an existing table of their choice. If they decide to choose to load to an existing table, they have the option to either append or overwrite their data in the table. Source file option: User can specify if their source file includes the column names as the first row of data and the separator used in it. For more detailed information on this feature, visit the documentation here. Lakehouse sharing and permission management We are announcing a capability for users with Admin and Member roles to share an individual Lakehouse with users without providing them access to the workspace. That will grant access to a specific lakehouse without exposing other items in the workspace. The users will get access to shared items through Data Hub or the link included in the sharing notification email. With access to a lakehouse, users can access the SQL endpoint and default dataset. That provides features like querying data using T-SQL and building Power BI reports on top of the lakehouse data. Permission management also allows users in Viewer role to get additional permissions to access Lakehouse data using Spark. These features will push data democratization in Fabric even further, enabling more collaborative work. Notebook Resources We are excited to announce the "Notebook resources" on Fabric notebook. This feature offers notebook users a writeable file system space where you can store small-sized files, such as code modules, datasets, and images. You can access them with code in the notebook as if you were working with your local file system. The Notebook Resource explorer provides a Unix-like file system to help you manage your folders and files. You can use common operations such as create/delete, upload/download, rename, duplicate, and search through the UI, and rich built-in snippets are provided through “Drag & Drop”. For more details please see How to use notebooks - Microsoft Fabric | Microsoft Learn. Notebook: Show cell output after session reconnection Fabric notebook now supports displaying the running cell output after reconnecting to the original session. This feature allows you to easily recover your ongoing work after accidentally closing the browser or leaving the live session. You don't need to take any additional action to enable it, as this feature is available by default on the Notebook. Notebook: Sharing and permission management Starting now, sharing a single notebook with your colleagues is easier than ever before, without having to grant workspace permissions. With the Notebook sharing feature, you can collaborate with team members and share your work conveniently. Additionally, we now support managing permissions for each Notebook instance. You can easily check and update the permissions of notebooks after they have been shared, ensuring proper access to the notebook. Notebook: New status bar Notebook recently upgraded the design of status bar. In this upgrade, we have styling refinement as well as functional enhancements. Now you can easily discover the “Save options” status, you can easily navigate to the failed cell, and you can find more useful information in the floating info card - the diagnostic information is especially helpful when you encounter issue and need service support! In the next release we’ll add another series of quick access entries on the status bar, so stay tuned for the upcoming new features! Data Science Microsoft Fabric and SynapseML Offer New Features for Large Language Models Microsoft Fabric and SynapseML has featured a variety of new features that have simplified the use of large language models. In our latest release, we have integrated the Azure OpenAI service with the distributed machine learning library SynapseML, which makes it easy to use the Spark distributed computing framework to process millions of prompts with the OpenAI service on Microsoft Fabric. The new OpenAI APIs that have been introduced into SynapseML include “OpenAICompletion”, “OpenAIEmbeddings”, “OpenAIChatCompletion”, and “OpenAIPrompt”. The following helps to better understand these new OpenAI APIs. Leveraging the new OpenAI APIs in SynapseML and Microsoft Fabric, we have also demonstrated how to perform Q&A on PDF Documents. You can read more about it here. Please note that native access to the Azure OpenAI service will be coming to Microsoft Fabric later this year. Real-time Analytics KQL Database Create a Continuous Data Connection from Event Hubs to KQL Database Azure Event Hubs is a big data streaming platform and event ingestion service that can process and direct millions of events per second. Now you can easily stream your Azure Events Hubs data directly into your Fabric KQL Database. There are 2 main steps required to stream the Event Hubs data to the KQL Database: Create a Microsoft Fabric platform-based data connection to a specific event hub instance. This data connection can be used across all Microsoft Fabric workspaces and is managed centrally. Connect this Microsoft Fabric-based data connection to a KQL database. This process creates a database-specific Event Hubs data connection. The connection streams data into the table you specified during setup, and the data will then be available to query using a KQL queryset. A pre-requisite for creating a cloud connection in Microsoft Fabric is to set a share access policy (SAS) on the event hub and collect information to be used later in the setting up the cloud connection. This step is performed in the Azure portal. Go to your event hubs instance and under Settings select the Shared access policies. Add a new SAS Policy or select an existing one: To Create the Cloud Connection from Azure Event Hubs and Fabric, go to the menu bar of your Fabric workspace and go to Manage Connection and gateways. In the New Connection form, you will need to enter details taken from the defined SAS policy. The final step is to connect a table in your KQL database to the event hub cloud connection defined above. This can be done by selecting the Event Hubs option in the Get Data menu bar: For more details, go to the docs: Get data from Azure Event Hubs. Query KQL Database from MATLAB In manufacturing and energy verticals, MATLAB is still heavily used. MATLAB is a programming and numeric computing platform used to analyze data, develop algorithms, and create models. We are happy to announce that Fabric now supports the querying of KQL database data directly from MATLAB. Now engineers that are familiar with MATLAB do not need to learn KQL to query the high performant RTA KQL database, but rather reuse their MATLAB skills to query the data. This functionality is supported by a very lightweight MATLAB Connector that securely connects MATLAB to a Fabric KQL Database. For more details, go to the docs page: Query data using MATLAB. Data Factory Dataflows Gen2 Power Query Editor Enhancements to output destination experience Auto-fix column names during mapping of columns dialog Auto-fix data types during mapping of columns dialog Improvements to existing experiences Rename a dataflow inside of the Power Query Editor A similar experience to other artifacts inside of Microsoft Fabric, you can now change the name of a Dataflow Gen2 inside of the Power Query Editor. Connectors Google Analytics (Connector Update) The Google Analytics connector has been updated to support Google Analytics Data API (Google Analytics 4). To use this new functionality, use “Implementation 2.0” when connecting. Existing connections will not be affected. Oracle Database (Connector Update) The Oracle connector has been updated to enable Azure AD-based Single Sign-On functionality through the on-premises data gateway. This will require the July release of the on-premises data gateway. Azure Databricks, Databricks (Connector Update) The Azure Databricks and Databricks connectors have been updated. Please find notes from the Databricks team below. Add a new DSRHandler to databricks-multicloud Fix UC_NOT_ENABLED and Catalog 'spark' not found error in legacy code path using Databricks.Contents Denodo (Connector Update) The Denodo connector has been updated. Please find notes from the Denodo team below. This new version adds graphical support for the specification of native SQL queries at data source creation time EQuIS (Connector Update) The EQuIS connector has been updated. Please find notes from the EQuIS team below. Remove "Beta" attribute Retrieve report content as .csv to remove the row limitation of .xlsx files Optimize handling of facility groups in navigation tree Show report and/or location folders in navigation tree even if one or the other is empty Snowflake (Connector Update) The Snowflake connector has been updated to include various performance improvements, such as usage of SQLBindCol. Users should experience better performance when running queries. Anaplan (Connector Update) The Anaplan connector has been updated. Please find notes from the Anaplan team below. This version of Power BI connector for Anaplan includes backend changes for compatibility with ongoing Anaplan infrastructure updates. There is no change to user facing connector features. Gateway and Connections SSO option in cloud connections Added support for Single Sign-On (SSO) via Azure AD in cloud connections. Currently this feature is only applied for DirectQuery mode in datasets. However, we plan to progressively extend SSO capabilities to other Fabric workloads in the future. Flag in cloud connections to allow them to be with gateway We have enhanced our security measures to allow users to disable cloud connections to be used with gateway connections. This precaution prevents the decryption and logging of cloud connection credentials within on-premises systems. Data pipelines Activities Teams Activity We’re excited to announce that the Teams activity is now available to use in your Data Factory data pipelines. In your pipeline, you can use the Team activity to customize a message to send to a Teams channel or a Teams group chat. For example, you can use the Teams activity to send a notification if a pipeline has failed, helping you to better monitor your data integration pipelines. Productivity Artifact parameterization support We’re excited to share that you can now parameterize Fabric artifacts in your data pipeline. This will allow you to use expressions, functions, parameters, and variables to dynamically refer to your Lakehouse or Data Warehouse, allowing more flexibility when designing your data pipelines. Performance improvement for moving parquet files to Lakehouse tables We’ve made performance improvements when copying parquet files to your Lakehouse! “Save as” support now available from your workspace “Save as” is now supported by data pipelines from your Data Factory workspace. When you are in your workspace, you can use the selection menu to save a new copy of your pipeline. This will allow you to build upon or edit existing pipelines without having to completely rebuild a data pipeline. Support column mapping for Lakehouse destination in Copy activity Data pipelines now supports column mapping when a Lakehouse is selected as a data destination. In the Mapping tab, you can now add, edit, or delete column mappings from your data source to your data destination. Data Activator Sample data The simulator that generates sample data now includes multiple event streams that you can use to try out building objects and alerts. If you want to try the simulator, check out our tutorial: End-to-end tutorial using simulated data. Adding multiple streams or properties The Data screen now gives you options to create objects directly from columns in your data streams, and gives you shortcuts to quickly create properties directly rather than having to create them and manually select the column. We’ve also planned a new UX to let you create multiple properties or objects across multiple streams in one go – stay tuned for that in the coming months! Designing triggers and alerts We’ve combined the first two steps of triggers (Reference to a property and selecting the value from an event) into one step. This means the first thing you’ll see is a chart that plots the values you select helping you understand the data more quickly. It’s currently called “Property field reference”, which will be updated to be more user-friendly soon! We’ve added many more functions to the library you can use to build triggers: Detect functions. Changes, Changes from/to Sends an event each time the field changes from/to a value, or changes at all. Is less than/greater than, Is false/true, Is equal/not equal Sends an event whenever the field meets that condition. Becomes less than/greater than, becomes true/false, Exits/Enters range Sends an event the first time the field meets that condition (if subsequent events also meet that condition, no further events are emitted) Summarize functions. Maximum, minimum, average, count over time. Calculates a aggregation of the values of all events in a time window. Filter functions Filter Filters individual events where the value meets a specified condition. Other events are dropped. This is useful for filtering out error values such as a sensor that returns -99 for errors. Property filter Filters out instances that match the criteria. This is useful if you only want a trigger to fire for certain instances (this will be more usable when the trigger changes described below are done). ‘Detect’ functions such as ‘Crosses above’ or ‘Changes to’ also have options to only trigger when the criteria is met a certain number of times, for example 3 times in an hour. This can help reduce noise in your triggers. Coming soon, we have a major change to the way triggers are defined. Early preview feedback showed that the detailed step-by-step setup for triggers was too complicated, so we’ve geared them around selecting 3 things: what you what to monitor, what condition you want to detect, and what action to take. This should make it easier to configure your triggers and make it clearer what’s needed to get alerts on your data. Trigger lifecycle Based on feedback to make the management of triggers simpler, we removed the Draft concept. You don’t need to Publish a trigger before you can start it. Triggers are either just Started or Stopped! Any time you make changes, you select Update to change the running trigger to use the new values. Testing actions The ‘Test action’ button confused early users so we’ve updated the messaging to clarify that it will send a test to the current user. It’s also only enabled if there is some data that met the criteria, which is used as sample data in the test message. Teams app Data Activator can now send alerts to you from our new Teams app. To install it, search the Teams store! A Teams administrator can set it up for everyone in your organization if you need to deliver messages broadly, rather than each user installing it themselves. Community Implementing a Lakehouse with Microsoft Fabric Courses This course is designed to build your foundational skills in data engineering on Microsoft Fabric, focusing on the Lakehouse concept. This course will explore the powerful capabilities of Apache Spark for distributed data processing and the essential techniques for efficient data management, versioning, and reliability by working with Delta Lake tables. This course will also explore data ingestion and orchestration using Dataflows Gen2 and Data Factory pipelines. This course includes a combination of lectures and hands-on exercises that will prepare you to work with lakehouses in Microsoft Fabric. Check out the course: Course DP-601T00: Implementing a Lakehouse with Microsoft Fabric Filter by product for User Groups & Events You can now filter by product for both Fabric User Groups and Events in the Fabric Community site. With the broad range of experiences within Fabric, product filtering allows you to easily find User Groups or events that match your interests! You can also quickly refilter your results by removing products either in the Product drop down or with the tiles below the search bar. Data Activator Community now live Data Activator now has its own community forum, alongside the other product forums across Fabric: https://test.community.fabric.microsoft.com/t5/Data-Activator-preview-Community/ct-p/dataactivator (or use the short URL https://aka.ms/dataActivatorCommunity!). If you’ve got any questions or feedback about Data Activator you can post there and the product team will be in touch. Power BI Reporting Smoothed and Leader lines – Updates We are excited to announce the launch of one of our most highly acclaimed features! Report creators can now create smoother line and area charts, providing a more polished look to their visualizations. To access this setting, go to Lines > Shape > Line Type. We've recently added leader lines for both line and area charts. This new feature creates a visual connection between each data point and its corresponding label. To access this feature, simply navigate to the Data labels > Options > Leader lines. These features are just the beginning of the many improvements we have in store for graphs, charts, plots, and markers in the coming months. Get ready for even more exciting updates! On-Object Interaction (Preview) – Updates The new on-object interaction feature released to preview back in March. This month we bring more improvements and bug fixes. Customize the Pane Switcher We’ve now added a new “+” button on the pane switcher to quickly add new panes directly from the pane switcher without having to go to the View ribbon. This menu also gives a brief description of what panes are available and what their functions are. Even better, the panes added to the switcher are saved across reports. Configure once and you’re done! You can also access the 2 preference settings released last month for “always show the pane switcher” and re-attaching the build menu as a pane by using the gear icon. CTRL click to open multiple panes in the Pane Switcher In addition to the right click option “open in new pane”, it is now even easier to open multiple panes from the pane switcher by simply holding down the CTRL key and clicking the pane you wish to open. Treemap sub-selections now supported: Noteworthy bug fixes Overlap of the on-object buttons on the formula bar has finally been resolved! We appreciate your patience as this bug was a bit trickier to fix the right way. Visual tooltip showing automatically when opening the build menu, blocking the formatting on-object button is now fixed. Selected visual type is now reflected in the ribbon visual gallery accordingly. Mini-toolbar’s fill color icon now reflects conditional formatting gradient as well. Thanks for continuing to try out the new preview and provide feedback. We’re working hard to react to your suggestions and add the necessary changes to make on-object work for you. Please continue to provide your comments directly in this blog post or in our community forum via the “Share feedback” button next to the preview switch. Modeling Edit your data model in the Power BI Service– Updates (Preview) The new data model editing in the Service feature released to preview in April. We've been busy reacting to your feedback and enhancing the experience. Below are the improvements we are adding this month: Relationship validation We are adding relationship validation in the Service, making it easier to create and edit relationships in the web! Like Power BI Desktop, as you define the properties of your relationship, the system will automatically validate it and offer appropriate choices for cardinality and cross filter selections. Please continue to submit your feedback directly in the comments of this blog post or in our feedback forum. Service Dataset details page revamp We are happy to announce the revamp of our dataset details page! Now, when you click on a dataset item in the OneLake data hub and workspace view, you will be directed to the redesigned page that not only enhances the look and feel but also introduces new capabilities for an improved user experience. Here's what you can expect to find on the dataset details page: Actions: You will find various actions that can be performed on the dataset, such as creating a report and refreshing the dataset. With this release, we have added the option to view the refresh history under the refresh menu. Dataset Metadata: Gain insights into the dataset through its description and last refresh time. Related Items: Explore existing related items associated with the dataset. Dataset Schema: Get a comprehensive view of the dataset's tables and columns. Clicking on a table provides a table preview, with export capabilities available using paginated reports behind the scenes. Additionally, we have made significant improvements to the related items list. It now showcases all the downstream and upstream dependencies for the dataset. This enhancement allows you to easily identify the sources of the dataset, composite model relations, reports, and dashboards associated with it. We believe that these updates will greatly enhance your experience with the dataset details page, providing you with a more intuitive and comprehensive understanding of your data. We look forward to your feedback as you explore these new features!" Mobile Datasets are coming to Power BI Mobile apps! In the next Power BI Mobile app release, we are adding a long-waited feature that will help dataset owners and report creators to manage their dataset directly from their phone. That means that you will be able to see in your mobile device datasets. Go to a workspace, make sure to select the “dataset” pill at the top and get the list of datasets, that you have access to in that workspace. When tapping on a dataset, you will get the dataset metadata pane, which includes the name, owner, sensitivity label and also the latest refresh status. From this pane you can also trigger a dataset refresh – all directly from your mobile app! Dataset owners will also get push notifications when schedule refresh fails. They will be able to view the failure details and be able to re-try the refresh while they are on-the-go. Developers Boost Your Custom Visuals Performance: Check out our Latest Article! We have recently published an article that focuses on techniques to improve the performance of custom visuals. In this article, we discuss the performance improvements we have made in visual rendering and load times. We identified and addressed certain bottlenecks in the code, and these improvements are available for any visual that has been updated to API version 4.2 and onwards. Along with these fixes, we also provide code practices and techniques that can greatly enhance the performance of rendering custom visuals. I encourage you to check out the article here. We believe that these techniques can make a significant impact on the performance of your custom visuals. Visualizations New visuals in AppSource Spider Chart for Power BI by VisioChart HTML Content (lite) Stacked Lipstick Bar Chart (Standard) Stacked Lipstick Column Chart (Standard) 100% Stacked Column Chart with Values instead of % (Standard) Dual Axis Scatter Chart (Standard) Category Comparison Bar Chart Stacked Column with Percentage and Total in Label (Standard) Likert Scale Chart for Power BI by ChartExpo Elevate Your Map Charts With Drill Down Map PRO Drill Down Map PRO by ZoomCharts is a custom map visual for Power BI that lets you show your data on an interactive map and give it location-based context. Features include: Built-in shape layers – use preset shapes for easy filtering of countries. Custom shape layer support – provide custom shapes through KML and GeoJSON files. Lasso tool – draw and save your own filter shapes on top of the map. Node clustering capabilities – clusters can be turned into donut or pie charts for category display. Map base layer lets you choose from 4 options – Azure maps, Custom (OpenStreetMaps, Google, CartoDB etc.), Image (e.g., floor plans), None (visualize shapes without a background). Aura, image, and custom label support. Popular use cases: Production – monitoring production data by location. Sales and marketing – mapping sales results by region. Public sector – visualizing environmental and sociodemographic data. Get Drill Down Map PRO now! Learn More about Drill Down Map PRO. ZoomCharts Drill Down PRO Visuals are known for their interactive drilldowns, smooth animations, rich customization options. All Drill Down PRO Visuals support: touch input devices, interactions, custom and native tooltips, filtering, bookmarks, and context menu. We are ready to present our new visual: an "all-in-one" Multi Target KPI card The Multi Target KPI card works with a single query and includes three additional indications, multiple categories, a pixel perfect alignment setting, and built-in conditional formatting. You can change the settings of layout type and color conditional formatting for additional measures in our visual, and it is fairly simple for non-designers to use! Just select the desired measure and category, if necessary. Add up to three additional indicators to provide the context you need for your metric. It will help you improve reporting performance and save you time when designing and developing supplemental measures. Start a new level of business dashboarding! Link to AppSource: https://appsource.microsoft.com/en-us/product/power-bi-visuals/ndividualentrepreneuralekseikolokolov1654522457293.80416480?tab=Overview Link to our website: (delete if not needed) https://data-2-speak.com/cardkpi Other WebView2 GA WebView2 is now generally available. Thanks to everyone who reported issues during the preview phase! Your input helped us improve reliability to higher than it was before we introduced WebView2. Please continue to report any issues using the “There was a problem with WebView2” dialog.Introducing High Concurrency Mode in Notebooks for Data Engineering and Data Science workloads in Microsoft Fabric
We are excited to announce a new high concurrency mode in Fabric for Data Engineering and Data Science. This allows users to share Spark compute across multiple notebooks within a workspace which means that you can run multiple Spark notebooks simultaneously on the same Spark session without compromising performance or security when paying for a single session. High concurrency mode offers an instant run experience avoiding session start delays and ~30X faster session start experience for the shared notebooks when running on custom pools. What is High Concurrency Mode? High concurrency mode allows sharing of Spark compute across multiple notebooks and allows their queries to execute in parallel. In this shared mode, the resources and configurations of each notebook are isolated from each other. As the session sharing is always within a single user boundary, users cannot access or modify the data or variables of another user's high concurrency session. High concurrency mode also leverages FAIR allocation to optimize the resource utilization and performance of the notebooks and ensures that each notebook gets a fair share of the executors available for the Spark application. Why Use High Concurrency Mode? High concurrency mode offers several benefits for Fabric Spark users, such as: Faster and easier Spark session start: You don’t have to wait for the Spark pool to spin up or configure the node sizes when you start a Spark session. As the session is already warmed up and running, attaching a new notebook to an existing Spark session gives a session start experience within ~5 seconds. You can also use custom Spark pools, which allow you to size the nodes, enable autoscaling, and dynamically allocate executors based on your Spark job requirements and with custom pools you would get a 30X faster session start experience for shared notebooks. Enhanced security and isolation: You can ensure that each user or query has its own isolated Spark session, which prevents data leakage or tampering. Do more by paying less: Achieve better compute cost savings by sharing a single session across multiple notebooks for your Data Engineering or Data Science workloads and only get billed only for the single session. How to Enable High Concurrency Mode? To enable high concurrency mode for your Fabric Spark workspace, you need to follow these steps: Go to the workspace settings in your Fabric workspace. Navigate to the Data Engineer/Science section. Select the Spark Compute menu Enable the High Concurrency Mode Option if its disabled (This option should be enabled by default for all Fabric Workspaces) 4. Save your changes. Once you enable high concurrency mode, you can run your notebooks in High Concurrency mode from the notebook menu ribbon. To learn more about using high concurrency in notebooks read: Sharing Spark Compute across Notebooks with High Concurrency Mode in Fabric. For more information on high concurrency mode, please read Overview of High Concurrency Mode in Microsoft FabricMicrosoft Fabric August 2023 update
Welcome to the August 2023 update. We have lots of features this month including the new layout switcher for Power BI, SSD caching in Synapse Data Warehouse, in-line Python support for KQL in Synapse Real-time Analytics, lookup activity for Data Factory Dataflows, and much more. Continue reading for more details on our new features! Contents Core Multitasking navigation improvement Monitoring Hub support for personalized column options Power BI Reporting New layout switcher New bubble range scaling setting On-Object Interaction (Preview) – Updates Resizing and positioning to the on-object menus On-object format subselections now supported in spotlight and focus mode Modeling Updates to ORDERBY function Data Connectivity New modern data connectivity and discovery experience in Dataflows Lakehouses (Connector Update) Service XMLA Write support for Direct Lake datasets Automatic replica synchronization for Dataset Scale-Out (preview) Mobile Choose your start-up content Developers Published AppSource visuals now generate leads in Partner center and CRM Publish a Power BI Project (PBIP) directly from Power BI Desktop Visualizations New visuals in AppSource Editor's pick of the quarter Drill Down Combo Bar PRO by ZoomCharts Sunburst Chart by Powerviz xViz Performance Flow by Lumel Synapse Data Warehouse SSD Caching Data Engineering High Concurrency Mode for Notebooks Data Science Model & Experiment Usability Improvements New Data Science Sample are Coming to Microsoft Fabric Real-time Analytics KQL Database support for inline Python KQL Database provisioning in a few seconds Data Factory Dataflows Gen2 Power Query editor Edit connection in Manage Connections Enable Staging labels Publish dataflow without staged queries Limit to number of queries in a Dataflow when publishing Refresh history Direct navigation to last failed refresh Statistics for tables Connectors Lakehouse connector update Other improvements Data pipelines Connectors FTP Connector Productivity Lookup activity for Lakehouse, Data Warehouse, and KQL Database Get Metadata activity for Lakehouse and Data Warehouse Pipeline run status now available Secure input and output options for Set Variable activity Data Activator Trigger and property design template Quickly assign columns as properties in an object Data Activator now supports Power BI visuals with a time axis Trigger Power Automate flows from Data Activator Core Multitasking navigation improvement In our latest update, we've introduced an enhancement that preserves all Fabric items opened in a single browser tab on the left navigation bar, even in the event of a page refresh. This ensures you can refresh the page without the concern of losing context. Monitoring Hub support for personalized column options We have updated Monitoring Hub to allow users to personalize activity-specific columns. You now have the flexibility to display columns that are relevant to the activities you're focused on. Power BI Reporting New layout switcher We’ve added new buttons that make it easy for you to quickly switch between web and mobile layouts while you’re developing your reports. You'll find the new switcher buttons down at the bottom of the screen, right next to the page navigator. New bubble range scaling setting We are introducing the new bubble range scaling setting for scatter chart and map visuals! This setting gives report creators more control over how the bubble (marker) sizes respond to the data, making it more accurate or distinctive based on preference. With the magnitude option, the bubble areas closely follow the data proportions. With the data-range option, the bubble size limits are mapped to data minimum and maximum. The auto option, which is the default setting for new reports, selects the appropriate option based on data characteristics. For more information, visit our docs. This setting can be adjusted in formatting panel, Markers > Shape > Range scaling for scatter charts or Bubbles > Size > Range scaling for maps. For reports authored in earlier Power BI versions, these settings default to (Deprecated) for scatter charts (which differs in handling negative values), and Data range for map charts. Azure Maps charts will also include this feature in a coming product update. In the figure above, the size of each country represents Urban Population, which is also shown on the y-axis. On-Object Interaction (Preview) – Updates The new on-object interaction feature released to preview back in March. We’ve been busy adding even more improvements to the preview, here’s what’s part of the Aug release: Resizing and positioning to the on-object menus We’ve now added the ability to resize the on-object menus horizontally, this is especially helpful when you’re working with long field names. We’ve also improved the positioning of the on-object menus to make better use of the canvas space. Previously, when a visual was near the bottom of the canvas, the on-object menu was super small and required scrolling to be able to see and use the field wells. Now, the on-object menu moves up and stretches into the canvas to bring the field wells into view without needing to scroll. Before After On-object format subselections now supported in spotlight and focus mode When spotlighting a visual or expanding the visual in focus mode, you can now use on-object formatting to subselect and format styles. In focus mode, it can be hard to tell when you’ve entered format mode with just the subtle border. To address this, we’ve added a button to the header to better indicate when you’re in format mode and how to exit format mode while staying in focus mode. Thanks for continuing to try out the new preview! We’re working hard to react to your suggestions and add the necessary changes to make on-object work for you. Please continue to provide your comments directly in this blog post or in our community forum via the “Share feedback” button next to the preview switch. Modeling Updates to ORDERBY function If there are blanks in the data, you can specify where to order them by adding ‘BLANKS LAST’ or ‘BLANKS FIRST’. For example, this is a perfectly valid expression: BiggestSpender = SELECTCOLUMNS ( INDEX ( 1, ALLSELECTED ( 'DimCustomer' ), ORDERBY ( SUM ( 'FactInternetSales'[SalesAmount] ), DESC BLANKS LAST) ), 'DimCustomer'[FullName] ) Specifying how blanks are handled is optional and can be combined with specifying the order direction (DESC/ASC). Valid values include: BLANKS DEFAULT. This is the default value. The behavior for numerical values is blank values are ordered between zero and negative values. The behavior for strings is blank values are ordered before all strings, including empty strings. BLANKS FIRST. Blanks are always ordered on the beginning, regardless of ascending or descending sorting order. BLANKS LAST. Blanks are always ordered on the end, regardless of ascending or descending sorting order. Read more in our documentation. Data Connectivity New modern data connectivity and discovery experience in Dataflows We are excited to introduce the new data connectivity and discovery experience in Dataflow, Dataflow Gen2, and Datamart. Today, users spend a lot of time finding the right data, the right connection info and credentials. With the new Get Data experience, we make it easy to browse different Fabric artifacts through the OneLake data hub. This improved experience aims to expedite this process and get you closer to the data that you’re looking for in the quickest way possible. Learn more in our blog post Announcing a new modern data connectivity and discovery experience in Dataflows Lakehouses (Connector Update) This update includes significant performance improvements to the Lakehouses connector. Be sure to update to the August version of Power BI Desktop and Gateway to experience these improvements! Service XMLA Write support for Direct Lake datasets We are excited to announce that Direct Lake datasets now support XMLA-Write operations. Now you can use your favorite BI Pro tools and scripts to create and manage Direct Lake datasets. Whether you prefer SQL Server Management Studio (SSMS), Tabular Editor, DAX Studio, or something else, you can connect to your Direct Lake datasets using XMLA endpoints and perform operations such as deploying, customizing, merging, scripting, debugging, and testing. You can use tools like Azure DevOps or GitHub to implement source control, versioning, and continuous integration for your data models. You can automate and streamline your development and deployment processes. You can also use PowerShell or REST APIs to automate tasks such as refreshing or applying changes to your Direct Lake datasets. XMLA Write is incredibly powerful and the key to data modelling efficiency and productivity. For more information about XMLA Write support in general, check out the article Dataset connectivity with the XMLA endpoint in the product documentation. Automatic replica synchronization for Dataset Scale-Out (preview) We are excited to announce that we have finalized Dataset Scale-Out configuration APIs and completed the replica synchronization feature. Specifically, you no longer need to enable Scale-Out at the workspace level by using a burdensome XMLA request. The XMLA command is deprecated and will no longer work. You can now enable Scale-Out on a dataset-by-dataset basis using the Power BI REST API for datasets. You also no longer need to synchronize read replicas manually if you want to take advantage of automatic replica synchronization. Automatic replica synchronization is enabled by default. However, it is also possible to disable automatic synchronization to synchronize the read/write and read replicas of a dataset manually for controlled refresh isolation. Learn more in the automatic replica synchronization announcement blog Mobile Choose your start-up content Power BI mobile app users can now choose which item they want to have open automatically whenever they launch the Power BI Mobile app. This feature saves time for users who mostly view a specific item on their mobile app, and don't want to waste time navigating from the app’s home page every time they open the app. To configure a launch item for yourself, open the item you want to see when you launch the app. This can be a specific report page, dashboard, scorecard, report in an app or entire app. When the item is open, open the More options (…) menu from the header and select Set as launch item. This will mark the item as the launch item. Only one item at a time can be marked as the launch item. Administrators can also use mobile device management (MDM) tools to remotely configure a launch item for a group of users (front-line workers, for example) to simplify their experience with the app. Developers Published AppSource visuals now generate leads in Partner center and CRM Until recently, your customer leads may have come solely from customers downloading them from AppSource.com. However, now you have access to even more leads through Power BI - both the desktop and web embed AppSource. To access these new leads, simply navigate to the Referrals workspace in Partner Center. Here, you can see all the leads you receive from Power BI, as well as those from AppSource.com. Plus, if you've connected your CRM, you'll see them there too. By utilizing these new leads from Power BI, you can potentially reach more customers and increase your business's growth. So be sure to check your Referrals workspace and CRM regularly to stay on top of your leads! Publish a Power BI Project (PBIP) directly from Power BI Desktop You are now able to directly publish your Power BI Project (PBIP) files directly from Power BI Desktop, eliminating the need to save it as PBIX in order to activate the Publish feature. From an opened Power BI Project, choose: File > Publish > Publish to Power BI or select Publish on the Home ribbon. Select the destination workspace: And that's it, your PBIP dataset and report (or just report if it's a Live Connect) will be published to the selected workspace: To learn more about Power BI Projects, visit: https://aka.ms/pbidesktopdevmode Visualizations New visuals in AppSource The following are new visuals this update: Performance Flow - xViz Time-lines by BI-Champ Composed Line Area Bar Chart by Devlup Funnels Galigeo For Power BI Radial chart by Devlup Funnels Editor's pick of the quarter accoPLANNING Enterprise - Planning Power BI Writeback Sunburst Chart by Powerviz Spider Chart for Power BI by VisioChart Advanced Trellis / Small Multiples - xViz Drill Down Combo PRO Zebra BI Cards Shielded HTML Viewer Drill Down Combo Bar PRO by ZoomCharts Drill Down Combo Bar PRO by ZoomCharts offers a wide selection of customization options, letting creators build everything from regular bar charts to box and whisker plots. This visual also offers powerful cross-chart filtering capabilities combined with intuitive on-chart interactions. MAIN FEATURES: Multiple chart types – choose between column, line, and area charts Full customization – customize X and Y axes, legend, outline and fill settings Stacking and clustering – choose normal, 100% proportional, or zero-based stacking Static and dynamic thresholds – set up to 4 thresholds Multi-touch device friendly – get the same experience on mouse and touch input devices POPULAR USE CASES: Sales and marketing – sales strategies, sales results, and campaign-by-campaign marketing metrics Human resources – hiring, overtimes and efficiency ratios by department Accounting and finance – financial performance by region, office, or business line Manufacturing – production efficiencies and quality metrics by product line ZoomCharts Drill Down Visuals are known for their interactive drilldowns, smooth animations, and rich customization options. They are mobile friendly and support: interactions, selections, custom and native tooltips, filtering, bookmarks, and context menu. Try Drill Down Combo Bar PRO now by downloading the visual from AppSource. Learn More about Drill Down Combo Bar PRO by ZoomCharts. Sunburst Chart by Powerviz We are thrilled to present the new Sunburst Chart by Powerviz, a powerful visualization designed to display hierarchical data in a user-friendly and intuitive format. With its concentric circle design, you can easily display part-to-whole relationships and gain valuable insights from your data. Key Features: Rich Customization: Control display style, labels, Center circle, fill patterns and dynamic images. Color Options: Choose from 30+ color palettes, including color-blind safe options. Ranking: Easily filter the Top/Bottom N by each level and show the remaining categories as "Others". Conditional Formatting: Easily identify outliers based on value or category rules. Interactive Features: Enjoy full interactivity with zoom, drill down, and cross-filtering for detailed exploration. Many other features included annotations, grid view, show condition, and accessibility support. Business Use Cases: Sales and Marketing: Market share analysis and customer segmentation. Finance: Department budgets and expenditures distribution. Operations: Supply chain management, identify inefficiencies in manufacturing process. Education: Course structure, curriculum creation. Human Resources: Organization structure, employee demographics. Check out our video Introducing Sunburst Visual by Powerviz – A Powerful Power BI Custom Visual Get Sunburst Visual for FREE from AppSource Check out the visual features in sample file. Step by Step instructions and documentation can be found here. To learn more, visit Powerviz website xViz Performance Flow by Lumel xViz Performance Flow by Lumel is an integrated business flow monitoring visual with an interactive KPI tree visualization for organizational performance management use cases. It offers insights into People, Places, Processes and Entities with Performance Indicators, Trendlines, and Advanced Alerting on Goals, Metrics and their Variances. Performance Flow is suitable for a wide range of use cases: Organization Performance - Unveil HR/Employee Performance insights through an interactive pictorial Org Tree Chart with performance attributes and trends. Process Flow - Visualize business flows in various stages using Swim lanes with connector lines, icons, KPI metrics, and trends in one single integrated view. Financial Performance – Cost Center analysis with Scorecards or KPI Trees. Sales Performance - Dive deep into Sales Performance at Regional, Departmental, and Salesperson Levels with Decomposition Trees. The visual is packed with analytical and interactive features like: Interactive hierarchy navigation including quick search, zooming, subtree analysis, etc. Streamline Processes using Swim Lanes like in Visio Uncover hidden/Dotted connections using Links Data-driven conditional formatting rules Custom Tabs for end-users Try xViz Performance Flow visual from Microsoft AppSource today. To know more about the visual refer to our website Performance Flow and our video xViz Performance Flow - Introduction Synapse Data Warehouse SSD Caching T-SQL queries targeting massive amounts of data not fitting in-memory cache suffer from cache misses and higher latency due to repetitive reads from remote storage. SSD caching stores frequently accessed data on local disks in highly optimized format, significantly reducing IO latency and accelerating query processing. For more information check https://learn.microsoft.com/en-us/fabric/data-warehouse/caching. Data Engineering High Concurrency Mode for Notebooks We are thrilled to announce session sharing in Fabric through High Concurrency mode for Data Engineering and Data Science workloads. You can run notebooks simultaneously on the same cluster without compromising performance or security when paying for a single session. Session sharing is strictly within a single user boundary offering enhanced security and isolation also allowing you to do more while paying less. To learn more about High Concurrency mode in Fabric for Data Engineering and Data Science check out the documentation : High Concurrency mode in Fabric Spark Data Science Model & Experiment Usability Improvements We’ve made several usability enhancements to our Model & Experiment tracking features. You can now stay informed with real-time notifications for model and experiment updates. Plus, users can now enjoy a more seamless browsing and comparison experience with improved Run List and Model List views. New Data Science Samples are Coming to Microsoft Fabric We're thrilled to announce that Microsoft Fabric is introducing two exciting new Data Science Samples that showcase the power of Microsoft Fabric capabilities. The first sample focuses on bank customer churn problem and aims to build a machine learning model to predict whether bank customers would churn or not. The second sample is about machine failure and revolves around the use of machine learning to have a more systematic approach to fault diagnosis to proactively identify issues and take actions before a machine's actual failure. Both these samples provide a comprehensive display of end-to-end data science workflow, demonstrating Microsoft Fabric versatility in addressing diverse real-world challenges with AI-driven solutions. We can't wait to see the incredible possibilities these samples will unlock for our customers! To check these new Data Science samples along with others, please visit Microsoft Fabric and then select Synapse Data Science. Click on Use a Sample to access all Data Science samples. Real-time Analytics KQL Database support for inline Python Fabric KQL Database supports running Python code embedded in Kusto Query Language (KQL) using the python() plugin . The plugin runtime is hosted in a sandbox, an isolated and secured environment hosted on KQL Database compute nodes. This sandbox contains the language engine as well as common mathematical and scientific packages. The plugin extends KQL native functionalities with a huge archive of OSS packages, enabling Fabric users to run advanced algorithms, such as machine learning, artificial intelligence, statistical tests, time series analysis and many more as part of the KQL query. The Python plugin runs a user-defined function (UDF) using a Python script. The Python script gets tabular data as its input, and produces tabular output. The plugin is disabled by default. Before you start, enable the Python plugin in your KQL database. To enable the plugin browse to your KQL Database, select Manage -> Plugins, and enabled the plugin by toggling the button to On. KQL Database provisioning in a few seconds The KQL Database provisioning process has been optimized. Now you can provision a KQL Database within a few seconds. All you need to do to Create a new KQL database, is to give it a name, and after a few seconds you will have a fully functional KQL Database, that you can now start ingesting and querying your data. Data Factory Dataflows Gen2 Power Query editor Edit connection in Manage Connections The Manage Connections feature was recently release with only the capability to view the linked connections to your dataflow and the ability to unlink a connection from your dataflow. We’ve now added the capability to Edit a connection credentials and gateway from within the dialog. We continue to work on improving this experience during this calendar year and we welcome you to give this feature a try and look forward to future improvements on this feature. Enable Staging labels The concept of staging data was introduced in Dataflows Gen2 for Microsoft Fabric and now you have the ability to define what queries within your Dataflow should use the staging mechanisms or not. Learn more about the staging mechanism used in Dataflows Gen2 from the Data Factory Spotlight blog post. Publish dataflow without staged queries With the introduction of the ability to set the behavior of a query to be staged or not, you can now set all your queries to be evaluated without any staging and load the data directly to a destination of your choice. Note that your Dataflow Gen2 must at least have a single query with a data destination defined. The main benefit of this pattern is that your data will not need to be staged first which could potentially save time if the evaluation of your Dataflow might be quick and doesn’t need the overhead of staging mechanisms or it simply goes more in line with how you want your solution to be designed for a number of reasons. Limit to number of queries in a Dataflow when publishing When creating a Dataflow Gen2, we are now modifying the maximum number of entities that can be part of a particular Dataflow. The new maximum number of entities is 50. If you have 51 or more entities in your Dataflow, you will receive a warning letting you know that you need to reduce the number of entities to a maximum of 50 before you can publish your Dataflow. You are still able to save your Dataflow as draft before you make any changes. Refresh history Direct navigation to last failed refresh If you are ever in a situation where your Dataflow refresh failed, you can now click the warning sign right next to the timestamp in the Refreshed column and get taken directly to the Refresh history dialog for that particular refresh attempt. The goal of this new improvement is to reduce the number of clicks to get you to the last failure. Statistics for tables Inside the refresh history dialog you are now able to drill down to a particular table and see the volume processed for it as well as the endpoint where the volume was processed. More information such as Duration, Start Time and End Time are still available to you in this dialog. Connectors Lakehouse connector update This update includes significant performance improvements to the Lakehouses connector. Be sure to update to the August version of Power BI Desktop and Gateway to experience these improvements! Other improvements We are actively listening to your feedback and the feedback of thousands of customers who are trying out Dataflow Gen2 today. Some of the feedback doesn’t directly translate to new features, but rather to fixes or quality improvements to our backend and how reliable our service can be. Our team has been able to triage and work on more than 600 fixes and improvements in the past month. The list below is a small list of some of the categories of these fixes and the impact that it’ll have on how you use Dataflow Gen2 today: Better error messages: We are actively improving the error messages and have improved some of the most common error messages in the past few weeks. There’ll be multiple improvements on the error messages and categorization front in the coming months but we’re happy to be making tactical changes today to improve the experience and make it clear to users on what could be happening to produce the error. Multitasking efforts: There have been some issues in regard to multitasking in Fabric. A few of those have been addressed, but we’re actively working on a much better experience for Dataflow Gen2 with multitasking in mind much similar to how other artifacts leverage multitasking capabilities. Reliability and performance: We’re continuously working towards improving the reliability and performance of Dataflow Gen2 in Microsoft Fabric. This will translate into much faster refreshes times and with more reliability. Data pipelines Connectors FTP Connector We’re excited to announce that the FTP connector is now available to use in your Data Factory data pipelines. In your data pipeline, you can create a new connection to your FTP data source to copy, extract, and transform your data. Productivity Lookup activity for Lakehouse, Data Warehouse, and KQL Database The Lookup activity now connects to Fabric Lakehouse, Data Warehouse, and KQL Database. This now makes it easier for you to read or look up records, table names, and other values from your Fabric artifacts to use in downstream activities in your data pipeline. Get Metadata activity for Lakehouse and Data Warehouse The Get Metadata activity now connects to Fabric Lakehouse and Data Warehouse, making it easy to retrieve metadata from your data in your Fabric artifacts to use downstream in your data pipeline. Pipeline run status now available We’ve recently added Pipeline run status so that developers can easily see the status of the pipeline run. You can now view your Pipeline run status from the Output panel. Secure input and output options for Set Variable activity We’ve added advanced settings for the Set Variable activity called Secure input and Secure output. When you enable secure input or output, you can hide sensitive information from being captured in logs. Data Activator Trigger and property design template As mentioned last month, we have been working on a new experience for designing triggers and it’s now available in our preview! You now see 3 cards in every trigger: Select, Detect, and Act: ‘Select’ is where you choose the value you want to monitor. It can be a direct reference to a column from an event stream, or to an existing property. Once you’ve selected an input, you’ll see a preview and can add grouping/smoothing or filters to get the right value: The Detect card is where you specify the conditions and thresholds that you want to take action on. You select the type of threshold, enter the values and can optionally choose things like firing the trigger every time the condition is met, or only when it’s met a certain number of times over a longer period. The top chart shows, for the instances you have selected in the preview, when this trigger would have fired. The bottom charts shows the overall nubmer of times the trigger would have fired for all instances. This gives you an idea of how many emails or Teams messages would have been generated! Finally the Act card lets you set the action you want Data Activator to take. You can choose the recipient, optional information etc. as you build it out. Quickly assign columns as properties in an object From the data view, you can quickly assign an event stream to a new or existing object and make multiple properties from one UI. You need to choose the key column that identifies the individual object instances that you care about (e.g. a package ID, employee ID, location name etc.). Then, in the ‘Assign properties’ dropdown, select the columns from your event stream that you want to use as properties in the object. You can use the ‘Assign to existing’ option to map a second event stream onto an existing object, combining data from two events. Data Activator now supports Power BI visuals with a time axis Data Activator now supports Power BI visuals with a time axis. In the screenshot below, we create an alert if the “occupancy” measure on the visual goes above 60%. Note that Data Activator has detected the presence of the time axis and is highlighting this in the alert pane. Trigger Power Automate flows from Data Activator When a data activator trigger fires, you can now trigger a Power Automate flow. This means that you can use Data Activator to drive actions in any system that Power Automate can connect to. You can send an alert in a 3rd party alerting system, log a ticket in a ticketing system, or call a REST API to trigger actions in an operational system. The list is (almost) endless! To connect Data Activator to Power Automate, you create a custom action. A custom action is a reusable action template that triggers a flow. Once you have made a custom action, you can use it in any trigger, in any Data Activator workspace or item. Here, we create a custom action to send an SMS message via a 3rd party connector: After creating the custom action you can use it in any trigger, by selecting it in the “Act” card in the trigger designer. The act card prompts you for the input fields that need to go to the flow. Here, we see the action card for our SMS action. It is prompting for the phone number and message: For the trigger creator, the action works just like natively-supported email and Teams actions. This means that you can get a Power Automate expert in your organization to create the custom action, then roll it out to all Data Activator users, even they don’t have any Power Automate experience. That is all for this month! Please continue sending us your feedback! If you haven’t already head over to the Fabric Community to join in the conversation!Microsoft Fabric September 2023 Update
Welcome to the September 2023 update. We have lots of features this month including updates to the monitoring hub, Fabric Metrics app, VS code integration for Data Engineering, Real-time data sharing and many more. Continue reading for more details on our new features! Contents Core Monitoring hub – column options OneLake OneLake file explorer (v.1.0.10) - Menu Option to View Workspaces and Items Online Power BI General Upgrade Power BI Desktop .NET Framework to 4.7.2 or newer Reporting Mobile layout interactive canvas Smart Narrative improvements Modeling Edit your data model in the Power BI Service - Updates Edit linguistic relationships in Q&A setup DAX function changes to MINX and MAXX Edit relationships in the properties pane (GA) Data Connectivity SAP HANA (Connector Update) Emplifi Metrics (New Connector) Service Row-level security test as role improvements Mobile Supporting AAD Shared device mode (preview) Developers Git integration Visualizations New visuals in AppSource Change Chart Beeswarm Chart Explore time-based data down to each millisecond with Drill Down TimeSeries PRO New Updates for accoPLANNING (Release 57) Zebra BI Charts on-visual settings Word Cloud by Powerviz Innofalls Charts: A Versatile and Interactive Visual Hierarchy Chart by MAQ Software Power BI JSON Report Theme Generator by BIBB Synapse Data Warehouse Data Warehouse Utilization Reporting in Fabric Metrics app Column-Level security is now available on Data Warehouse and SQL Endpoint Data Warehouse Row Level Security SQL Projects support for Fabric Data Warehouse Deployment Pipelines Default Dataset Improvements Data Engineering VS Code Integration Introducing Notebook File System support in Synapse VS Code extension for Data Engineering and Data Science workload in Microsoft Fabric Import Notebook in Workspace view Mssparkutils new API for fast data copy Support diff versions when saving notebook Notebook Sharing with Execute-only mode Notebook resources .whl file support Real-time Analytics OneLake shortcut to delta tables from KQL DB Real-Time Data Sharing: Introducing database shortcuts in Real-Time Analytics Sample Gallery – Explore what you can achieve in Real-Time analytics Model and Query data as graphs using Kusto Query Language (KQL) Easily connect to KQL Database from Power BI desktop Eventstream now supports AMQP format connection string for data ingestion Eventstream supports data ingestion from Azure IoT Hub Improved Eventstream Creation Speed Data Factory Data Pipeline Dataflow Gen2 New Service Principal authentication kind Bug fixes and reliability improvements Community New learning path: Implement a Lakehouse with Microsoft Fabric Core Monitoring hub – column options We have released a new feature inside the monitoring hub to better users customize their experience. Column options gives users more room to operate. Users can select and reorder the columns that meet the scene according to their customized needs. Regardless of whether the user switches to any other tool or scene in Fabric, these columns options are persisted along with the filter until the user returns again. You can now easily reorder the columns with drag and drop inside the option menu. You do not need to worry about the ones at the bottom. All the selected column options will be bubbled up to the top of the menu when you open it next time. Try it out yourself! OneLake OneLake file explorer (v.1.0.10) - Menu Option to View Workspaces and Items Online We are excited to announce the new release of OneLake file explorer for Windows! Now you can seamlessly transition between using the OneLake file explorer app and the Fabric web portal. When browsing data in OneLake file explorer, right click on a workspace and select “OneLake->View Workspace Online.” Selecting this will open the workspace browser on the Fabric web portal. Similarly for items within your workspace. In addition, you can now easily find your client-side logs, which you may need to troubleshoot issues. Right-click on the OneLake icon in the Windows notification area, select Diagnostic Operations, then Open logs directory. Get started by downloading the latest OneLake file explorer. Power BI General Upgrade Power BI Desktop .NET Framework to 4.7.2 or newer Power BI Desktop with .NET Framework 4.5 will no longer be supported after 9/30/2023. Please upgrade to .NET Framework 4.7.2 or newer to avoid any support issues. Reporting Mobile layout interactive canvas With this latest update we are happy to announce that we have made the mobile canvas interactive. This new capability provides the ability to test how buttons, slicers, and visuals will behavior on the app before publishing the report. But that's not all. With this canvas interactivity, users can now interact with visuals directly and adjust Table and Matrix column headers to align perfectly with mobile screens. Smart Narrative improvements The Smart Narrative visual can now display either text or numeric values when showing data for a column. Previously this was limited to only numeric values. Learn more about the Smart Narrative’s summarization capabilities at Create smart narrative summaries - Power BI | Microsoft Learn. Modeling Edit your data model in the Power BI Service - Updates The new data model editing in the Service feature was released to preview in April. We’ve been busy reacting to your feedback and enhancing the experience. Below are the improvements coming later this month: Improve layouts limitations Changes you make to layouts will now persist between Desktop and the Service, including: Changes made to data model layouts in Desktop will now be incorporated into the Service upon uploading the .PBIX file. Similarly, changes made to layouts in the data model within the Service will now be incorporated in the Desktop upon downloading the .PBIX file. Please continue to submit your feedback directly in the comments of this blog post or in our feedback forum. Edit linguistic relationships in Q&A setup The Q&A visual is an effective way to help users further understand their data by asking questions and receiving answers in the form of visuals. It offers users a way to explore their data in ways not covered by the rest of the report without requiring deeper knowledge of their data model or report authoring. However, while the Q&A engine is good at answering precise questions about data, it may not be able to associate every word or phrase a user inputs with data in the model. For example, answering “what are our best consoles this year?” may require connecting the term consoles to the name products in the model, and understanding that the term best corresponds to the highest sales values. These terms are contextual, however – users could mean something completely different asking for console and best in other industries, organizations, or even datasets. To help authors ensure that the Q&A visual provides consistent and accurate answers based on the unique language their report consumers actually use, we introduced Q&A setup tools with an emphasis on providing Q&A with synonyms for column and table names in the model. This way, authors can explicitly define console as referring to products, and users will always receive the correct answers when they ask similar questions in the future. However, synonyms (nouns) are only half of the picture. The other half of the terms (adjectives, verbs, prepositions, adverbs) can’t be defined with such straightforward mappings because they must be understood as a part of a phrase – they qualify other terms or relate other terms together. Best in the previous example is one; asking “who sold the most books” requires us to know that stores sell books, connecting stores to books. There are many types of these linguistic relationships, so we built a new tab entirely to help you create and manage linguistic relationships for your data. You can get into the Q&A setup menu using the gear icon on the Q&A visual or the Q&A setup option in the Modeling tab of the ribbon, then selecting the new Relationships tab. There, you’ll be able to define a variety of relationships, including verb, adjective, noun, preposition, and more. Choose a type which fits the term you’re trying to define (for example, “best” is an adjective), then follow the prompts to define what it means in the context of your data. Our investment into Q&A does not stop here. Even in a world where natural language capabilities are increasingly driven by large language models, there is value in the precision, consistency, and customizability of our sophisticated Q&A engine. In the other direction, defining synonyms and relationships can be a lengthy process just asking to be streamlined with the power of AI-generated suggestions. Keep an eye out in the future for the ways we’re bridging the two to bring out the best of both worlds! DAX function changes to MINX and MAXX We have added an optional variant parameter to MINX and MAXX DAX functions. These functions ignore text and Boolean values when there are variants or mixed data types, such as text and numeric. Now with the new optional variant parameter set to TRUE, the functions will consider the text values. Here is an example of a variant measure. In the table below the variant measure is ordered in ascending order: MINX and MAXX without the variant parameter set, or set to FALSE, will ignore text and Boolean data types. MINX and MAXX with the variant parameter set to TRUE will now include text values. Boolean values are still ignored. The default of the MINX and MAXX optional variant parameter is FALSE, so all existing DAX expressions using MINX and MAXX will not be impacted by this change. To allow text values to be included, you can update the DAX expression to include TRUE as the third parameter. Read more about these changes at Microsoft Learn: MINX function (DAX) - DAX | Microsoft Learn MAXX function (DAX) - DAX | Microsoft Learn Edit relationships in the properties pane (GA) We are excited to announce the general availability of the edit relationships in the properties pane! The edit relationships in the properties pane has been available for public preview since the October 2022 release. Now you can click on any relationship line in the modeling to edit your relationship in the properties pane. This is available in the Model view of Power BI Desktop and in web modeling of the Power BI service. Edit relationships without running queries to preview the data and only validating the relationship when you click apply changes. A welcome relief for those using DirectQuery storage mode, as these queries can take time and impact your data sources. You can learn more about how to utilize this feature at Create and manage relationships in Power BI Desktop - Power BI | Microsoft Learn. Data Connectivity SAP HANA (Connector Update) The update enhances the SAP HANA connector with the capability to consume HANA Calculation Views deployed in SAP Datasphere by taking into account SAP Datasphere’s additional security concepts. This enables consumption of Calculation Views in Datasphere and allows customers to connect to HANA Cloud views without the need for additional privileges on the _SYS_BI schema. Emplifi Metrics (New Connector) We are happy to announce the release of the new Emplifi Metrics connector. Please find release notes from the Emplifi team below: “Integrating social media insights alongside the rest of your marketing or business intelligence data gives you a holistic understanding of your entire digital strategy, all in one place. With Emplifi Power BI Connector, you’ll be able to include social media data from the Emplifi Platform in your charts and graphs and combine them with other data you own. The Power BI Connector is a layer between Emplifi Public API and Power BI itself. It helps you work with your data intuitively, directly in the Power BI tool. The majority of data and metrics available in the Emplifi Public API are also available in the Connector. Please visit the official documentation for more information about Emplifi Public API and a list of available metrics. You’ll find it here: https://api.emplifi.io/.” Service Row-level security test as role improvements We have listened to your feedback about the row-level security test as role experience in the Service and have made several improvements this month including: We have added a new dropdown, allowing you to choose and test any report connected to the dataset. You can now easily see important permissions details pertaining to a specific individual while role testing for that person. We have improved our warning and error messages. To learn more about validating row-level security roles in the Service read our documentation. Mobile Supporting AAD Shared device mode (preview) We, in Power BI continue our investments in empowering frontline workers with data and insights in their work. And in this month release, the Power BI mobile apps add support in Azure active directory shared device mode. Shared Device Mode enables you to configure devices to be shared by a group of employees. This is very common for frontline workers, who do not use their personal device for work related tasks but are getting a work-related mobile device from a shared pool of devices for these tasks. The Shared Device Mode enables single sign-on (SSO) and device-wide sign out for Microsoft Power BI and all other apps that support Shared Device Mode. Once a user signs in into a supported app, the user will be logged into any application compatible with Shared Device Mode as well. Once a user signs out of a supported app, all other applications integrated with shared device mode sign out, to to prevent unauthorized or unintended access by the next user. Back to the frontline worker scenario: when an employee picks a device from a pool at the start of their shift, they need only a single sign-in to one supported app (for example Power BI), and they’re automatically signed in to all other apps that support shared device mode on the device, so the device and the apps are ready for work with the user’s account. At the end of their shift, when they sign out of one app, they're signed out globally from all other apps that support shared device mode, having the device is ready for the next employee and can be safely handed off. Developers Git integration Paginated reports now supported with git integration Since Git integration has been launched few months ago, we supported only 2 items in Power BI- Power BI reports and Power BI datasets. We are now excited to add a new item- Paginated reports! After connecting your workspace to Azure DevOps, you can commit your paginated reports and have them versioned in your repository. After that, you can open the .rdl files directly from git in Power BI Report Builder, edit and push the changes into git. The workspace will identify the changes and will prompt developers to update the workspace with the changes to the paginated report. With this feature, we are adding paginated reports developers to enjoy the collaboration, versioning and modern developer workflows offered in Power BI and Fabric. Learn more about using git with paginated reports. Checkout and Switch Branches in a Workspace A workspace can connect to a single branch at a time. Once you’ve setup your separate workspace to work with git, there might be scenarios where you want to change just the connection of a branch. We have now added 2 new features to help you achieve this much faster: Checkout branch- checkout allows you to easily create a new branch, based on the current state of the workspace. After choosing to checkout, you can create a new branch that will be connected to the WS, while the uncommitted changes are retained. This is useful in cases of conflicts, that allows you to commit your changes to a backup branch, and then manage the merge conflict in the git repo. Switch branch- Workspace admins can decide to change just the connection of the branch very easily. It’s helpful in cases you want to start working on a new branch and wipe clean your WS content for that, or when you are moving from your regular project to small bug fixes (assuming they are on the same content), that should happen on a different branch. New E2E CI/CD tutorial now available Many users have been asking, since the release of git integration, how to work with deployment pipelines and git as part of a CI/CD process. These two features can help build a compelling e2e flow, when connected properly. Follow this step-by-step guide to learn how to make the most out of both tools together. Visualizations New visuals in AppSource Change Chart Pro Circle Card by Devlup Funnels Apex Milestone Trend Dot Chart Activity Gauge by Powerviz Bridger Visual by BI Samurai Number lines by BI-Champ Funnel Chart by Powerviz Date Picker by Powerviz Bar Chart with Export Data Feature Wordcloud by Powerviz PackedBubbleChart Box Ploty by Devlup Funnels LeapLytics - Markdown Viewer Aimplan Data Input Table Beeswarm Chart Overview of Beeswarm Chart Beeswarm chart shows the distribution of data along one axis or both axes while also showing individual points. It is like a one-dimensional or two-dimensional scatter plot but with closely-packed, non-overlapping points. How to use this Visual The visual is intuitive and easy to follow. Only 1 field is mandatory i.e. Bee Category. Tip: If you don’t have any category, you can simply add a calculated column with any name and use it in the category field Bee Size field is optional and is represented in the visual by bubble area or size. If this field is missing then all bubbles will be of same size Bee Label field is also optional. It is used to add labels to each individual bubbles. This chart can be used in 3 ways Y – Axis only X – Axis only Scatter Plot (Both Y and X Axes) 1) Y-Axis Only As shown in picture below, if you only use a column/field in “Beeswarm Y-Axis” section while keeping “Beeswarm X-Axis” section empty, your data will be plotted on Y-Axis categorized according to the “Bee Category” section 2) X-Axis Only Similarly, if you only use a column/field in “Beeswarm X-Axis” section while keeping “Beeswarm Y-Axis” section empty, your data will be plotted on X-Axis categorized according to the “Bee Category” section 3) Scatter Plot (Both X and Y Axes) If you use columns/fields in both sections i.e “Beeswarm X-Axis” section and “Beeswarm Y-Axis” section, the visual will work like a scatter plot with data points plotted on X-Axis and Y-axis . This is depicted below You can show images inside bubbles as well using the “Images” field. For example, the report below shows top millionaires of the world with their net worth on Y-Axis and their Age on X-Axis. Its available in demo file. Formatting Options These options are available in the format pane of the visual under “Settings”. 1) Bubble size As the name implies, this setting simple increases / decrease the bubble size 2) Lower Upper and Left Space Beeswarm chart produces non colliding bubbles or in other words circles don’t overlap. This can often cause bubbles do go outside the chart area especially when there is a concentration of data around minimum and maximum data values. This is where these settings are very useful to bring data inside the chart area. These settings increase or decrease the Y-Axis and X-Axis range to achieve this objective. Try adjusting these settings on the demo file’s sheet “ScatterPlot Beeswarm” to see how they work. Purchase of Premium Features To use the visual without Watermark in Power BI Online, please purchase the Visual’s license for a 1 year period. The license is for unlimited number of users and viewers. If you have any questions, please send me email at [email protected] Download Download the demo file here. Download the custom visual from APPSOURCE Explore time-based data down to each millisecond with Drill Down TimeSeries PRO Drill Down TimeSeries PRO lets you create timeseries charts that are easy to explore on any device. Using its intuitive interactions, users can drill down to months, days or hours by clicking on the chart. Combine up to 12 series and choose between multiple charts - line, column, area. Learn more about Drill Down TimeSeries PRO! MAIN FEATURES: On-chart interactions – pan, zoom and drill down Full customization – set intervals, min/max values, colors, fonts, stacking and clustering Static and dynamic thresholds – set up to 4 thresholds to demonstrate KPIs Cross-chart filtering – select data points on multiple charts instead of using slicers Touch device friendly – explore your data anywhere POPULAR USE CASES: Finance – stock exchange indices, capital ratios, transaction volumes Sales – web traffic, audience reach, lead volume IT – network traffic, response times, syslog and error trends Logistics – inventory movement and turnover, loading time, fleet maintenance costs Get Drill Down TimeSeries PRO now! This visual comes with 30 days free access to paid features. ZoomCharts Drill Down Visuals are known for interactive drilldowns, smooth animations and rich customization options. They are mobile friendly and support: interactions, selections, custom and native tooltips, filtering, bookmarks, and context menu. New Updates for accoPLANNING (Release 57) accoPLANNING for Power BI empowers business users with writeback capabilities, advanced planning, forecasting, budgeting, project management and analysis solutions. We have some new and exciting features for you in this release: • Custom calculated rows and columns configuration. It allows you to customize your own client-side calculations in your already established accoPLANNING table. • End user Lock cells. This gives the flexibility for the end user to lock cells and make sure these will not be affected by splashing or use as an approval indicator. • Hide rows and columns. This Allows users to hide rows and columns - adding flexibility in making asymmetric column and row selections in the grid for better reporting or just for ad-hoc purpose to get a better overview. The latest update also includes a range of new features and improvements designed to increase your productivity: • Our navigation has undergone a major overhaul, with the most notable change being the revamped toolbar. • Better API error message handling in the grid. • Support for automatically expanding all rows/columns/both, eliminating the need to manually adjust the size of each cell. With the accoPLANNING visual, you combine the planning and reporting process in Power BI. For more information, visit our website. https://www.accotool.com/ https://appsource.microsoft.com/en-us/product/power-bi-visuals/WA200002600?tab=Overview https://youtu.be/kNIVC6rBJWA Zebra BI Charts on-visual settings Zebra BI visuals for Power BI are taking another step further to make the user experience as seamless and interactive as possible. Zebra BI Charts has received some important updates so that you save time while customizing your visuals. Thanks to the on-visual settings, you don’t need to go to the visualizations pane every time you want to make a change. COMMENTS SETTINGS can now be adjusted directly on the visual by simply clicking on the settings icon next to the comment box. You can adjust the title, which variances you display, the icon, gap between the comments, and padding. CATEGORY SETTINGS let you adjust several settings by just clicking on the category area. Customize the axis font (family and color), trim /rotate long labels, adjust the label density, and set the gap between the columns. LEGEND SETTINGS come in handy when you want to rename the entries and don’t want to search for this option in the formatting pane. Adjust margins, use aliases in tooltips and switch comparisons. STACKED CHARTS SETTINGS on the visual let you adjust the Top N feature, set color of the chart, and display labels as %. Try it on your data for free. Word Cloud by Powerviz We are excited to announce the new Word Cloud by Powerviz, it's an advanced visual which empowers you to create some of the most high-quality and creative word art in the Power BI. Key Features: Word Styling: Make your word clouds pop with personalized text styles. It offers font styling, direction & text editing features. Color Options: Choose from 30+ color palettes, including color-blind safe options. Shapes: Want to make a statement? Create captivating word clouds by choosing shapes from icons and images, or upload your own image. Exclude: No more hassles anymore! Easily remove unwanted words, symbols from the text to create a clean and focused word cloud. Ranking: Filter out Top/Bottom N Words. Conditional Formatting: Easily spot words with dynamic rules. Many other features included lasso/reverse lasso, grid view, show condition, and accessibility support. Business Use Cases: Marketing: Sentiment analysis & SEO keywords. Education: Brainstorming, Improve engagement. Market Research: Opinion poll, open-ended survey. Presentation: Capture attention & easy communication. Try Word Cloud Visual for FREE from AppSource Check out the visual features in demo file Step by Step instructions and documentation To learn more, visit Powerviz website. Introducing Word Cloud by Powerviz – A Powerful Power BI Custom Visual on YouTube Innofalls Charts: A Versatile and Interactive Visual Innofalls Charts is a powerful visual that offers a wide range of interactive features. Our bar and waterfalls charts come with a drill-down history, enabling orientation and facilitating comparisons. All charts can be stacked or grouped into columns. Waterfalls: Our special attention to waterfalls includes running totals and delta waterfalls, enabling effective comparisons of measures or members. The waterfalls are expandable, drillable, stackable, and offer intermediate sums. Drill Down History: Keep track of your drill downs with visual feedback. Change your drill downs retroactively and combine multiple drill downs for in-depth exploration. New Comparison Features: Effortlessly make comparisons while viewing your report and incorporate deviation charts with a single click. Interactivity: Enjoy various interactive features such as drill, expand, undo/redo, dynamic comparisons, and zoom for enhanced data exploration. IBCS Formatting: Our IBCS theme provides scenario formatting, equal scaling, highlights, deviations and more for consistent and standardized reports. Animations: Enhance data comprehension and reduce change blindness with animations. Promptly detect resorted or added data elements. Discover the full potential of Innofalls Charts and get started today. Innofalls Charts is free for Power BI Desktop! Get started today [ https://www.innofalls.com/blog-get-now ] Visit our website [ https://www.innofalls.com/ ] Hierarchy Chart by MAQ Software Struggling to visualize complex hierarchical relationships? Improve understanding and decision-making with the Hierarchy Chart by MAQ software. In today's data-driven environment, visualizing hierarchical relationships is key to business insights and decision-making. From sales structures to budget allocations, our Power BI-verified visual offers a customizable way to represent these structures easily, catering to various business needs. Figure 1: Sample visual showing a color-coded organization chart (with a tooltip). Key business uses: Organization/HR: Organize human resources by department and hierarchy. Operations: Illustrate manufacturing processes, breaking down components or ingredients hierarchically. Finance: Show budget allocations by division or project, with color-coded subcategories to highlight differences. Sales: Visualize sales structure, territory responsibilities, and targets with group field indicators showing performance status. IT: Visualize IT assets by category and availability with detail. Key features: Color-code cards using a legend. Adjust card (i.e. box) size, borders, and corners. Customize the appearance of the links connecting each card. Interact with cards to control levels and cross-filter visuals. Zoom and reposition the visual with ease. Try out the features of the Hierarchy Chart by MAQ Software today on the visual’s sample report. Learn more about our Power BI custom visuals on our website. Power BI JSON Report Theme Generator by BIBB https://powerbithemegenerator.bibb.pro/ BIBB recently released its take on the Power BI Theme Generator, aiming to streamline the theming process for users. This new tool offers a user-friendly interface, allowing for an easy selection of colours and generation of JSON themes. Within BIBB's generator, users can choose colours in various ways, from manual selection to trending combinations and even importing from images or external sources like Coolors. Synapse Data Warehouse Data Warehouse Utilization Reporting in Fabric Metrics app We are excited to announce that Compute utilization reporting for Data warehouse and SQL Endpoint are now available in Public Preview for Microsoft Fabric across all regions! In the capacity-based SaaS world of Microsoft Fabric, customers can purchase a Fabric SKU and get entitled to a set of Capacity Units (CU). Various workloads, including data warehouse, consume capacity units based on usage. Fabric metrics app provides visibility into capacity usage for all Fabric workloads, including data warehouse in one place. It is used by Capacity or SQL warehouse admins to identify CU usage trends across warehouse items within a capacity, monitor overload information, and understand the cost of running operations which leads to informed capacity sizing decisions. For more information, see the detailed blog Data Warehouse Utilization Reporting in Fabric Capacity Metrics App Column-Level security is now available on Data Warehouse and SQL Endpoint We are excited to announce that Column-Level security is now available on Data Warehouse and SQL Endpoint. Column-level security simplifies the design and coding of security in your application, allowing you to restrict column access to protect sensitive data. For example, ensuring that specific users can access only certain columns of a table pertinent to their department. The access restriction logic is located in the database tier rather than away from the data in another application tier. The database applies the access restrictions every time data access is attempted from any tier. This restriction makes your security more reliable and robust by reducing the surface area of your overall security system. In addition, column-level security also eliminates the need for introducing views to filter out columns for imposing access restrictions on the users. You can implement column-level security with the GRANT T-SQL statement. Only Azure Active Directory authentication is supported. Data Warehouse Row Level Security We are thrilled to announce that Row-Level security is now available in Fabric Warehouse and SQL Endpoint. Row-Level Security enables you to use group membership or execution context to control access to rows in a database table. Row-Level Security (RLS) simplifies the design and coding of security in your application. RLS helps you implement restrictions on data row access. For example, you can ensure that workers access only those data rows that are pertinent to their department. Another example is to restrict customers' data access to only the data relevant to their company. The access restriction logic is located in the database tier rather than away from the data in another application tier. The database system applies the access restrictions every time that data access is attempted from any tier. This makes your security system more reliable and robust by reducing the surface area of your security system. Implement RLS by using the CREATE SECURITY POLICY Transact-SQL statement, and predicates created as inline table-valued functions. SQL Projects support for Fabric Data Warehouse We are excited to announce the Microsoft Fabric Data Warehouse as a supported target platform in the SQL Database Projects extension available inside of Azure Data Studio! SQL Database Projects for Azure Data Studio provides a way to design, edit, and publish schemas for SQL databases from a source controlled project. For a complete development workflow, build and deploy your database projects in CI/CD pipelines, such as GitHub Actions or Azure DevOps. A SQL project is a local representation of SQL objects that comprise the schema for a single database, such as tables, stored procedures, or functions. Use the SQL Database Projects to extract and publish warehouse schemas directly from and to Fabric Data Warehouse. Other compatible databases include SQL Server, Azure SQL Database, Azure SQL Managed Instance, and Azure Synapse SQL (serverless and dedicated). Get started today with the insiders build of ADS. Deployment Pipelines In today’s world, analytics is a vital part of decision making in almost every organization. Fabric's deployment pipelines tool provides BI creators with a production environment where they can collaborate to manage the lifecycle of organizational content. Deployment pipelines enable creators to develop and test content in the service before it reaches the users. Supported content types include reports, paginated reports, dashboards, datasets, dataflows, and now warehouses! The deployment process lets you clone content from one stage in the deployment pipeline to another, typically from development to test, and from test to production. The connections between the copied items are kept during the copy process as well. In addition, Fabric applies the configured deployment rules to the updated content in the target stage. You can also deploy content programmatically, using the deployment pipelines REST APIs. You can learn more about this process in Automate your deployment pipeline using APIs and DevOps. Default Dataset Improvements We have released some new capabilities to enhance the default dataset experience: Turn off automatically adding new objects to the default dataset – navigate to the Warehouse artifact settings to toggle this on/off Updates to the manage default dataset dialog in the model view Filter by schema Filter by object (table/view) We’ve heard your feedback about the ability to turn off the default dataset. Thank you for the feedback, we’re planning on delivering this feature in the coming months! Data Engineering VS Code Integration Introducing Notebook File System support in Synapse VS Code extension for Data Engineering and Data Science workload in Microsoft Fabric The Synapse VS Code extension empowers users to develop their notebook artifacts directly within the VS Code environment. Once users download the .IPYNB file to their local desktop, they gain the flexibility to edit, debug, and execute their notebook code locally. Furthermore, they have the option to select the kernel provided by the extension to execute and debug their notebook code on the remote Fabric Spark compute. Python developers commonly structure reusable functions as modules in the form of .py files, a well-established software engineering best practice. Now, with the incorporation of the Notebook File System within the VS Code Synapse extension, developers can apply this principle to their Fabric notebook development. In addition to running and debugging notebook code, developers can also execute and debug code from imported modules. This integration enhances code modularity and facilitates efficient development workflows. When a user clicks "Open Notebook Folder," the VS Code Synapse extension not only opens the selected .IPYNB file but also downloads all associated files, including .PY modules, from the notebook file system to the local VS Code environment. This feature ensures that the entire set of necessary files and modules is readily available for seamless development and collaboration within the VS Code interface. In this example, there is a .PY module named US2Europe.py which contains a function to covert the datetime format from US fashion to Europe’s one. To invoke this function from the notebook side and execute it on the worker node, you need to import the module containing the function and apply into the data frame To synchronize changes between the local environment and the remote workspace in the VS Code Synapse extension, users can take advantage of two key actions: Publish Resource Folder: Clicking the "Publish Resource Folder" button enables users to upload their local changes to the remote workspace. This ensures that any modifications made locally are reflected in the remote environment. Update Resource Folder: The "Update Resource Folder" button allows users to pull changes from the remote workspace into their local VS Code environment. This ensures that any updates made by collaborators or on the remote side are mirrored locally. In the event that a file has been deleted in the remote workspace but still exists in the local environment, the Synapse extension will automatically handle this situation as follows: The deleted file will be moved to a backup folder named "__backup" under the root directory of the building folder. This approach preserves a copy of the deleted file for reference, ensuring that no data is lost during the synchronization process. These actions provide users with an efficient and robust means to manage the synchronization of their files and code between local and remote environments while maintaining data integrity. Import Notebook in Workspace view We are excited to announce that we now have the “Import Notebook” entry on the Workspace -> New -> Import item! You can easily import one or more files (support .ipynb, .py, .sql, .scala, .r file format) as Fabric Notebook items in the target workspace. Mssparkutils new API for fast data copy We now support a new method in mssparkutils that can enable large volume of data move/copy much faster, which is: Mssparkutils.fs.fastcp() You can use mssparkutils.fs.help("fastcp") to check the detailed usage. According to our benchmark testing, we achieved ~7x to ~180x speed up (varies as different source type, data size and regions) compares to the traditional file system copy method mssparkutils.fs.cp(). As the below example shows, it takes ~5s to copy a 10G file from ADLS Gen2 account to Fabric Lakehouse. Support diff versions when saving notebook We now support viewing and comparing the differences between 2 versions of the same notebook when there are saving conflicts. This is particularly useful when more than one users are working on a same notebook instance via different clients, like: Publish notebook from Fabric VSCode extension Some users are using Manual save mode while others are collaborating Collaborating on notebook with unstable network With the diff function the users in collaboration session can get notified and compare the changes from other user’s edits, and they can choose a version to keep, or save a copy of any version, to easily handle the save conflicts without losing the code accidently. Notebook Sharing with Execute-only mode Fabric Notebook now supports checking the “Run” operation separately when sharing a notebook, if you just selected the “Run” operation, the recipient would see a “Execution-only” notebook after opening the item. With the “Execution-only” mode you can easily spread your notebook without worrying about someone messing up the contents, while the recipients can still run the cells. Notebook resources .whl file support We now support uploading the .whl files in the Notebook Resources explorer, you can put your own compiled libs here, use Drag & Drop code snippet to install them in the session and import the libraries in code conveniently. Real-time Analytics OneLake shortcut to delta tables from KQL DB Now you can create a shortcut from KQL DB to delta tables in OneLake, allowing in-place data queries. With this enhancement, you can effortlessly query delta tables in your Lakehouse or Warehouse directly from KQL DB. Learn more. Real-Time Data Sharing: Introducing database shortcuts in Real-Time Analytics A database shortcut in Real-Time Analytics is an embedded reference within a KQL database to a source database in Azure Data Explorer allowing in-place data sharing. The behaviour exhibited by the database shortcut is similar to that of a follower database. If you are a data provider hosting data in Azure Data Explorer and want to share this data with consumers in Microsoft Fabric, you can create a database shortcut. Database shortcut enables sharing in real-time, with no need to build or maintain a data pipeline. All database changes, including schema and data, on the provider side are instantly available on the consumer side. You can then consume data in Microsoft Fabric by using a KQL Queryset, PBI, Notebooks etc. easily . While today, you can share data from an Azure Data Explorer database with a KQL Database, soon you will also be able to share data from a KQL Database with other KQL Database(s) in Microsoft Fabric. Check out the announcement blog to learn more: Real-Time Data Sharing in Microsoft Fabric blog Sample Gallery – Explore what you can achieve in Real-Time analytics Real-Time analytics now offers a comprehensive sample gallery with multiple datasets allowing you to explore, learn and get started quickly. Selecting a sample in Real-Time Analytics automatically creates a KQL Database with data and prewritten KQL Queryset from any of the 6 sample datasets. This will let users to get started with KQL Database and KQL Queryset without the hassle of creating database, finding the right real time dataset, ingesting data, and writing queries. Each sample lets you explore different streaming and real time scenarios. The samples include data and queries for Stock analytics, Weather analytics, IOT analytics, Log analytics, Metrics analytics and Automotive operations analytics. Each dataset is accompanied by a KQL Queryset which includes sample queries introducing you to KQL capabilities such as. aggregations, search , transformations using update policies, user defined functions, geospatial and time-series analytics. Model and Query data as graphs using Kusto Query Language (KQL) We are excited to share with you a new public preview feature of Kusto Query Language (KQL) that allows you to model and query data as graphs. Graphs are useful for representing complex and dynamic data that involve many-to-many, hierarchical, or networked relationships, such as social networks, recommendation systems, connected assets, or knowledge graphs. By using KQL with graph semantics, you get the speed and scale of KQL queries with the flexibility and expressiveness of graphs. You can query graphs using KQL graph semantics, which has a simple and intuitive syntax that works well with the existing KQL features. You can also mix graph queries with other KQL features, such as time-based, location-based, and machine-learning queries, to do more advanced and powerful data analysis. For example, you can use time-aware graphs to examine how the graph evolves over time, or use geospatial queries to analyze the spatial distribution or proximity of nodes and edges. To learn more about graph semantics in KQL, check out the following resources: - Introduction to graph semantics in KQL: https://aka.ms/kusto/graph-overview - Graph operators and functions reference: https://aka.ms/kusto/graph-operators We hope you enjoy this new feature and find it useful for your data analysis needs. As always, we welcome your feedback and suggestions on how to improve KQL and its graph semantics extension. Happy querying! Easily connect to KQL Database from Power BI desktop Power BI desktop just released new ways to easily connect to a KQL database. There are two ways to easily locate your KQL Database: Open the Get Data dialog and select the Microsoft Fabric menu item. Select KQL Databases from the list: Open the OneLake data hub dialog from the main ribbon. Select KQL Databases from the list: Eventstream now supports AMQP format connection string for data ingestion AMQP, stands for Advanced Message Queuing Protocol, is a protocol that supports a wide range of messaging patterns. It's a widely used communication protocol that facilitates efficient and reliable communication among IoT devices and systems. Now in Eventstream, you can create a Custom App source or destination and select AMQP format connection string for ingesting data into Fabric or consuming data from Fabric. Eventstream supports data ingestion from Azure IoT Hub Azure IoT Hub is a cloud-hosted solution that provides secure communication channels for sending and receiving data from IoT devices. With Eventstream, you can easily stream your Azure IoT Hub data into Fabric and perform real-time processing before storing it in Kusto Database or Lakehouse. To add an Azure IoT Hub source, follow these three simple steps: Choose "Azure IoT Hub" as your eventstream source. Give this source a name and enter your Azure IoT Hub details. You may need to enter the share access key from your Azure IoT Hub to establish the cloud connection. On the canvas, select the IoT source and click "Data Preview" to get a preview of the incoming data from your Azure IoT Hub. Improved Eventstream Creation Speed In response to user feedback, we've made significant improvements to the Eventstream creation, enabling you to create an Eventstream item in seconds. Setting up your Eventstream is now faster and smoother than ever before. These improvements save you valuable time and allow you to ingest and transform your data streams more efficiently. Data Factory Data Pipeline The Outlook email & Teams activities in Data Factory Pipelines can now be used in collaborative shared pipeline New pipeline activities are now available as well! If you’ve used the Custom activity type in ADF pipelines in the past, you’ll be very familiar with the new Azure Batch activity in Fabric data pipelines. This will allow you to execute custom executables from Azure Batch for powerful pipeline workflows. We’ve also enabled Azure Functions and KQL scripts as activities in Fabric pipelines. We've updated the Pipeline output monitoring view to focus on simplification, easily representing pipeline run status with customizable output layouts Pipeline lineage now includes lineage for Connections Dataflow Gen2 New Service Principal authentication kind Service Principal has been added as an authentication type for a set of data sources that can be used in Dataset, Dataflow, Dataflow Gen2 and Datamart. Azure service principal is a security identity that is application based and can be assigned permissions to access your data sources. Service principals are used to safely connect to data, without a user identity. Supported data sources include: Azure Synapse analytics Azure SQL database Azure data Lake gen 2 Azure data lake Azure blob storage Web Dataverse SharePoint online Note: service principal is not supported on the on-premises data gateway and virtual network data gateway. Support is planned by the end of 2023. Read more about Service Principal support from the official announcement post. Bug fixes and reliability improvements We continue to listen to your feedback and we’re actively working and making strategic fixes and quality improvements to our service. The following is a short list of the most impactful fixes and improvements delivered in the past couple of weeks. Revert changes for the limit of 50 queries per dataflow. In our previous monthly update we implemented a limit to the number of queries that a dataflow could have. We reverted this limit until further notice due to a number of undesired experiences created, but we will reintroduce this limit again in the near future. If you have dataflows with more than 50 queries currently, we encourage you to start splitting them into multiple dataflows with lower number of queries, as this will give you a head-start when the limit is reintroduced in the future. This documentation article describes several best practices when developing complex dataflows that you may find useful: Best practices for designing and developing complex dataflows - Power Query | Microsoft Learn Multiple fixes and improvements to common error messages and scenarios. Thanks to multiple customers who have reached out to us through our support system and our Community Forum, we’ve been able to address several common issues and improve the error messages that were quite frequent. Some of the error that have been fixed and/or modified to provide more descriptive information are: “The current row is too large to write.” “We cannot convert the value null to type Table.” “Dataflow refresh transaction failed with status Failed” Errors with Dataflow connector when creating linked entities. This issue has been fixed. We want to emphasize our commitment to our customers on improving the product and take your feedback into account in our decision making. We encourage you to engage directly with us through our support channel or through the Community forums. Community New learning path: Implement a Lakehouse with Microsoft Fabric To learn more and get started, check out the Implement a Lakehouse with Microsoft Fabric learning path. That is all for this month! Please continue sending us your feedback and as always, keep voting on Ideas to help us determine what to build next. We are looking forward to hearing from you!Microsoft Fabric October 2023 update
Microsoft Fabric October 2023 update We have a lot of features this month including updates to the Item type icons, Keyword-Based Filtering of Tenant Settings, On-object Interaction updates, Eventstream Kafka Endpoints and many more. Continue reading for more details on our new and updated features! Core Item type icons Admin Keyword-Based Filtering of Tenant Settings Power BI Reporting Power BI Desktop OneDrive and SharePoint integration On-object Interaction Updates (preview) New! Date Hierarchy on data flyout New! Placeholder text for direct text editing New! Ribbon and Funnel charts now support on-object formatting. Power BI Home in Desktop Deduplication rules for composite models on Power BI Datasets and Analysis Services Modeling Edit your data model in the Power BI Service – Updates Model explorer public preview with calculation group authoring and creating relationships in the properties pane. Data connectivity Snowflake (Connector Update) Planview OKR (New Connector) BitSight Security Ratings (Connector Update) Starburst Enterprise (Connector Update) Service OneLake data hub – Quick access to your data Explorer pane - Quick access Favorite items Visualizations New visuals in AppSource ValQ Plan Date Picker by Powerviz Drill Down Network PRO: Show Categorical Relationships with Ease TMap 2.1 Inforiver Enterprise brings writeback to Fabric Lakehouse & Warehouse Synapse Data Warehouse V-Order write optimization SKU guardrails for burstable compute Data Science Semantic Link (Public Preview) Real-time Analytics KQL Database Capacity Reporting KQL Database Auto scale algorithm – improvements Filtering and visualizing Kusto data in local time with special PowerBI optimization Eventstream UX Improvement on Event Processor Eventstream Kafka Endpoints and Sample Code Data Factory Dataflow Gen2 Data connectivity SAP HANA (Connector Update) New certified connector: Emplifi Metrics Bug fixes and reliability improvements Pipelines Activities AML activity Deactivate/reactivate activity state (Preview) Productivity Category redesign of activities Copy runtime performance improvement Integer data type available for variables Pipeline name now supported in System variables. Support for Type editing in Copy activity Mappings Data Activator Data Activator is now in public preview Core Item type icons We’ve heard your feedback that the icons across Fabric are difficult to tell apart from large list views and other areas of the UI, and that the overall appearance of them was overly monochromatic and flat. Our design team has completed a rework of the item type icons across the platform to improve visual parsing and enrich the way users may form a habituated understanding of what icons operate within different semantic categories of working with data. To learn more about the thinking behind this update, see the detailed blog Admin Keyword-Based Filtering of Tenant Settings Microsoft Fabric has recently introduced keyword-based filtering for the tenant settings page in the admin portal. This new feature allows admins to locate the necessary tenant settings quickly and easily by filtering them based on keywords. To use keyword-based filtering, tenant admins simply enter the keywords they are looking for into the search bar at the top of the tenant settings page. The portal will then filter and display only those settings with matching keywords in the title or description. For instance, an admin could search for "preview" to access all tenant settings currently in preview, or "B2B" to view settings associated with the B2B data sharing feature. To learn more about the feature, check out this blog post. Power BI Reporting Power BI Desktop OneDrive and SharePoint integration OneDrive and SharePoint integrations are now easier than ever in Power BI Desktop! While the ability to open, save, and share reports in OneDrive and SharePoint was released to preview in May, the capabilities are now improved and on by default. You can: Open reports stored in OneDrive and SharePoint through the file menu. Save files directly to OneDrive and SharePoint. Share reports stored in OneDrive and SharePoint directly from Power BI Desktop. These are important changes because many Power BI authors use OneDrive and SharePoint to collaborate on their reports before publishing through the Power BI service. These new updates will streamline their workflow. Ad-hoc reporting is made easier and simpler, and new users starting out will be comfortable with the familiar Office interface. The new Desktop features are complimented by the ability to view Power BI reports stored in OneDrive and SharePoint directly in your browser. Previously, viewing a report stored in OneDrive or SharePoint required downloading both the Power BI file, and Power BI Desktop. The new capability allows users to interact with their reports in seconds. We understand that every organization has its unique needs and preferences. If you prefer not to have some of these features available in your organization, learn how to turn them off in the Fabric admin portal. Learn more about these features in our public preview announcement blog post. On-object Interaction Updates (preview) New! Date Hierarchy on data flyout When working with dates, you may choose to swap from a hierarchy to the raw date field. This is still available when right clicking directly on the date field. For greater discoverability, we’ve now added the ability to switch between date hierarchy and raw date on the data flyout as well. Option 1: Swap using right click. New! Option 2: Swap using data flyout. New! Placeholder text for direct text editing Previously, when turning on a text element such as a title for Tables, there was no reserved space to begin typing in characters directly on-object. Users had to go to the format pane to add a title. Now, when turning on a text element that does not have an “auto” value, we show a placeholder for users to begin typing directly on the visual. When using direct text edit, you’ll also see placeholders appear if you accidentally delete all the characters, but the text element is still ON. This again reserves the space for you to be able to come back and a text value using on-object. Placeholders only appear when the visual(s) are selected while editing, deselecting the visual will remove all placeholders so authors can preview what will be published. New! Ribbon and Funnel charts now support on-object formatting. Power BI Home in Desktop Deduplication rules for composite models on Power BI Datasets and Analysis Services Tables and measures in your model must have unique names. If you use composite models on Power BI Datasets and Analysis Services, it’s easy to get into a situation where tables and measure names have duplicate names and are not unique. Up to this point, when that happened one of the tables or measures would be renamed for you. For example, if you created a composite model from two sources and both sources defined a table called ‘Customers’”, one of the tables would be renamed ‘Customer 2’. This resulted in confusing situations as it was not clear which source the ‘Customer 2’s table came from. The same applies to measures: if you had two sources that both contained a measure called ‘Total Sales’ one would be renamed ‘Total Sales 2’ in the composite model. This month we are giving you more control! You can now apply a name disambiguation rule to a source in a composite model when you anticipate name conflicts with tables or measures from another source. You can set up a text to be added as a prefix or suffix to table names, measure names or both. Additionally, you can choose to add that text only when a duplication occurs or if you prefer to have it added all the time. Going back to the example above, let’s say that one of the sources you are combining is for marketing and the other is for sales. You can now set up a deduplication rule on the source connection so the ‘Customer’ table from the marketing source is named ‘Customer (marketing)’: You will find these options under Settings in the dialog that shows when you set up the composite model connection to a Power BI dataset or Analysis Services model: After you make the connections and set up the deduplication rule, your field list will show both ‘Customer’ and ‘Customer (marketing)’ according to the deduplication rule you set up: Note that you can: Specify whether you want the text to be added to the table or measure name as a prefix or a suffix. Apply the deduplication rule to tables, measures, or both. Choose to apply the deduplication rule only when a name conflict occurs or apply it all the time. The default is to apply the rule only when duplication occurs. In our example, any table or measure from the marketing source that does not have a duplicate in the sales source will not get a name change. If you do not specify a deduplication rule or the deduplication rules you specified do not resolve the name conflict the standard deduplication actions are still applied. Read more about composite models on Power BI datasets and Analysis Services in our documentation. Modeling Edit your data model in the Power BI Service – Updates The new data model editing in the Service feature was released to preview in April. We’ve been busy reacting to your feedback and enhancing the experience. Below are the improvements coming later this month: Manage relationships dialog. Now you can easily view and edit all the relationships within your data model in the Service! In the Home tab, simply select the ‘Manage relationships’ button. This will open the revamped ‘Manage relationships’ dialog, which provides a comprehensive view of all your relationships, along with their key properties, in one convenient location. From here you can then choose to create new relationships or edit an existing relationship. Additionally, you have the option to filter and focus on specific relationships in your model based on cardinality and cross filter direction. Mark as date table Within the Service, you can now mark a table in your data model as a date table. Marking a date table in your model allows you to use this table for various date-related elements including visuals, tables, quick measures, and more, with full Time Intelligence support. To set a date table in the Service, right-click on the desired table and choose ‘Mark as date table > Mark as date table’ in the menu that appears. Next, specify the date column by selecting it from the dropdown menu within the 'Mark as date table' dialog. Power BI will then perform validations on the selected column and its data to ensure it adheres to the 'date' data type and contains only unique values. Please continue to submit your feedback directly in the comments of this blog post or in our feedback forum. Model explorer public preview with calculation group authoring and creating relationships in the properties pane. We are excited to announce that the model explorer is now available for public preview in the model view. You can see all your dataset semantic modeling objects in one place and easily navigate between them. Finally, full visibility of the semantic model! An additional properties pane now shows for the dataset semantic model. And new icons are also showing on the Data pane! Additional UI changes will continue through December for the Data pane and properties pane. Also note that measure groups will always show at the top, followed by calculation groups, and then finally the other tables in the model. From the model explorer, not only can you now see the calculation groups with their calculation items, but you can also create and edit them in Desktop! Calculation groups are a powerful feature that allows you to apply dynamic calculations to your existing measures. For example, you can create a calculation group that applies time intelligence functions as calculation items, such as year-to-date, quarter-to-date, or month-to-date, to any measure in your model. Learn more at aka.ms/calculationgroups. To author these in Desktop, you go to the model view and click on the new “Model” tab of the Data pane. If you click on the Calculation groups node, you have three options to create a new one. (1) Ribbon button, (2) Context menu, (3) Properties pane. After clicking “New calculation group”, if you do not have the model property discourage implicit measures turned off, you will be told this setting needs to be on to create the calculation group. Once turned on, the calculation group is created, and you are landed directly in the first calculation item to define the DAX expression you want to apply to your existing measures. You can alter this DAX expression in the DAX formula bar. Optionally, you can also add a dynamic format string to the calculation item from the properties pane. New calculation items can be created from the context menu (right-click) of the Calculation items node or the calculation group, and in the properties pane of the Calculation items node. Controlling the order of the calculation items can be done in the Calculation items node properties pane or using the context menu (right-click) of the calculation items themselves! Known issues to be fixed before public release: Renaming calculation items will not reflect in the report view slicer until another calculation item added or deleted. Changing the order of calculation items in properties pane moves away from the properties pane. Changing precedence of the calculation groups in the properties pane moves focus away from the properties pane. Is hidden showing “Mixed values.” Cursor not being in the calculation item on creation. Another option also now available from the model explorer is the ability to create relationships in the properties pane. Just like edit relationships in the properties pane allows you to simply change table and columns without previewing data or validating until you click Apply change, this experience is now available to create relationships too. Simply choose new relationship from the model explorer Relationships node. This will show an empty relationship properties pane to fill out and then apply when ready! Learn more about adding and editing relationships at Create and manage relationships in Power BI Desktop - Power BI | Microsoft Learn. Known issues to be fixed: Setting the relationship as one too many even in a valid way will result in an error. This has been fixed but waiting for the changes to be included in Desktop, which may be after public release. The model explorer does show you other modeling features that do not yet have authoring paths in Desktop. These features are perspectives and cultures. These will still need to be authored outside of Desktop throughout XMLA write external tools or through XMLA directly. Learn more about XMLA write at Dataset connectivity and management with the XMLA endpoint in Power BI - Power BI | Microsoft Learn and external tools at External tools in Power BI Desktop - Power BI | Microsoft Learn. Finally, learn more about perspectives at Perspectives in Analysis Services tabular models | Microsoft Learn which work well with the personalized visuals feature of Power BI reports at Let users personalize visuals in a report - Power BI | Microsoft Learn. And cultures is the translations features of semantic models, learn more at Translations in Analysis Services tabular models | Microsoft Learn. These features are available in the latest version of Power BI Desktop. To use them please turn on the model explorer public preview switch. Go to File > Options and settings > Options > Preview features in the GLOBAL section. Data connectivity Snowflake (Connector Update) The Snowflake connector has been updated to support better implementation of “LIMIT 1” queries, resulting in performance improvements. Planview OKR (New Connector) We are excited to release the new Planview OKR connector. Here are the release notes from the Planview team. Planview Objectives and Key Results (OKRs) are an outcome-driven framework adopted by organizations who want to define key organizational goals and track progress toward achieving them. Defining OKRs creates organizational clarity by enabling organizations to answer the questions “Where do we want to go?” (objectives) and “How will we measure our efforts to get there?” (key results). OKRs can be created at different levels of an organizational structure – such as enterprise, portfolio, program, or team – and are connected using parent/child relationships. Linking organizational and team goals in a hierarchical way aligns work delivery to company strategy and provides a single line of sight into value delivered by the organization. Connect now to your OKR data with the custom connector Planview OKR. BitSight Security Ratings (Connector Update) The BitSight Security Ratings connector has been updated with minor bug fixes. Starburst Enterprise (Connector Update) The Starburst Enterprise connector has been updated. Here are the release notes from the Starburst team: Added optional Connection string (non-credential properties) in the Advanced Section to allow the use of other ODBC connection parameters. Changed data source display from Starburst Enterprise to a value based on the connection details ({“Host”:”sep.example.com”, “Port”:”443”}) to allow distinction between multiple Starburst Galaxy or clusters connected as separate data sources. Fixed an issue with port 8443 for OAuth 2.0 authentication. Fixed query folding with timestamp columns. Service OneLake data hub – Quick access to your data OneLake data hub is the central location for users to find and reuse existing organizational data. It allows users to browse through their data and discover insights that can help them make better decisions. We are happy to introduce some of the new features that we recently added to enhance the OneLake data hub discovery experience. Explorer pane - Quick access The Explorer pane enables users to navigate through the workspaces hierarchy and scope the data items to a specific workspace. With the recent enhancements, we added a Quick access section at the top of the Explorer pane. The Quick access section contains an active workspace, pinned workspaces, and recently used workspaces. Favorite items Another new feature is the favorite items. Users can now filter to view items that were marked previously as favorite in Power BI. Users can also find favorite items from within the data hub experience, and they will also show up across all the experiences including Home and Browse. This feature helps users to keep track of the data items that are most important for them. Visualizations New visuals in AppSource Step Filter ValQ Plan valQ Plan is a newer and re-architected Power BI-certified edition of valQ with numerous transformational updates and a simplified no-code experience. valQ helps users build complex business plans, what-if simulations, create & compare budgets, forecasts and scenarios – all within Microsoft Power BI. It can be used as your standalone and integrated business planning software tool or used in conjunction with your existing planning platforms. YT video URL - https://www.youtube.com/watch?v=w4AFQzr58j4 Feature highlights: Niche Use Cases such as Scenario Planning, Strategic Planning or Value Drive Trees and What-If Simulations. Typical Planning use cases: Financial planning & analysis (FP&A), 3-statement financial modelling, and Supply Chain Demand Planning etc. Excel-like grid input experience for your rolling forecasts with no-code auto forecasting algorithms, and goal seek. Integrated reporting: One-click IBCS templates for variance reporting, presentation mode with live charts and tables, enhanced export to Excel & PDF Intuitive modelling: Significantly enhanced formula editor experience, dynamic templating and more Time intelligence: Supports more than 12 periods allowing you to create 90-day rolling forecasts, 52-week forecasts, and even 10-year strategic plans. ValQ Plan can be purchased directly from Microsoft AppSource. Date Picker by Powerviz The Ultimate Date Slicer for Power BI. The Date Picker visual comes with a modern calendar view along with highly requested features like Presets, Pop-up mode, Default Selection, Themes, and more. This is a must-have date slicer for all the Power BI reports. It has rich formatting options to match your brand style guide and to meet your business needs. KEY FEATURES: Display Mode: Choose between Pop-up and Canvas modes. Presets: Many commonly used presets like Today, Last Week, YTD, MTD, or create your own preset using field. Default Selection: Enforce the selection of the desired period when a user opens the report without any custom DAX. Filter Type: Choose between Range and Start/End types. Multiple Date Ranges: Flexibility to select multiple date ranges. Themes: 15+ pre-built themes with full customization. Holidays and Weekends: Multiple formatting options. Import/Export JSON: Build templates and share your designs. Many more features and customizable options. Check out the video - Introducing Date Picker by Powerviz. Get Powerviz Date Picker for FREE from AppSource. Download the demo file here. Step-by-step guide and detailed documentation of all features. To learn more, visit Powerviz website. Drill Down Network PRO: Show Categorical Relationships with Ease Drill Down Network PRO by ZoomCharts is designed for effortlessly visualizing categorical data and automatically detects relationships based on category structure. Use your existing category-based data with few adjustments, and quickly create an interactive chart that makes relationships between each data category easy to read. Visit our site to learn more! Main features: Cross-chart filtering – eliminate slicers by selecting data points directly on the charts. Category-based customization – apply image to node and choose its type, shape, color, font. Link styling – configure 'from' and 'to' decorations and show link value. Display up to 9 data categories. Touch-input device friendly – explore data on any device. Popular use cases: Accounting & Finance – show cost attribution. Human Resources – analyze salary data by department. Production – map production volumes by product or factory. Sales & Marketing – visualize marketing campaigns. ZoomCharts Drill Down Visuals are known for interactive drilldowns, smooth animations, and rich customization options. They support interactions, selections, custom and native tooltips, filtering, bookmarks, and context menu. Use them to create visually appealing and intuitive reports that business users will love on every device. Try Drill Down Network PRO now from the AppSource! TMap 2.1 Newly released TMap 2.1 has been added new features for drilling down Donut Map, Choropleth Map, Bar Chart Map, Pie Chart Map and Stacked Bar Chart Map by georegion's names. Drilling down maps by georegion's names doesn't need to prepare a hierarchy of pre-built polygon layers and will save data professional's time and cost to extract deeper insights from the data on different levels of geographic regions. Screenshot 1 (Drilldown donut map for company A's sale in Asia by subregions) Realigning georegions will become much easier because you only need to change georegion's name in the lowest level and the polygon layer for upper level will be automatically generated during the drilling down process. Screenshot2 (Drilldown donut map for company A's sale in Asia by realigned subregions) You can go to Microsoft AppSource( https://appsource.microsoft.com/en-us/product/power-bi-visuals/mylocsinc1648311649136.tmap?exp=ubp8) to download and try it. To learn more on how to use it, please read tutorials (https://www.mylocs.ca/tutorials.html#drilldown-donut-map-name). Inforiver Enterprise brings writeback to Fabric Lakehouse & Warehouse Inforiver now supports three new data writeback destinations from Power BI: Microsoft Fabric Lakehouse, Fabric Datawarehouse, and Dataverse, along with other popular DWH, DL, and Databases. Here is our 2-minute demo and overview The latest Inforiver Enterprise release supports writeback to Fabric for these use cases. Create business projections by entering/editing directly in Power BI reports. Create & manage rolling forecasts by blending actual & forecast data series. Create dynamic simulations at the enterprise, BU, or department level and roll up projections (to the top) or distribute projections (to granular dimensions) based on chosen allocation/distribution methods. Create multiple scenarios, each containing its own set of simulations. Write back data, including comments & threaded conversations Create periodic report snapshots. Support multiple data input types and writeback from multiple Power BI users in reading mode. The above, combined with advanced audit, security & governance capabilities, make Inforiver the most advanced data input and write-back solution in the market. Inforiver supports writeback in the following deployment configurations: (a) Azure SAAS service managed by Inforiver or (b) managed by customers in their private tenant. Visit our FAQ page to learn more. For Fabric Writeback Proof of Concept (PoC), contact Inforiver here. Synapse Data Warehouse V-Order write optimization V-Order optimizes parquet files to enable lightning-fast reads under the Microsoft Fabric compute engines such as Power BI, SQL, Spark and others. Warehouse queries in general benefit from faster read times with this optimization, still ensuring the parquet files are 100% compliant to its open-source specification. Starting this month, all data ingested into Fabric Warehouses use V-Order optimization. SKU guardrails for burstable compute Synapse Data Warehouse on Microsoft Fabric has the desired flexibility to allow better performance under peak demand by providing burstable compute. SKU Guardrails ensure that customers are operating within the right boundaries for their capacity, preventing peak workloads from consuming all capacity units in a short duration. To learn more about SKU guardrails for burstable compute, check out the blog Data Warehouse SKU Guardrails for Burstable Capacity Data Science Semantic Link (Public Preview) We are pleased to introduce the Public Preview of Semantic Link, an innovative feature that seamlessly connects Power BI datasets with Synapse Data Science within Microsoft Fabric. As the gold layer in a medallion architecture, Power BI datasets contain the most refined and valuable data in your organization. With Semantic Link, we unlock this data's potential beyond traditional business intelligence by making it accessible to notebooks and Python in Microsoft Fabric. Real-time Analytics KQL Database Capacity Reporting A KQL Database will utilize capacity via Operations that can be monitored using the Microsoft Fabric Capacity Metrics. KQL Database Consumption - This is the number of seconds that your KQL database is active in relation to the number of virtual cores used by your database. For example, if my database uses 4 virtual cores and is active for 10 minutes then you will utilize 2,400 seconds of Capacity Units. An auto-scale mechanism is utilized to determine the size of your KQL database. This ensures the most cost-optimized and best performance based on your usage pattern. Read more in the blog: Understanding Fabric KQL DB Capacity | Microsoft Fabric Blog | Microsoft Fabric KQL Database Auto scale algorithm – improvements Users do not need to worry about how many resources are needed to support their workloads in a KQL database. KQL Database has a sophisticated in-built auto scaling algorithm. The algorithm ensures that the optimal number of resources are allocated to support the workloads, with minimum cost. The auto scaling algorithm is multi-dimensional, based on the following dimensions: Hot cache- how much data to store in hot cache for immediate response. This metric is impacted by the caching policy defined by the customer. Memory – how much memory required for meta data and data for optimal query performance. CPU usage- how much compute resources are needed to process queries, update policies, materialized views etc Ingestion - how much compute resources are needed to for data ingestion based on ingestion rates and load times. Filtering and visualizing Kusto data in local time with special PowerBI optimization Datetime values in Kusto (aka ADX/KQL database in Fabric) are assumed to be in UTC. There are good reasons why you should always keep it this way. On the other hand, in many cases you want to visualize the datetime values in a specific time zone and filter the data using values expressed in local time. This is correct but may lead to severe performance degradations if not done correctly. We recently implemented some optimizations that will make some such scenarios much more efficient. In addition, for the many users who are using PowerBI with Fabric KQL database for time series analysis, here is the optimal recommendation: Create a function that will receive the time range as parameters, shift it to UTC, filter the table and shift the filtered rows from UTC to the same time zone. For more details and examples refer to the following blog: Filtering and visualizing Kusto data in local time Eventstream UX Improvement on Event Processor The Event Processor within Eventstream is a powerful no-code editor, enabling you to process and manage your real-time data streams efficiently. You can easily aggregate and filter data streams using temporal functions before they reach your lakehouse or Kusto database. The recent UX improvements introduce a full-screen mode, providing a more spacious workspace for designing your data processing workflows. The insertion and deletion of data stream operations have been made more intuitive, making it easier to drag and drop and connect your data transformations. Eventstream Kafka Endpoints and Sample Code We've expanded the Custom App feature with a range of new endpoints in sources and destinations. Now, you can seamlessly connect your applications to Fabric Eventstream using protocols like EventHub, AMQP, and Kafka. To simplify your setup process, we've included sample Java code for your convenience. Simply add it to your application, and you'll be all set to stream your real-time event to Eventstream. Data Factory Dataflow Gen2 Data connectivity SAP HANA (Connector Update) The update enhances the SAP HANA connector with the capability to consume HANA Calculation Views deployed in SAP Datasphere by taking into account SAP Datasphere’s additional security concepts. This enables consumption of Calculation Views in Datasphere and allows customers to connect to HANA Cloud views without the need for additional privileges on the _SYS_BI schema. New certified connector: Emplifi Metrics We are happy to announce the release of the new Emplifi Metrics connector. Please find release notes from the Emplifi team below: “Integrating social media insights alongside the rest of your marketing or business intelligence data gives you a holistic understanding of your entire digital strategy, all in one place. With Emplifi Power BI Connector, you’ll be able to include social media data from the Emplifi Platform in your charts and graphs and combine them with other data you own. The Power BI Connector is a layer between Emplifi Public API and Power BI itself. It helps you work with your data intuitively, directly in the Power BI tool. Most data and metrics available in the Emplifi Public API are also available in the Connector. Please visit the official documentation for more information about Emplifi Public API and a list of available metrics. You will find it here: https://api.emplifi.io/.” Bug fixes and reliability improvements We would like to thank our community for reporting issues and providing feedback through the Data Factory Community Forum. We continue to work on improving the overall experience and reliability of Dataflow Gen2 in Microsoft Fabric. With over 400 work items closed in the last month, you should notice a better experience in the overall Dataflow Gen2 authoring and refresh experience. We encourage you to visit our community forum and provide any feedback or inquire about any possible issues that you might have with Dataflow Gen2. Your feedback is helping us make this a better product each day. Pipelines Activities AML activity We’re excited to announce that the Azure Machine Learning activity is now available to use in your Data Factory data pipelines. In your pipeline, you can use the Azure Machine Learning activity to connect to your Machine Learning pipelines or enable batch prediction scenarios such as identifying possible loan defaults, determining sentiment, and analyzing customer behavior patterns. Deactivate/reactivate activity state (Preview) We’re excited to share that you can now deactivate one or more activities from a pipeline, allowing you to skip activities during pipeline validation and during pipeline runs. This will help to improve developer efficiency, allowing you to comment out parts of your pipeline without deleting anything from your pipeline canvas. Deactivated activities can be reactivated at any time. Learn more here. Productivity Category redesign of activities We’ve redesigned the way activities are categorized to make it easier for you to find the activities you’re looking for with new categories like Control flow, Notifications, and more! Copy runtime performance improvement We've made improvements to the Copy runtime performance. According to our tests results, with the improvements users can expect to see the duration of copying from parquet/csv files into Lakehouse table to improve by ~25%-35%. Integer data type available for variables We now support variables as integers! When creating a new variable, you can now choose to set the variable type to Integer, making it easier to use arithmetic functions with your variables. Pipeline name now supported in System variables. We’ve added a new system variable called Pipeline Name so that you can inspect and pass the name of your pipeline inside of the pipeline expression editor, enabling a more powerful workflow in Fabric Data Factory. Support for Type editing in Copy activity Mappings You can now edit column types when you land data into your Lakehouse table(s). This makes it easier to customize the schema of your data in your destination. Simply navigate to the Mapping tab, import your schemas, if you don’t see any mappings, and use the drop-down list to make changes. Data Activator Data Activator is now in public preview Data Activator reached a big milestone this month, with its release to public preview. This means Data Activator is now available to all Fabric users, without the need to sign up to be a preview user. You can use Data Activator to drive alerts and actions from your Fabric data. Want to try it out right now? Open a Power BI visual and select the “trigger action” menu option, to create a Data Activator alert: Data Activator also works with real-time streaming data in EventStreams. To make a Data Activator alert on your EventStreams items, create a “reflex” destination: To learn more about Data Activator, check out the following: Data Activator demo video Data Activator documentation on Microsoft Learn
