Pipeline CI/CD Improvement, Get Item ID activity from workspace ID and item name
Currently when following any branch strategy that requires a separate ephemeral feature (or other) workspace running a notebook from the same workspace requires too much ground work To find the ID of the notebook you need a way to call the APIs to GetItems in the workspace to find the correct ID of the notebook (or other item like the lakehouse), this is overly convoluted requiring API calls through web activity or separate notebook in a stand alone workspace with fixed ID that returns the item IDs in the pipeline's workspace Idea: add a pipeline activity that retrieves the item ID of an item using a workspace ID and item name or a function in pipeline expressions that allows the same to be done so this can be inlined without a dedicated pipeline activity.
Allow Workspace Identities as an authentication method for Fabric SQL Databases
Currently the only method of authentication in Fabric pipeline connections to Fabric SQL databases is using OAUTH2, meaning that the credentials of a specific user need to be used (with all the attendant issues around user/credential expiry). By allowing the option of workspace identity for authentication, no user creds would be needed. Obviously correct access controls would need to be defined, but that's standard SQL Server maintenance.
Add Native Flatten Transformation for web connectors and Denodo Connector Support in Data pipeline
Currently, Microsoft Fabric Data Pipelines do not provide a native Flatten activity for handling hierarchical JSON, XML, or nested data structures. To perform flatten operations, users must switch to Dataflows Gen2, which introduces additional complexity, development effort, and execution overhead. Additionally, there is no native Denodo connector available in Fabric Data Pipelines, creating challenges for organizations that use Denodo as their enterprise data virtualization platform. Proposed Enhancements 1. Native Flatten Activity in Data Pipelines Provide a built-in Flatten activity similar to capabilities available in Dataflow transformations. Key features: Flatten nested JSON and XML structures directly within pipelines. Support multiple nested levels. Dynamic schema handling. Array expansion and hierarchy traversal. Low-code configuration experience. Integration with Copy Activity and incremental load patterns. 2. Native Denodo Connector Introduce a fully supported Denodo connector for Microsoft Fabric Pipelines. Key capabilities: Read and write support for Denodo Virtual Views. Authentication using OAuth, Service Principal, Username/Password, and Managed Identity. Metadata discovery and schema import. Support for incremental data extraction. High-performance data movement and query pushdown. Business Benefits Eliminates the need to create separate Dataflows solely for flattening data. Simplifies ingestion of API and semi-structured data sources. Reduces pipeline complexity and maintenance effort. Improves performance and developer productivity. Enables seamless integration with Denodo-based enterprise architectures. Accelerates migration from Azure Data Factory, SSIS, and other ETL tools to Microsoft Fabric. These enhancements would significantly improve Microsoft Fabric's enterprise integration capabilities by reducing development overhead and providing a more complete end-to-end data engineering experience within Data Pipelines.Built-in Audit Activity in Microsoft Fabric Pipelines Similar to SSIS Package Logging
In Microsoft Fabric Data Pipelines, implementing audit logging currently requires creating and maintaining custom bookmark tables, audit tables, stored procedures, and additional pipeline activities. This increases development effort, complexity, and maintenance overhead for every project. I would like Microsoft Fabric to provide a native Audit Activity similar to the logging and auditing capabilities available in SSIS packages. Proposed Features: Built-in Audit Activity that can be added to any pipeline. Automatically capture: Pipeline Name Activity Name Run ID Start Time End Time Duration Status (Success/Failed/Skipped) Rows Read/Written Error Message Trigger Information Environment/Workspace Details Automatic watermark/bookmark management for incremental loads. Predefined audit tables generated by Fabric. Out-of-the-box monitoring dashboard and reporting. Support for custom audit attributes and business metrics. Ability to write audit logs directly to Lakehouse, Warehouse, or Eventhouse. Business Benefits: Reduces development effort and repetitive implementation. Eliminates the need for manually maintaining bookmark and audit tables. Standardizes auditing across all Fabric projects. Improves monitoring, troubleshooting, and compliance. Accelerates migration from SSIS to Microsoft Fabric. This enhancement would significantly improve enterprise ETL/ELT development and provide a consistent auditing framework across Microsoft Fabric data integration solutions.
Add Pre-Deployment Impact Analysis and Destructive-Change Warnings
Before deploying Microsoft Fabric items from Development to Test or Production, Deployment Pipelines should automatically perform a detailed impact analysis. The validation should identify: • Downstream reports, semantic models, notebooks, pipelines, and other dependent items that may be affected • Breaking schema changes, including removed columns or changed data types • Operations that could delete, truncate, or overwrite existing data • Missing connections, permissions, dependencies, or deployment rules • Configuration differences between source and target environments The results should be displayed with severity levels such as Information, Warning, and Critical. Administrators should be able to block deployments containing critical changes or require manual approval before proceeding. This feature would prevent broken reports, failed refreshes, and accidental data loss, making Fabric deployments safer and easier to manage.
Fabric > Dataflow Gen 2 > Destination: Allow this to be a dataflow parameter like table name can be
I recently disovered parameters in dataflow which was a big unlock in organization/efficiency, but not being able to specify the detination dynamically (outside of very end table name) nullifies that features' usefulness by having to make at least one per lakehouse because of this. Please update the Destination UI to allow a Parameter to be selected for the lakehouse (including folder/schema/dbo) to be populated by a dataflow parameter.
Support Multi-Sink Copy Activity (Read Once, Write to Multiple Destinations) in Fabric Data Pipeline
Currently, a Copy activity in Microsoft Fabric Data Pipelines supports only one destination (sink). In many enterprise ETL scenarios, the same source dataset needs to be loaded into multiple destinations, such as multiple Warehouses, Lakehouses, SQL Databases, or external systems. The current options are: Create multiple Copy activities, which read the source multiple times. Use a staging table or Lakehouse, which still requires additional read operations for each destination. Both approaches lead to: Increased Capacity Unit (CU) consumption Additional OneLake/storage I/O Longer pipeline execution times Higher operational costs Increased load on source systems Requested Enhancement: Please introduce a Multi-Sink Copy Activity (Fan-Out capability) that reads the source dataset only once and writes it to multiple destinations during the same pipeline execution. Alternatively, provide an in-memory dataset cache that can be reused by multiple downstream Copy activities without re-reading the source data. Benefits: Read source data only once Reduce Capacity Unit (CU) consumption Reduce storage I/O Improve pipeline performance Lower operational costs Reduce load on source systems Simplify enterprise ETL pipeline design This enhancement would significantly improve the efficiency and cost-effectiveness of Microsoft Fabric Data Pipelines, especially when the same dataset must be distributed to multiple destinations.Fix the broken Diff Compare for Data Pipelines in Deployment Pipelines
The Compare view for Data Pipelines in Deployment Pipelines is all over the place. Nothing aligns between environments, even if there is no diff or only a small diff in reality. Please fix it - the Compare view works fine for other item types.Allow scheduler configuration with variable libraries
The new scheduler is a great idea (Unlocking Flexibility in Fabric: Introducing Multiple Scheduler and CI/CD Support | Microsoft Fabric Blog | Microsoft Fabric) However, the fact that we cannot control this using variable libraries means that if you have scheduled refreshed disabled in dev and test then after every prod deployment you have to renable the refresh. There is also no way to turn off this new scheduler and go back to the old scheduler which did not move across environments. Please either enable us to control the scheduler via depoyment pipelines ASAP or give us a tenant switch to disable this till this feature is supported.