Hi, I generally use the mirroring feature to get data from an on-premise supported source at bronze level of the medallion architecture. In order to perform some data transformations and put the related data in a next lakehouse or warehouse I could use a dataflow gen2 that doesn't allow to select a mirrored database as a source. Moreover, also inside a pipeline it isn't possible to select a mirrored database as a source. I think that using a mirrored database as a source could be a good thing to have! Many thanks ... View more

We need support for Service Principal authentication when creating Lakehouse connections.   Right now, Lakehouse connections can only be created using a user account. As a result, Dataflow Gen2, Pipeline copy activity, and Copy Job cannot use a Service Principal (SPN) when connecting to a Fabric Lakehouse as a source or destination.   This is an issue for us. We want to avoid user-based connections because: We need environment-specific identities (dev/test/prod) to prevent accidental cross-environment writes (e.g., dev pushing data to prod). Tying connections to personal user accounts creates fragility - things break when someone leaves the project or the organization. Service Principal support for Lakehouse connections is required to ensure secure, reliable, and environment-isolated authentication.   The screenshot shows that only user authentication is available, not service principal authentication. ... View more

its well known that intellisense has always been broken with things like ListList.Contains or pressing . after typing certain prefixes does not correctly show options, or typing null enter puts null.type in. Also a huge issue is no intellisense in Dataflows both need fixing ... View more

When Power BI executes queries against SQL Server, it connects using the Mashup Engine or Mashup Engine (%HostContext%)) as program_name. In a crowded environment, it can be difficult to identify which semantic model is responsible for activity or issues on the SQL Server side. The desired outcome is to be able to run sp_whoisactive (or query sysprocesses) and see the semantic model name in the program_name column (or something that easily idenfity the source of the query). For example:   Mashup Engine - Semantic Model "Sales_Model_A" (Name or GUID) ... View more

I am following the documentation at https://learn.microsoft.com/en-us/fabric/data-factory/dataflow-gen2-parameterized-dataflow to parameterize my dataflow and pass in the parameters from my variable library. I am able to pass the parameters to the dataflow from the pipeline to the dataflow but once I try to parameterize the workspace id and dataflow id in the pipeline, the dataflow parameters option disappears.   I've found a temporary workaround by saving it both ways and then manually merging the source file, but every time I open the pipeline to make edits, any saves overwrite my work around.   Can the UI be updated to show the dataflow parameters when variables are used for the dataflow?   Passing Parameterized values to the dataflow:          parameterizing the dataflow connection breaks/hides the ability to pass the parameters to the dataflow:       Manually edited pipeline-content.json:     ... View more

Currently, only one person can edit a Fabric Dataflow at a time, requiring constant "takeover" which slows down team productivity. This is especially painful for teams across different time zones or when multiple people need to work on different parts of the same dataflow. I propose allowing multiple users to collaborate on dataflows simultaneously, similar to how Power BI reports support co-authoring.   To prevent conflicts, implement edit locking where actively edited sections are locked while others can view in real-time. This would show who's editing what, allow parallel work on different queries, and eliminate the frustrating takeover cycle. This aligns with modern collaboration standards in Microsoft 365 and would significantly speed up dataflow development for enterprise teams. ... View more

Currently, the Lakehouse destination only supports Organizational account authentication.   This means the destination connection is depending on a specific user's account.   Please make it possible to use Service Principal authentication instead.   By going through the destination settings inside the Dataflow Gen2, I can see that the options for connecting to the various destinations are:   * **Lakehouse**    * Organizational account * **Warehouse**    * Organizational account    * Service Principal * **Fabric SQL Database**    * Organizational account    * Service Principal * **Azure SQL Database**    * Basic (username/password)    * Organizational account    * Service Principal    * Workspace identity    * Note: Basic and Service Principal options can use Azure Key Vault references in this connector. * **Azure Data Explorer**    * Organizational account    * Service Principal * **SharePoint**    * Anonymous    * Organizational account    * Service Principal    * Workspace identity * **Azure Data Lake Storage Gen2**    * Account key    * Organizational account    * Shared Access Signature (SAS)    * Service Principal    * Workspace identity   (Sorry about the bad formatting).   The Lakehouse destination only supports Organizational account - why? ... View more

There should be a secure way to pass secrets or keys when making an API call (e.g. web.contents) in Power Query M.   Suggestions:   A) Make it possible to enter and edit secrets in Power Query, similar to how we enter and edit data source credentials in Power Query. Make these secrets encrypted and redacted. The secrets should be available to use as variables in the M code. When entering a secret, we would give it a name, which we can reference as a variable in the M code. We would only see the name, but the value would not be visible (the value would be redacted).   B) Make an Azure Key Vault connector for Power Query, that will make it possible to fetch secrets from AKV and assign the secret to a variable in the M query. This variable can be passed as part of the header in an API call.   C) Make it possible to redact some variables in M, so that we can use secret variables without the value being visible in the Power Query user interface.    The main point is that we want a way to use secured and hidden secrets as part of API calls in Power Query. ... View more

Enable to modify the mashup document of a Dataflow Gen2 by passing a variable from a pipeline before an execution of the Dataflow ... View more

Please make it possible to open a dataflow gen2 without taking it over. I wish we could open a dataflow gen2 even if it is "owned" by someone else in the workspace. I think that the user who opens the dataflow gen2 should need to apply their own - or shared - data source connections in order to see any data in the query steps. Otherwise, they would be able to use the owner's connections to interact with the data sources, and I don't think that's a good idea. Because the owner might not want other workspace members to use his/her connections. So I think that the user who opens the dataflow gen2 should need to apply their own - or shared - data source connections in order to see any data in the query steps. I think it should be possible to open the item and start editing it, and not having to formally "take over" until you are ready to publish the changes. Or some times you just want to open it for inspection without making any changes. And if you do start to edit the dataflow, but then realize you aren't able to authenticate to all the data sources, then it should be possible to cancel so that the dataflow remains unchanged and the previous connections keep working. Would be nice to have: before a user saves (publishes) changes to the dataflow, there could be a validation step to check that the user has authenticated properly to all of the item's data sources and data destinations, so that they won't cause the dataflow to stop working. If the validation fails, the dataflow should keep running with the previous connections and the changes should be undone. ... View more

Hi, in order to use in some scenarios the dataflow gen2 that uses the Web API connector, as an alternative respect to the data pipeline, it could be more useful that the dataflow manages automatically the pagination avoiding to implement a recursive Power Query function to do it. Many thanks ... View more

Enable an experience that allows the end-user to setup incremental refresh for Dataflows Gen2 in Data Factory for Microsoft Fabric ... View more

Currently, detailed run log access from the Monitoring Hub for Dataflow Gen2 (CI/CD), notebooks and others items are not available. The idea is to use Monitoring Hub to debug and have detailed run log access directly from the Hub.   Vote if it's USEFUL for you !  ... View more

During development, it’s hard to identify which stage or step takes the most time. Similar to Pipeline, a detailed execution history view per activity (for Dataflow, each table and its associated applied steps) would help developers optimize logic and improve performance. Since both Dataflow Gen2 and Pipeline are part of Data Factory, having similar monitoring features would greatly enhance development efficiency. ... View more

Dataflows are supported by the Deployment pipelines, but there is no Git integration for dataflows. With the introduction of Git integration of workspaces, it would be fantastic if Dataflows could be stored and traced and reviewed in Git same like it is possible now for datasets and reports. ... View more

It's not possible to run a Dataflow Gen2 as Service Principal.   This is documented here: Background Jobs - Run On Demand Execute - REST API (Dataflow) | Microsoft Learn     It would be great if it was possible to run Dataflow Gen2 as a Service Principal.   The ability to run Dataflow Gen2 as Service Principal would enable: identity isolation between environments using dedicated service principals per dev/test/prod environments avoid reliance on user accounts users may leave the project, so we want to avoid reliance on user accounts ... View more

Please fix this, so the Dataflow will get validated automatically after syncing changes from Git or using deployment pipelines:   "When you sync changes from GIT into the workspace or use deployment pipelines, you need to open the new or updated dataflow and save changes manually with the editor. This triggers a publish action in the background to allow the changes to be used during refresh of your dataflow. You can also use the on-demand Dataflow publish job API call to automate the publish operation."   https://learn.microsoft.com/en-us/fabric/data-factory/dataflow-gen2-cicd-and-git-integration#limitations-and-known-issues   Because this currently doesn't happen automatically, it causes confusion, unnecessary manual work, and often we forget this and therefore get unexpected results when running the dataflow. ... View more

Orchestrating a refresh of a Dataflow Gen2 with CI/CD and Git support isn't possible in Fabric data pipelines. We request you to bring this feature in Fabric so that users can integrate these dataflow gen 2 with CI/CD and Git integration in data pipeline. ... View more