Dataflow topics

[BUG] Dataflow Gen2 destination picker 404 — api.powerbi.com lakehouses UK region

sahillihas20 — Wed, 26 Aug 2026 23:16:25 GMT

Hi community,

We have a complete blocker with Dataflow Gen2 (CI/CD) destination configuration in our UK-region Fabric tenant.

The Error:
When clicking "Add default destination" and selecting a Fabric Lakehouse from the OneLake catalog, this error fires immediately for ALL users including the
workspace Admin:
DataSource.Error: Downstream service call to url 'https://api.powerbi.com/v1/workspaces/{WORKSPACE_ID}/lakehouses' failed with status code 404. (Region: uk)

The Lakehouse IS visible in the OneLake catalog — it appears in the picker. The 404 fires the moment it is clicked.

Root cause hypothesis:
The destination picker calls the legacy Power BI REST API: api.powerbi.com/v1/workspaces/{id}/lakehouses ← returns 404

For Fabric-native workspaces in the UK region, the correct endpoint should be: api.fabric.microsoft.com/v1/workspaces/{id}/lakehouses

The OneLake catalog listing (which uses a different path) resolves the lakehouse correctly — it's only the destination configuration call that uses the wrong endpoint.

Workarounds attempted — all failed:
1. UI — Admin user attempted same steps → same 404
2. queryMetadata.json via Git — Added destination fields, pushed to prod branch → Fabric silently strips loadEnabled/destination on every sync, creating a permanent conflict loop (workspace ≠ Git on every update)
3. Authentication methods — Before the error began firing immediately, a "Connect to default data destination" screen appeared offering two authentication options for the Lakehouse connection:
- Organizational account (Sahil Singh / user account)
- Service principal

Both were attempted. Neither succeeded — the 404 followed in both cases.

Subsequently, this connection credentials screen stopped appearing altogether. The 404 now fires immediately upon clicking lh_dhl_prod in the OneLake catalog, before the connection setup dialog is even shown. This regression suggests the failure point has moved earlier in the destination configuration flow.

Questions:
1. Is this a known issue for UK-region tenants?
2. Is there a way to configure the Dataflow Gen2 destination via Fabric REST API directly to bypass the broken UI?
3. Are there other file-based approaches (mashup.pq metadata) for CI/CD Dataflow Gen2 destinations?

Happy to provide Session IDs and further details to the Fabric team.

I have attached the relevant images, PFA...

Thanks

iifnull() failing on non-null input

cmjcf — Mon, 17 Aug 2026 03:40:08 GMT

The documentation for iifNull() says this:

Returns the first not null item when given two or more inputs.

We have a dataflow which handles the contents of a number of files, using a specified CSV format. One of the columns is named 'Dealer'. Due to an error in an upstream system, a significant number of files came in with the header misspelled 'Delaer'. There are so many of these files that we cannot simply edit them manually to fix. I have the following expression in a Derived Column transform:

iifNull('Dealer',byName('Delaer'))

I understand this expression to mean "Use the value of the column 'Dealer' if it is not null, else use the value of the drifted column 'Delaer'." (That name is not in the list set by the dataset, so it is drifted, and leaving out byName() causes validation to fail.)

When the file does not contain the spelling error, my expected result is that it simply retains the correct value in 'Dealer' because this is not null. However, when actually run, this expression actually returns null, and causes the column in the Derived Column transform to not be created, resulting in a "Resolved attribute(s) [name] missing from [names]" error.

Dataflow Gen2 - failing to load into the lakehouse

gabrifly — Thu, 23 Jul 2026 12:39:15 GMT

Hello. I have a dataflow gen2, fetching data from an azure data factory environment and loading them into a Fabric Lakehouse. It has been working flawlessfly for the last 3 months, but since two weeks, on some of the tables, I get this error:

"There was a problem refreshing the dataflow: "Something went wrong, please try again later. If the error persists, please contact support.". Error code: EntityUserFailure. (Request ID: 6d111afc-11ee-459f-9e31-db4d08c32c44)."

"XXX - ABC Entity Management_WriteToDataDestination: There was a problem refreshing the dataflow: "Couldn't refresh the entity because of an issue with the mashup document MashupException.Error: Error in creating lakehouse table in a version, InnerException: Tables can only contain uppercase or lowercase letters, numbers or the underscore character., Underlying error: Tables can only contain uppercase or lowercase letters, numbers or the underscore character. Details: Reason = Expression.Error;ErrorCode = Lakehouse036;Message = Tables can only contain uppercase or lowercase letters, numbers or the underscore character.;Message.Format = Tables can only contain uppercase or lowercase letters, numbers or the underscore character.;ErrorCode = Lakehouse019;InnerErrorCode = Lakehouse021,Lakehouse019;Microsoft.Data.Mashup.Error.Context = User GatewayObjectId: 1ee1c1a1-a11v-32bt-a19b-5ff6d874f432". Error code: 104100. (Request ID: 66d111afc-11ee-459f-9e31-db4d08c32c44)."

The weird part is that there is no consistency between which tables go to error and which ones proceed to being loaded. Sometimes it works sometimes it does not. Also, there are no issue with the naming, as all the columns are respecting the requirements laid out from the error. And above all, it was working flawlessly until few weeks ago and nothing a

Data pipeline failed. How do you troubleshoot?

rajan889262 — Wed, 15 Jul 2026 13:52:36 GMT

Data pipeline failed. How do you troubleshoot?

Required parameters are missing in the request data flow gen2 fabric

Dharma_Teja_M — Tue, 07 Jul 2026 10:35:49 GMT

Hi Team,

I have created a Dataflow Gen2 in Microsoft Fabric with Dataverse as the source. When I run the dataflow independently, it executes successfully.

However, when I invoke the same dataflow from a pipeline and pass the Dataverse URL as a dynamic parameter, the execution fails.

Below is the error message for reference:

Error:

Required parameters are missing in the request

Dataflow refresh job failed with status: Failed.

Data Mapping Flow with DataLake as Sink

LaraHaegele — Wed, 01 Jul 2026 12:52:15 GMT

Hello everyone,
I am currently trying to read data from a source as changelog and writing it into a merged DataLake table. So basically cdc connector logic but as for SHIR the connector is not supported.
I always get the issue:
Operation on target merge_lakehouse_landing_to_delta failed: {"StatusCode":"DFExecutorUserError","Message":"Job failed due to reason: com.microsoft.dataflow.Issues: DF-SRC-002 - 'tableName' (Table Name) is required - \naction sink(\n) ~> sinkName,EXE-0001,Dataflow cannot be analyzed as a graph,[564 737]","Details":""}
I use scriptLines to transform the lines inside the dataflow and if I provide json File as sink the pipeline runs without errors at least. If I provide a lakehouse table even if I hardcode the table name in the dataset I get this error (already recreated it so an old version of the dataset cannot be the issue).
Why is that so can I not merge the data directly into lakehouse or is there another issue I've overseen why lakehouse doesn't work as sink here?
Thank you in advance
Lara

ELT JSON Semi Structured Loop

icassiem — Tue, 30 Jun 2026 07:27:33 GMT

Good day,

Apologies this is a bit of a duplicated but because i only recently have plan approved, so i dont get the time to do deep research or learning as yet

Solved: Re: ELT JSON / Semi structured - Microsoft Fabric Community

My source are mostl API JSON that i have to transform but before i can transform i need to loop, i dont know if dataflow is like powerbi powrqury window or i do it in powerbi one example and past the script in dataflow but how would i do api loops then unpivot the json, would i need pythin to first downlad all looped api content then pass on dataflow etc?

Regards

Dataflow Activity Parameters Lose Expression and Get Hardcoded After Reopening Pipeline

svac — Mon, 29 Jun 2026 13:09:44 GMT

Hi! I’m using a Dataflow Gen2 activity in a Fabric pipeline and passing parameters via dynamic content (pipeline library variables).

When I configure the parameters, they are correctly stored as expressions:

"value": {

"value":

"@pipeline().libraryVariables.Data_engineering_Variable_Library_workspace_id",

"type": "Expression"

}

The pipeline validates and saves successfully. However, when I reopen the pipeline, Fabric rewrites the parameter values as evaluated static string:

"value": "f31c2823-10f1-4e97-bc15-6f39f4981d57",

"type": "String"

This removes the expression and effectively hardcodes the value.

Is there a supported way to persist dynamic expressions for Dataflow Gen2 parameters in pipeline activities?
Are there recommended patterns for passing workspaceId/lakehouseId dynamically without the UI flattening expressions?

Thanks!!

Dataflow Gen2 authentication to Lakehouse without user dependency?

Parchitect — Mon, 29 Jun 2026 07:19:31 GMT

Hi,

We have run into a challenge with Dataflow Gen2 in Microsoft Fabric.

When creating a dataflow and selecting a destination, for example a Lakehouse in the same workspace, the authentication appears to require an Organizational account. In practice, this means that the connection becomes tied to a specific user.

This creates governance and operational challenges in larger organizations. If the user who created or authenticated the dataflow leaves the company, changes role, or loses access, someone else needs to take over the dataflow and re-authenticate the connection. In an organization with thousands of users, this quickly becomes difficult to manage and track.

I understand that in some scenarios it may be possible to use a service principal, but that raises another question: how many service principals should be created and maintained? Should it be one per workspace, per domain, per team, per solution, or something else?

For this specific scenario, Workspace identity feels like a much more natural fit. A workspace-scoped identity would reduce the dependency on individual user accounts and make the solution more robust over time.

Microsoft’s documentation describes Workspace identity as a Fabric-managed service principal associated with a workspace, where Fabric manages the credentials and can use the identity with resources that support Microsoft Entra authentication. The Lakehouse connector documentation also shows that, for Dataflow Gen2, the supported authentication type is currently Organizational account.

My questions are:

What is the recommended best practice for authenticating Dataflow Gen2 when writing to a Lakehouse in the same workspace, without creating a dependency on a personal user account?
Is using a service principal the recommended approach today?
If so, what is the recommended pattern: one service principal per workspace, per team, per solution, or something else?
Are there any plans to support Workspace identity for this scenario?

It would be very helpful to hear how other organizations are handling this, especially in larger environments where ownership changes and user lifecycle management can otherwise become quite difficult.

Thanks in advance!

Dataflow Gen2 Copying Old Data

Russell_DC — Tue, 23 Jun 2026 20:58:30 GMT

I have a dataflow that is transforming data and then moving it to a staging table in a Silver Lakehouse. I keep having issues where the number of rows moved to that lakehouse doesnt match what I am seeing in the dataflow preview or what it should be. A simple example of this is that this should have read 2 and written 2 rows but instead it read and wrote 3 rows. Upon investigating this was the 3 rows from the previos time it ran, suggesting a caching issue! I have others where I have 578 new rows in my table within the dataflow and then in the Staging table it has 212 rows. I'm very confused as to what is going on and concerned I may need to convert to notebooks for my transformations?

amazon redshift ODBC driver

mklevemann — Wed, 17 Jun 2026 20:13:41 GMT

I was told by the vendor that provides me data in an amazon redshift data warehouse that the ODBC 1.x connector will not be working for desktop and I need to move to the 2.x connector. My question is will this ODBC change also affect Dataflow Gen2 accessing the amazon redshift data warehouse?

Gen2 dataflow read only mode issue

Lokamba — Mon, 15 Jun 2026 13:38:08 GMT

Hi Team,

I have admin access to the workspace. When I opened the Gen2 flow created by Rahul R., I was unable to edit it, because it is showing in read-only mode. Could you please help to fix this issue.

"Keyword not supported: variables" error using Dataflow gen2 when evaluating Fabric.Warehouse()

JuanFcoCalquin — Tue, 09 Jun 2026 21:24:09 GMT

Dataflow Gen2 fails in Power Query editor when evaluating Fabric.Warehouse() navigation query.

The issue occurs before refresh execution, while editing/evaluating standard M steps. It also occurs with a basic Dataflow Gen2 using Enter Data as source and Fabric Warehouse as destination.

Error:
Keyword not supported: variables.
Parameter name: keyword.

Environment:
Fabric capacity: F8
Client version: 2606.1.29545-train

DataFlow Gen 2 Error:Fast copy user error

PolinaOr — Thu, 28 May 2026 11:17:02 GMT

Hi, everyone

Getting the following error when trying to load from Lake House. Other 2 tables from it are working fine. Preview on the errorred table is also fine. I don't have fast copy required for this table.

Error Code: Fast Copy User Error, Error Details: Couldn't refresh the entity because of an issue with the mashup document MashupException.Error: DataSource.Error: Pipeline execution failed (runId: ed5c354c-46f1-47ee-a93f-d31a1ba53837). Operation on target ca-9b399285-c5d3-49ce-9dcd-043b12b9c145 failed: ErrorCode=DeltaSnapshotError,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Delta table snapshot doesn't exist.,Source=Microsoft.DataTransfer.ClientLibrary,''Type=Microsoft.Data.DeltaLake.DeltaLakeException,Message=table doesn't exist,Source=Microsoft.Data.DeltaLake,' Details: Reason = DataSource.Error;RunId = ed5c354c-46f1-47ee-a93f-d31a1ba53837 (Request ID: aa7cc8a3-d52e-4a00-bb19-850271030939).

Is ADF Fabric version compactable with Github Proxima on version control?

zihanwang — Tue, 19 May 2026 21:01:07 GMT

Hi, I am an internal Microsoft employee. we're evaluating whether to use Data Factory in Microsoft Fabric for a new SQL→Kusto pipeline. We need Git integration with GitHub Enterprise Server (specifically msft.ghe.com, which uses SAML SSO), instead of Azure DevOps repo. Does Fabric Data Factory support GitHub Enterprise Server as a source control backend, similar to how classic ADF does? If so, are there any known limitations compared to classic ADF's GitHub Enterprise Server integration (e.g., PAT authentication, branch management)? Thanks!

What actually influences CU consumption in Dataflows Gen2?

mrbartuss — Thu, 23 Apr 2026 15:16:38 GMT

I’m trying to better understand what really drives CU consumption in Dataflows Gen2, because my observations don’t fully match my expectations.

I have two Dataflows Gen2, both orchestrated via a pipeline:

Dimensions dataflow

Loads 19 tables into a Lakehouse
All tables come from one Excel file on SharePoint, each from a different sheet
Transformations: only Select Columns
Runtime: ~4 minutes

Facts dataflow

Loads 8 tables
All tables come from different Excel files on SharePoint
Heavy transformations:
- Multiple pivot / unpivot
- Merges
Two tables have >15M rows
Runtime: ~10 minutes

Intuitively, I expected the facts dataflow to consume significantly more CUs due to:

Much larger row counts
More complex transformations

However, In practice, the dimensions dataflow actually consumes more CUs than the facts dataflow.

What are the main factors that actually influence CU consumption in Dataflows Gen2?

Is there any way of editing column mapping with advanced destination editor in Gen2 CICD dataflows?

JoshT — Fri, 10 Apr 2026 15:01:54 GMT

The below documentation refers to a preview feature where we can incorporate variable libraries into Gen2 dataflow destinations. It makes explicit the point that once the editor is used, the destinations GUI will no longer work:

https://learn.microsoft.com/en-us/fabric/data-factory/dataflow-gen2-advanced-edit-data-destinations

Is there any means of updating the underlying column mapping once the editor has been used? The query does not appear to specify any columns, and adding columns into the destination does not appear to automatically map. I've tried switching the workspace and warehouse IDs to string literals (as they were before I edited), but the GUI still doesn't work. Has anyone thought about this workflow and the fact that output columns might change over time?

Only option I can see so far is to delete and recreate the query, which isn't ideal

Variable library usage leads to error with dynamic data destination

JamesMurkin1 — Fri, 10 Apr 2026 12:27:35 GMT

Hi,

I am trying to use a dataflow to load data from an external PostgreSQL database into a Lakehouse. I have dev, test, and prod workspaces created and linked through a deployment pipeline, with identical lakehouses in each. Therefore, as the dataflow goes from one workspace to the next, the destination needs to be managed accordingly (as it doesn't automatically update as part of the deployment process).

I have tried using a variable library (based on this documentation) that contains the Workspace and Lakehouse IDs for each stage, but run into an error.

I create the WorkspaceID variable using this query (with an equivalent one for the lakehouse ID), and disable staging:

This produces the expected value:

However, as soon as I add a table, I get the following error with the variables:

The import Variable.ValueOrDefault matches no exports. Did you miss a module reference?

Details

Reason = Expression.Error

ErrorCode = 10079

Any help would be appreciated - thanks!

Unable to delete auto-generated Semantic Model (empty, no lineage)

bdpr_95 — Thu, 09 Apr 2026 10:21:02 GMT

Hi everyone,

I’m unable to delete an auto-generated Semantic Model (e.g. DataflowsStagingWarehouse1) in a Fabric-enabled workspace.

Context:

The Semantic Model exists in the workspace
It has no tables / no schema
Lineage View shows no dependencies
No reports or other artifacts are connected
I’m Workspace Admin

Issue: When trying to delete it, I get:

Model_FailToRemoveDatasetMessage (400)

Even though the model is completely empty, deletion is blocked, which suggests a hidden backend dependency (possibly from Dataflows Gen2 / Warehouse / Lakehouse staging).

Questions:

Is this a known Fabric issue?
Is there any supported way to clean up these orphan Semantic Models?
Or is Microsoft Support backend cleanup the only option?

Thanks in advance for any confirmation or guidance

Fabric Dataflow Gen 2

negideepika06 — Wed, 25 Mar 2026 13:09:47 GMT

I have one source account file from which I’m deriving 12 different tables in Dataflows, all of which are driven from the same account file.

For example, in the account file there is a column called SegmentAllocation that contains nested JSON. In my scenario, I’m expanding this JSON structure (which is sourced from the account file) into multiple columns/tables.

My dataflow runs successfully for all 12 derived tables plus the main account table. However, for 4 of these tables—whose destination is a Lakehouse—the entire table is populated with NULL values; every row shows NULL for all columns.

I’ve tried several troubleshooting steps:

Dropping the destination tables and recreating them with the same schema as defined in the dataflow.
Ensuring the schema fully matches both the dataflow and the Lakehouse destination.
Re‑mapping the dataflow to the default data destination.

Despite all of this, these 4 tables still end up with completely NULL values.