Dataflow topics

Dataflow Gen2 parameter works with current value but fails when passed from Fabric Pipeline

eyeampraveen — Tue, 27 Oct 2026 14:36:11 GMT

I have a Dataflow Gen2 that calls a SOAP API using Web.Contents().

The Dataflow has an APIKey Text parameter, which I use in Power Query:

v_apikey = Text.From(APIKey)

The value is inserted into the SOAP request header/body before making the Web.Contents() call.

The issue:

API key entered as the Dataflow parameter's current value → Works
Exact same API key passed from a Fabric Pipeline parameter → Fails

The failure eventually results in:

Expression.Error:
There weren't enough elements in the enumeration to complete the operation.
The SOAP XML navigation also shows an empty table:
#table({"Name", "Table"}, {})

I'm trying to understand the runtime execution path:

Pipeline parameter

→ Dataflow activity

→ Dataflow parameter

→ Power Query

→ Web.Contents()

Does Dataflow Gen2 apply Pipeline parameter overrides before Power Query/Web.Contents is evaluated, including any metadata evaluation?

Are there known limitations when a runtime Dataflow parameter is used inside the POST body of Web.Contents()?

Also, is there a safe way to verify that the Pipeline-supplied API key reached Power Query correctly without exposing the key itself?

Unable to Submit a Microsoft Fabric Technical Support Ticket: Only 'Add New Contract' Available

ShubhamJain2 — Tue, 27 Oct 2026 10:01:17 GMT

Hello Microsoft Fabric Community,

I am seeking assistance with accessing Microsoft Fabric technical support for an unresolved Dataflow Gen2 authentication issue.

I have an existing discussion covering the complete technical investigation, troubleshooting history and diagnostic results:

Dataflow Gen2 Lakehouse connection fails with AADSTS70008 | Original Discussion

Rather than repeating the entire troubleshooting history, I am opening this focused discussion to seek help with the remaining support-access issue.

1. Brief technical background

My original Dataflow Gen2 connection to LakeHouse_Training in the Nikolai_Schuler_Udemy workspace fails with:

AADSTS70008: The provided authorization code or refresh token has expired due to inactivity.

Attempts to update credentials or create a new connection to the original Lakehouse also result in an Invalid credentials message.

I completed a clean-room diagnostic test using the same Microsoft Entra tenant, organizational account and Fabric trial capacity.

The results were:

Created a new workspace: Fabric_OAuth_Diagnostic.
Created a new Lakehouse: LH_OAuth_Diagnostic.
Created a new Dataflow Gen2: DF_OAuth_Diagnostic.
Successfully established a fresh Organizational account connection.
Successfully imported CSV data and loaded it into a new Lakehouse table.
The Dataflow execution completed in approximately 22 seconds.

However, connecting to the original LakeHouse_Training from the same diagnostic environment continues to fail.

This indicates that the issue appears to be specific to the original Lakehouse or its connection/authentication path. The underlying root cause remains unconfirmed.

2. Microsoft Fabric technical support access issue

I attempted to obtain technical assistance through the official Microsoft Fabric Support portal and the Contact Support option in Power BI/Fabric.

However, the Support plan dropdown only displays:

Add new contract

The Microsoft Fabric Subscription option is missing, and I cannot proceed with submitting a technical support request.

My environment:

Microsoft Fabric trial capacity.
Fabric Free per-user license.
Organizational member account in a Microsoft Entra tenant.
Global Administrator permissions in the Entra tenant.

I do not have a Unified or Premier support contract.

I also previously submitted an Azure Basic Support request for this issue, but it was closed because technical break-fix support is not covered by that plan.

3. Microsoft documentation

According to Microsoft's current documentation, users with a Fabric Free license can create support tickets for Fabric products.

The documentation also states that the support-plan selection can be made using Microsoft Fabric Subscription or a Unified/Premier support plan. The Add new contract option is intended for users who already have a Unified or Premier contract that is not listed.

Documentation:

https://learn.microsoft.com/en-us/power-bi/support/create-support-ticket

Unfortunately, my support form does not display the documented Microsoft Fabric Subscription option.

4. Assistance requested

Could someone from Microsoft Fabric Support or the Community Support team please clarify:

Why is the Microsoft Fabric Subscription option missing from the support-plan dropdown?
Is there any additional prerequisite or account configuration required for a Fabric Free user with an active trial capacity to submit a technical support ticket?
Is there an alternative route to obtain a technical escalation for this Dataflow Gen2/Lakehouse connection issue?

I would particularly appreciate assistance from a Microsoft Fabric employee or someone who can route this to the appropriate Data Factory or Lakehouse connector support team.

I can provide the detailed Entra sign-in logs, trace IDs, correlation IDs, timestamps and other diagnostic information privately if required.

Thank you for your time and assistance.

Dataflow Gen2 Lakehouse connection fails with AADSTS70008

ShubhamJain2 — Tue, 20 Oct 2026 13:41:30 GMT

I am learning Microsoft Fabric and preparing for the DP-600 certification. I have spent several days trying to resolve a Dataflow Gen2 Lakehouse authentication failure, and I need help identifying the root cause.

Environment

Microsoft Fabric trial capacity, with a Fabric Free per-user license.
Personal Azure Free account and a newly created Microsoft Entra tenant.
Workspace: Nikolai_Schuler_Udemy.
Lakehouse: LakeHouse_Training.
Dataflow: Dataflow 1.
Authentication: Organizational account, OAuth 2.0.
Privacy level: Organizational.
Gateway: None.

The Entra tenant was originally created through an Azure signup using a personal Gmail account. I created an onmicrosoft.com member account with Global Administrator permissions and use that account for Fabric. The original Gmail identity exists in the tenant as a guest.

Problem

Dataflow Gen2 cannot connect to the Lakehouse. The query editor displays “Credentials are required to connect to the Lakehouse source.”

When creating or updating the connection, Fabric reports:

AADSTS70008: The provided authorization code or refresh token has expired due to inactivity. Send a new interactive authorization request for this user and resource.

HTTP status: 400
Cluster: https://api.powerbi.com

The same failure occurs when editing credentials for an existing connection. The account picker can open, but reauthentication has not restored access. A newly created connection appears in Manage Connections, yet the Dataflow editor still cannot connect.

Troubleshooting completed

Tested Chrome, Edge, and Edge InPrivate.
Retried interactive sign-in using the onmicrosoft.com member account.
Confirmed Microsoft Authenticator MFA is registered.
Tested existing and newly created Lakehouse connections.
Used Edit credentials in Manage Connections and Gateways.
Confirmed the Lakehouse and SQL analytics endpoint are accessible in Fabric.
Enabled GitHub integration and successfully synchronized the workspace to a public GitHub repository. This did not resolve the authentication error.
Checked Entra sign-in logs. A previous Power BI Data Refresh sign-in failed with error 70008, and the credential update continues to fail with the same error.

Support-access problem

Microsoft's current support documentation says Fabric Free users can submit support tickets for Fabric products. However, the Fabric Help and Support form only offers “Add new contract”; “Microsoft Fabric Subscription” is missing.

I submitted an Azure subscription-management request, but it was closed because Basic Support does not cover technical issues. I still have no technical support case.

Questions

Why does a fresh OAuth authorization continue to fail with AADSTS70008 in this new tenant?
Is there a known issue with Dataflow Gen2, Power Query Online, or the Lakehouse connector's OAuth token handling?
Could the tenant identity setup or Fabric trial capacity affect OAuth authorization or Fabric support eligibility?
How can I obtain a proper Fabric technical support escalation as a Fabric Free user?

Could a Microsoft Fabric employee or Data Factory specialist investigate this and advise on the next diagnostic step? I can provide the full trace ID, correlation ID, request ID, tenant details, and screenshots privately if needed.

Best Practices for Handling Incremental Data Loads in Dataflow Gen2

codeautomation — Fri, 18 Sep 2026 15:26:56 GMT

Hi everyone,

I am exploring different approaches for handling incremental data loads with Dataflow Gen2 in Microsoft Fabric.

For larger datasets, refreshing the entire dataset every time can become inefficient, so I am interested in understanding how others are designing their dataflows to process only new or changed records.

A few questions:

How are you identifying and filtering changed records between dataflow runs?

Is it better to manage incremental logic directly inside Dataflow Gen2, or use a Fabric pipeline to control the process?

How do you handle failed runs or partially processed data without creating duplicate records?

Are there any recommended patterns for maintaining good performance as the volume of historical data grows?

I would appreciate hearing about approaches that have worked well in real Fabric environments.

Configure Dataflow Gen2

UncleLewis — Tue, 15 Sep 2026 20:31:27 GMT

Hi,

Using DataFlow Gen2
I don't see a gear icon anywhere near default destination so I can configure

What am I missing?
Thanks,
w

[BUG] Dataflow Gen2 destination picker 404 — api.powerbi.com lakehouses UK region

sahillihas20 — Wed, 26 Aug 2026 23:16:25 GMT

Hi community,

We have a complete blocker with Dataflow Gen2 (CI/CD) destination configuration in our UK-region Fabric tenant.

The Error:
When clicking "Add default destination" and selecting a Fabric Lakehouse from the OneLake catalog, this error fires immediately for ALL users including the
workspace Admin:
DataSource.Error: Downstream service call to url 'https://api.powerbi.com/v1/workspaces/{WORKSPACE_ID}/lakehouses' failed with status code 404. (Region: uk)

The Lakehouse IS visible in the OneLake catalog — it appears in the picker. The 404 fires the moment it is clicked.

Root cause hypothesis:
The destination picker calls the legacy Power BI REST API: api.powerbi.com/v1/workspaces/{id}/lakehouses ← returns 404

For Fabric-native workspaces in the UK region, the correct endpoint should be: api.fabric.microsoft.com/v1/workspaces/{id}/lakehouses

The OneLake catalog listing (which uses a different path) resolves the lakehouse correctly — it's only the destination configuration call that uses the wrong endpoint.

Workarounds attempted — all failed:
1. UI — Admin user attempted same steps → same 404
2. queryMetadata.json via Git — Added destination fields, pushed to prod branch → Fabric silently strips loadEnabled/destination on every sync, creating a permanent conflict loop (workspace ≠ Git on every update)
3. Authentication methods — Before the error began firing immediately, a "Connect to default data destination" screen appeared offering two authentication options for the Lakehouse connection:
- Organizational account (Sahil Singh / user account)
- Service principal

Both were attempted. Neither succeeded — the 404 followed in both cases.

Subsequently, this connection credentials screen stopped appearing altogether. The 404 now fires immediately upon clicking lh_dhl_prod in the OneLake catalog, before the connection setup dialog is even shown. This regression suggests the failure point has moved earlier in the destination configuration flow.

Questions:
1. Is this a known issue for UK-region tenants?
2. Is there a way to configure the Dataflow Gen2 destination via Fabric REST API directly to bypass the broken UI?
3. Are there other file-based approaches (mashup.pq metadata) for CI/CD Dataflow Gen2 destinations?

Happy to provide Session IDs and further details to the Fabric team.

I have attached the relevant images, PFA...

Thanks

iifnull() failing on non-null input

cmjcf — Mon, 17 Aug 2026 03:40:08 GMT

The documentation for iifNull() says this:

Returns the first not null item when given two or more inputs.

We have a dataflow which handles the contents of a number of files, using a specified CSV format. One of the columns is named 'Dealer'. Due to an error in an upstream system, a significant number of files came in with the header misspelled 'Delaer'. There are so many of these files that we cannot simply edit them manually to fix. I have the following expression in a Derived Column transform:

iifNull('Dealer',byName('Delaer'))

I understand this expression to mean "Use the value of the column 'Dealer' if it is not null, else use the value of the drifted column 'Delaer'." (That name is not in the list set by the dataset, so it is drifted, and leaving out byName() causes validation to fail.)

When the file does not contain the spelling error, my expected result is that it simply retains the correct value in 'Dealer' because this is not null. However, when actually run, this expression actually returns null, and causes the column in the Derived Column transform to not be created, resulting in a "Resolved attribute(s) [name] missing from [names]" error.

Dataflow Gen2 - failing to load into the lakehouse

gabrifly — Thu, 23 Jul 2026 12:39:15 GMT

Hello. I have a dataflow gen2, fetching data from an azure data factory environment and loading them into a Fabric Lakehouse. It has been working flawlessfly for the last 3 months, but since two weeks, on some of the tables, I get this error:

"There was a problem refreshing the dataflow: "Something went wrong, please try again later. If the error persists, please contact support.". Error code: EntityUserFailure. (Request ID: 6d111afc-11ee-459f-9e31-db4d08c32c44)."

"XXX - ABC Entity Management_WriteToDataDestination: There was a problem refreshing the dataflow: "Couldn't refresh the entity because of an issue with the mashup document MashupException.Error: Error in creating lakehouse table in a version, InnerException: Tables can only contain uppercase or lowercase letters, numbers or the underscore character., Underlying error: Tables can only contain uppercase or lowercase letters, numbers or the underscore character. Details: Reason = Expression.Error;ErrorCode = Lakehouse036;Message = Tables can only contain uppercase or lowercase letters, numbers or the underscore character.;Message.Format = Tables can only contain uppercase or lowercase letters, numbers or the underscore character.;ErrorCode = Lakehouse019;InnerErrorCode = Lakehouse021,Lakehouse019;Microsoft.Data.Mashup.Error.Context = User GatewayObjectId: 1ee1c1a1-a11v-32bt-a19b-5ff6d874f432". Error code: 104100. (Request ID: 66d111afc-11ee-459f-9e31-db4d08c32c44)."

The weird part is that there is no consistency between which tables go to error and which ones proceed to being loaded. Sometimes it works sometimes it does not. Also, there are no issue with the naming, as all the columns are respecting the requirements laid out from the error. And above all, it was working flawlessly until few weeks ago and nothing a

Data pipeline failed. How do you troubleshoot?

rajan889262 — Wed, 15 Jul 2026 13:52:36 GMT

Data pipeline failed. How do you troubleshoot?

Required parameters are missing in the request data flow gen2 fabric

Dharma_Teja_M — Tue, 07 Jul 2026 10:35:49 GMT

Hi Team,

I have created a Dataflow Gen2 in Microsoft Fabric with Dataverse as the source. When I run the dataflow independently, it executes successfully.

However, when I invoke the same dataflow from a pipeline and pass the Dataverse URL as a dynamic parameter, the execution fails.

Below is the error message for reference:

Error:

Required parameters are missing in the request

Dataflow refresh job failed with status: Failed.

Data Mapping Flow with DataLake as Sink

LaraHaegele — Wed, 01 Jul 2026 12:52:15 GMT

Hello everyone,
I am currently trying to read data from a source as changelog and writing it into a merged DataLake table. So basically cdc connector logic but as for SHIR the connector is not supported.
I always get the issue:
Operation on target merge_lakehouse_landing_to_delta failed: {"StatusCode":"DFExecutorUserError","Message":"Job failed due to reason: com.microsoft.dataflow.Issues: DF-SRC-002 - 'tableName' (Table Name) is required - \naction sink(\n) ~> sinkName,EXE-0001,Dataflow cannot be analyzed as a graph,[564 737]","Details":""}
I use scriptLines to transform the lines inside the dataflow and if I provide json File as sink the pipeline runs without errors at least. If I provide a lakehouse table even if I hardcode the table name in the dataset I get this error (already recreated it so an old version of the dataset cannot be the issue).
Why is that so can I not merge the data directly into lakehouse or is there another issue I've overseen why lakehouse doesn't work as sink here?
Thank you in advance
Lara

ELT JSON Semi Structured Loop

icassiem — Tue, 30 Jun 2026 07:27:33 GMT

Good day,

Apologies this is a bit of a duplicated but because i only recently have plan approved, so i dont get the time to do deep research or learning as yet

Solved: Re: ELT JSON / Semi structured - Microsoft Fabric Community

My source are mostl API JSON that i have to transform but before i can transform i need to loop, i dont know if dataflow is like powerbi powrqury window or i do it in powerbi one example and past the script in dataflow but how would i do api loops then unpivot the json, would i need pythin to first downlad all looped api content then pass on dataflow etc?

Regards

Dataflow Activity Parameters Lose Expression and Get Hardcoded After Reopening Pipeline

svac — Mon, 29 Jun 2026 13:09:44 GMT

Hi! I’m using a Dataflow Gen2 activity in a Fabric pipeline and passing parameters via dynamic content (pipeline library variables).

When I configure the parameters, they are correctly stored as expressions:

"value": {

"value":

"@pipeline().libraryVariables.Data_engineering_Variable_Library_workspace_id",

"type": "Expression"

}

The pipeline validates and saves successfully. However, when I reopen the pipeline, Fabric rewrites the parameter values as evaluated static string:

"value": "f31c2823-10f1-4e97-bc15-6f39f4981d57",

"type": "String"

This removes the expression and effectively hardcodes the value.

Is there a supported way to persist dynamic expressions for Dataflow Gen2 parameters in pipeline activities?
Are there recommended patterns for passing workspaceId/lakehouseId dynamically without the UI flattening expressions?

Thanks!!

Dataflow Gen2 authentication to Lakehouse without user dependency?

Parchitect — Mon, 29 Jun 2026 07:19:31 GMT

Hi,

We have run into a challenge with Dataflow Gen2 in Microsoft Fabric.

When creating a dataflow and selecting a destination, for example a Lakehouse in the same workspace, the authentication appears to require an Organizational account. In practice, this means that the connection becomes tied to a specific user.

This creates governance and operational challenges in larger organizations. If the user who created or authenticated the dataflow leaves the company, changes role, or loses access, someone else needs to take over the dataflow and re-authenticate the connection. In an organization with thousands of users, this quickly becomes difficult to manage and track.

I understand that in some scenarios it may be possible to use a service principal, but that raises another question: how many service principals should be created and maintained? Should it be one per workspace, per domain, per team, per solution, or something else?

For this specific scenario, Workspace identity feels like a much more natural fit. A workspace-scoped identity would reduce the dependency on individual user accounts and make the solution more robust over time.

Microsoft’s documentation describes Workspace identity as a Fabric-managed service principal associated with a workspace, where Fabric manages the credentials and can use the identity with resources that support Microsoft Entra authentication. The Lakehouse connector documentation also shows that, for Dataflow Gen2, the supported authentication type is currently Organizational account.

My questions are:

What is the recommended best practice for authenticating Dataflow Gen2 when writing to a Lakehouse in the same workspace, without creating a dependency on a personal user account?
Is using a service principal the recommended approach today?
If so, what is the recommended pattern: one service principal per workspace, per team, per solution, or something else?
Are there any plans to support Workspace identity for this scenario?

It would be very helpful to hear how other organizations are handling this, especially in larger environments where ownership changes and user lifecycle management can otherwise become quite difficult.

Thanks in advance!

Dataflow Gen2 Copying Old Data

Russell_DC — Tue, 23 Jun 2026 20:58:30 GMT

I have a dataflow that is transforming data and then moving it to a staging table in a Silver Lakehouse. I keep having issues where the number of rows moved to that lakehouse doesnt match what I am seeing in the dataflow preview or what it should be. A simple example of this is that this should have read 2 and written 2 rows but instead it read and wrote 3 rows. Upon investigating this was the 3 rows from the previos time it ran, suggesting a caching issue! I have others where I have 578 new rows in my table within the dataflow and then in the Staging table it has 212 rows. I'm very confused as to what is going on and concerned I may need to convert to notebooks for my transformations?

amazon redshift ODBC driver

mklevemann — Wed, 17 Jun 2026 20:13:41 GMT

I was told by the vendor that provides me data in an amazon redshift data warehouse that the ODBC 1.x connector will not be working for desktop and I need to move to the 2.x connector. My question is will this ODBC change also affect Dataflow Gen2 accessing the amazon redshift data warehouse?

Gen2 dataflow read only mode issue

Lokamba — Mon, 15 Jun 2026 13:38:08 GMT

Hi Team,

I have admin access to the workspace. When I opened the Gen2 flow created by Rahul R., I was unable to edit it, because it is showing in read-only mode. Could you please help to fix this issue.

"Keyword not supported: variables" error using Dataflow gen2 when evaluating Fabric.Warehouse()

JuanFcoCalquin — Tue, 09 Jun 2026 21:24:09 GMT

Dataflow Gen2 fails in Power Query editor when evaluating Fabric.Warehouse() navigation query.

The issue occurs before refresh execution, while editing/evaluating standard M steps. It also occurs with a basic Dataflow Gen2 using Enter Data as source and Fabric Warehouse as destination.

Error:
Keyword not supported: variables.
Parameter name: keyword.

Environment:
Fabric capacity: F8
Client version: 2606.1.29545-train

DataFlow Gen 2 Error:Fast copy user error

PolinaOr — Thu, 28 May 2026 11:17:02 GMT

Hi, everyone

Getting the following error when trying to load from Lake House. Other 2 tables from it are working fine. Preview on the errorred table is also fine. I don't have fast copy required for this table.

Error Code: Fast Copy User Error, Error Details: Couldn't refresh the entity because of an issue with the mashup document MashupException.Error: DataSource.Error: Pipeline execution failed (runId: ed5c354c-46f1-47ee-a93f-d31a1ba53837). Operation on target ca-9b399285-c5d3-49ce-9dcd-043b12b9c145 failed: ErrorCode=DeltaSnapshotError,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Delta table snapshot doesn't exist.,Source=Microsoft.DataTransfer.ClientLibrary,''Type=Microsoft.Data.DeltaLake.DeltaLakeException,Message=table doesn't exist,Source=Microsoft.Data.DeltaLake,' Details: Reason = DataSource.Error;RunId = ed5c354c-46f1-47ee-a93f-d31a1ba53837 (Request ID: aa7cc8a3-d52e-4a00-bb19-850271030939).

Is ADF Fabric version compactable with Github Proxima on version control?

zihanwang — Tue, 19 May 2026 21:01:07 GMT

Hi, I am an internal Microsoft employee. we're evaluating whether to use Data Factory in Microsoft Fabric for a new SQL→Kusto pipeline. We need Git integration with GitHub Enterprise Server (specifically msft.ghe.com, which uses SAML SSO), instead of Azure DevOps repo. Does Fabric Data Factory support GitHub Enterprise Server as a source control backend, similar to how classic ADF does? If so, are there any known limitations compared to classic ADF's GitHub Enterprise Server integration (e.g., PAT authentication, branch management)? Thanks!