Best Practices for Handling Incremental Data Loads in Dataflow Gen2
Hi everyone, I am exploring different approaches for handling incremental data loads with Dataflow Gen2 in Microsoft Fabric. For larger datasets, refreshing the entire dataset every time can become inefficient, so I am interested in understanding how others are designing their dataflows to process only new or changed records. A few questions: How are you identifying and filtering changed records between dataflow runs? Is it better to manage incremental logic directly inside Dataflow Gen2, or use a Fabric pipeline to control the process? How do you handle failed runs or partially processed data without creating duplicate records? Are there any recommended patterns for maintaining good performance as the volume of historical data grows? I would appreciate hearing about approaches that have worked well in real Fabric environments.Dataflow gen2: use libraby variable for destination ?
Hello everyone, One of my coworkers decided to use Dataflow Gen2 to simplify maintenance. After a quick analysis of the capacity unit (CU) consumption, the results are fully acceptable — so we agreed to continue using this approach. The only issue I’m facing concerns the destination output configuration. I’d like to set it dynamically as a variable library. According to Microsoft’s documentation, this can be done through the API or via Git. However, after exploring the OneLake File Explorer, I located the mashup script in the correct folder. Even after making changes and saving them, nothing seems to update — the Dataflow Gen2 still uses the previous destination value. Is there another step required to apply or refresh these changes? For context, when I open the Dataflow Gen2 in Fabric, it still shows the old destination, even though the script file in OneLake is correctly synced. Many thanks for your ideas and insights.
[BUG] Dataflow Gen2 destination picker 404 — api.powerbi.com lakehouses UK region
Hi community, We have a complete blocker with Dataflow Gen2 (CI/CD) destination configuration in our UK-region Fabric tenant. The Error: When clicking "Add default destination" and selecting a Fabric Lakehouse from the OneLake catalog, this error fires immediately for ALL users including the workspace Admin: DataSource.Error: Downstream service call to url 'https://api.powerbi.com/v1/workspaces/{WORKSPACE_ID}/lakehouses' failed with status code 404. (Region: uk) The Lakehouse IS visible in the OneLake catalog — it appears in the picker. The 404 fires the moment it is clicked. Root cause hypothesis: The destination picker calls the legacy Power BI REST API: api.powerbi.com/v1/workspaces/{id}/lakehouses ← returns 404 For Fabric-native workspaces in the UK region, the correct endpoint should be: api.fabric.microsoft.com/v1/workspaces/{id}/lakehouses The OneLake catalog listing (which uses a different path) resolves the lakehouse correctly — it's only the destination configuration call that uses the wrong endpoint. Workarounds attempted — all failed: 1. UI — Admin user attempted same steps → same 404 2. queryMetadata.json via Git — Added destination fields, pushed to prod branch → Fabric silently strips loadEnabled/destination on every sync, creating a permanent conflict loop (workspace ≠ Git on every update) 3. Authentication methods — Before the error began firing immediately, a "Connect to default data destination" screen appeared offering two authentication options for the Lakehouse connection: - Organizational account (Sahil Singh / user account) - Service principal Both were attempted. Neither succeeded — the 404 followed in both cases. Subsequently, this connection credentials screen stopped appearing altogether. The 404 now fires immediately upon clicking lh_dhl_prod in the OneLake catalog, before the connection setup dialog is even shown. This regression suggests the failure point has moved earlier in the destination configuration flow. Questions: 1. Is this a known issue for UK-region tenants? 2. Is there a way to configure the Dataflow Gen2 destination via Fabric REST API directly to bypass the broken UI? 3. Are there other file-based approaches (mashup.pq metadata) for CI/CD Dataflow Gen2 destinations? Happy to provide Session IDs and further details to the Fabric team. I have attached the relevant images, PFA... Thanks
Dataflow Gen2 - failing to load into the lakehouse
Hello. I have a dataflow gen2, fetching data from an azure data factory environment and loading them into a Fabric Lakehouse. It has been working flawlessfly for the last 3 months, but since two weeks, on some of the tables, I get this error: "There was a problem refreshing the dataflow: "Something went wrong, please try again later. If the error persists, please contact support.". Error code: EntityUserFailure. (Request ID: 6d111afc-11ee-459f-9e31-db4d08c32c44)." "XXX - ABC Entity Management_WriteToDataDestination: There was a problem refreshing the dataflow: "Couldn't refresh the entity because of an issue with the mashup document MashupException.Error: Error in creating lakehouse table in a version, InnerException: Tables can only contain uppercase or lowercase letters, numbers or the underscore character., Underlying error: Tables can only contain uppercase or lowercase letters, numbers or the underscore character. Details: Reason = Expression.Error;ErrorCode = Lakehouse036;Message = Tables can only contain uppercase or lowercase letters, numbers or the underscore character.;Message.Format = Tables can only contain uppercase or lowercase letters, numbers or the underscore character.;ErrorCode = Lakehouse019;InnerErrorCode = Lakehouse021,Lakehouse019;Microsoft.Data.Mashup.Error.Context = User GatewayObjectId: 1ee1c1a1-a11v-32bt-a19b-5ff6d874f432". Error code: 104100. (Request ID: 66d111afc-11ee-459f-9e31-db4d08c32c44)." The weird part is that there is no consistency between which tables go to error and which ones proceed to being loaded. Sometimes it works sometimes it does not. Also, there are no issue with the naming, as all the columns are respecting the requirements laid out from the error. And above all, it was working flawlessly until few weeks ago and nothing a
Required parameters are missing in the request data flow gen2 fabric
Hi Team, I have created a Dataflow Gen2 in Microsoft Fabric with Dataverse as the source. When I run the dataflow independently, it executes successfully. However, when I invoke the same dataflow from a pipeline and pass the Dataverse URL as a dynamic parameter, the execution fails. Below is the error message for reference: Error: Required parameters are missing in the request Dataflow refresh job failed with status: Failed.
