DataFlow gen 2 not updating the data in my Warehouse

Do you mean new columns or new rows?

When using a warehouse table as destination for your dataflow gen 2, I think you need to set up a mapping of columns from the dataflow gen 2 vs. the columns in the data warehouse table. This mapping is done when you set up the table destination in the dataflow gen 2.

I don't think this dynamically changes if you add more columns in your dataflow.

I think you need to add the new columns to the table in the data warehouse, and then edit the mapping of the columns in the dataflow gen 2 so that the columns in the dataflow gen 2 matches the columns in your data warehouse table.

When using Lakehouse I have experienced problems when adding columns in a lakehouse table (the problem wasn't actually in the Lakehouse, but in the SQL analytics endpoint and semantic model of the Lakehouse - the table had disappeared).
My solution was to create a new table from the Dataflow Gen2 which included the new columns . Maybe you will need to do the same thing in Warehouse, if you want to add columns to a table, but I haven't tested that. Maybe you don't need to do that.

hello, I have noticed the same

My dataflow runs successfully but sometimes the new rows (or just some) are not being reflected in the warehouse table. First I copy tables to a lakehouse and then I do some transformations in the dataflow and push it to the warehouse. When I see that new lines are not reflected, I check if the information is in the lakehouse, and it is there in fact but for some reason it gets lost in the data flow. I have also checked if it is available in the datalfow (to make sure the transformations are not excluding them somehow) but they are there.

Is there something I can do to make sure the data is correctly updated?

Hello, I had the same issue some days ago.
I have a Dataflow Gen2 that reads many tables from a Lakehouse, makes some transformation and finally write modified data in a Warehouse overwriting the destination tables.
Although execution logs indicate that data was read and written without errors, tables in the Warehouse doesn't contain any new or updated data, as if no overwrite happened. Except for the CALENDAR table, that is generated inside the Dataflow Gen2 using only M code (no source from the Lakehouse).
In the Lakehouse, new and updated records are available. Checking Dataflow steps today, I can see those records.

I would know why Dataflow didn't update my Warehouse.

Hi,

in the last month I've read many posts and articles about this problem. I discovered it's an architectural feature: Lakehouse SQL Endpoint needs some time to get refreshed after table writing. So, downstream tasks may not get the newly updated data until refresh is completed.
In my scenario the problem is occasional and it started to trigger in december, six month later the first run of the pipeline.

What I've discovered today is the official API that force the refresh of the entire SQL Endpoint of a Lakehouse. Here you can find all the details about this API. Also, in this Blog Post you can find the meaning of each return value.

Hi,

This is a common gotcha with Dataflow Gen2 → Warehouse destinations. The column mapping is static, it's set once when you configure the destination and doesn't auto-update when you change your dataflow schema.

Root cause: When you add new columns to your dataflow transformation, the existing column mapping in the data destination settings still references the old schema. New columns are silently ignored.

How to fix it:

Update the Warehouse table schema first Add the new columns to your Warehouse table using T-SQL:

ALTER TABLE dbo.YourTable ADD NewColumn1 NVARCHAR(255), NewColumn2 INT;
Reconfigure the Dataflow destination mapping

Open your Dataflow Gen2
Click on the data destination step
Click Settings, go to the Column mapping section
You should now see the new columns available for mapping
Map source columns, destination columns
Refresh and validate Run the dataflow and check the warehouse table.

For handling schema changes at scale, check out the official guidance on schema drift: How to handle schema drift in Dataflow Gen2 , Microsoft Learn

Pro tip: If your schema changes frequently, consider using a Lakehouse as destination instead of a Warehouse. Lakehouse tables (Delta format) handle schema evolution more gracefully with mergeSchema:

df.write.format("delta").option("mergeSchema", "true").mode("append").save("Tables/your_table")
Then create a Warehouse SQL view on top of the Lakehouse SQL Endpoint for reporting.

Reference: Dataflow Gen2 data destinations and managed settings

Hope this helps! Give a kudos. Let me know if you need more details on the intermediate landing zone approach.