Reply
supri03
Helper I
Helper I

DataFlow gen 2 not updating the data in my Warehouse

Hey Community, 

I have this automated pipeline:

supri03_0-1704973514135.png

 

 

 

The DataFlow performs some transformations and dumps data into the Warehouse. And from the warehouse, I create my report. The pipeline ran successfully, however, the new columns I added in my dataflow are not reflected in my warehouse. Why is this happenning? And how can I make sure the data reflects? Cause I'm getting a lot of data and want it to see the warehouse and dashboard updated. 

 

Thanks !

4 REPLIES 4
ilseeb
Advocate I
Advocate I

hello, I have noticed the same

My dataflow runs successfully but sometimes the new rows (or just some) are not being reflected in the warehouse table. First I copy tables to a lakehouse and then I do some transformations in the dataflow and push it to the warehouse. When I see that new lines are not reflected, I check if the information is in the lakehouse, and it is there in fact but for some reason it gets lost in the data flow. I have also checked if it is available in the datalfow (to make sure the transformations are not excluding them somehow) but they are there.

Is there something I can do to make sure the data is correctly updated?

fabricator1
Advocate II
Advocate II

Do you mean new columns or new rows?

When using a warehouse table as destination for your dataflow gen 2, I think you need to set up a mapping of columns from the dataflow gen 2 vs. the columns in the data warehouse table. This mapping is done when you set up the table destination in the dataflow gen 2.

I don't think this dynamically changes if you add more columns in your dataflow.

I think you need to add the new columns to the table in the data warehouse, and then edit the mapping of the columns in the dataflow gen 2 so that the columns in the dataflow gen 2 matches the columns in your data warehouse table.

When using Lakehouse I have experienced problems when adding columns in a lakehouse table (the problem wasn't actually in the Lakehouse, but in the SQL analytics endpoint and semantic model of the Lakehouse - the table had disappeared).
My solution was to create a new table from the Dataflow Gen2 which included the new columns . Maybe you will need to do the same thing in Warehouse, if you want to add columns to a table, but I haven't tested that. Maybe you don't need to do that.

avatar user
Anonymous
Not applicable

Hello @supri03 ,

We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet .
In case if you have any resolution please do share that same with the community as it can be helpful to others .
Otherwise, will respond back with the more details and we will try to help .

avatar user
Anonymous
Not applicable

Hi @supri03 ,

We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet .
In case if you have any resolution please do share that same with the community as it can be helpful to others .
Otherwise, will respond back with the more details and we will try to help .

avatar user

Helpful resources

Announcements
Las Vegas 2025

Join us at the Microsoft Fabric Community Conference

March 31 - April 2, 2025, in Las Vegas, Nevada. Use code MSCUST for a $150 discount!

December 2024

A Year in Review - December 2024

Find out what content was popular in the Fabric community during 2024.

Top Solution Authors (Last Month)
Top Kudoed Authors (Last Month)