Skip to main content
cancel
Showing results for 
Search instead for 
Did you mean: 

Find everything you need to get certified on Fabric—skills challenges, live sessions, exam prep, role guidance, and more. Get started

Reply
supri03
Helper I
Helper I

DataFlow gen 2 not updating the data in my Warehouse

Hey Community, 

I have this automated pipeline:

supri03_0-1704973514135.png

 

 

 

The DataFlow performs some transformations and dumps data into the Warehouse. And from the warehouse, I create my report. The pipeline ran successfully, however, the new columns I added in my dataflow are not reflected in my warehouse. Why is this happenning? And how can I make sure the data reflects? Cause I'm getting a lot of data and want it to see the warehouse and dashboard updated. 

 

Thanks !

3 REPLIES 3
fabricator1
Advocate II
Advocate II

Do you mean new columns or new rows?

When using a warehouse table as destination for your dataflow gen 2, I think you need to set up a mapping of columns from the dataflow gen 2 vs. the columns in the data warehouse table. This mapping is done when you set up the table destination in the dataflow gen 2.

I don't think this dynamically changes if you add more columns in your dataflow.

I think you need to add the new columns to the table in the data warehouse, and then edit the mapping of the columns in the dataflow gen 2 so that the columns in the dataflow gen 2 matches the columns in your data warehouse table.

When using Lakehouse I have experienced problems when adding columns in a lakehouse table (the problem wasn't actually in the Lakehouse, but in the SQL analytics endpoint and semantic model of the Lakehouse - the table had disappeared).
My solution was to create a new table from the Dataflow Gen2 which included the new columns . Maybe you will need to do the same thing in Warehouse, if you want to add columns to a table, but I haven't tested that. Maybe you don't need to do that.

Hello @supri03 ,

We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet .
In case if you have any resolution please do share that same with the community as it can be helpful to others .
Otherwise, will respond back with the more details and we will try to help .

Hi @supri03 ,

We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet .
In case if you have any resolution please do share that same with the community as it can be helpful to others .
Otherwise, will respond back with the more details and we will try to help .

Helpful resources

Announcements
FabricCarousel_June2024

Fabric Monthly Update - June 2024

Check out the June 2024 Fabric update to learn about new features.

July Newsletter

Fabric Community Update - July 2024

Find out what's new and trending in the Fabric Community.

Top Kudoed Authors