Earn a 50% discount on the DP-600 certification exam by completing the Fabric 30 Days to Learn It challenge.
Hey Community,
I have this automated pipeline:
The DataFlow performs some transformations and dumps data into the Warehouse. And from the warehouse, I create my report. The pipeline ran successfully, however, the new columns I added in my dataflow are not reflected in my warehouse. Why is this happenning? And how can I make sure the data reflects? Cause I'm getting a lot of data and want it to see the warehouse and dashboard updated.
Thanks !
Do you mean new columns or new rows?
When using a warehouse table as destination for your dataflow gen 2, I think you need to set up a mapping of columns from the dataflow gen 2 vs. the columns in the data warehouse table. This mapping is done when you set up the table destination in the dataflow gen 2.
I don't think this dynamically changes if you add more columns in your dataflow.
I think you need to add the new columns to the table in the data warehouse, and then edit the mapping of the columns in the dataflow gen 2 so that the columns in the dataflow gen 2 matches the columns in your data warehouse table.
When using Lakehouse I have experienced problems when adding columns in a lakehouse table (the problem wasn't actually in the Lakehouse, but in the SQL analytics endpoint and semantic model of the Lakehouse - the table had disappeared).
My solution was to create a new table from the Dataflow Gen2 which included the new columns . Maybe you will need to do the same thing in Warehouse, if you want to add columns to a table, but I haven't tested that. Maybe you don't need to do that.
Hello @supri03 ,
We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet .
In case if you have any resolution please do share that same with the community as it can be helpful to others .
Otherwise, will respond back with the more details and we will try to help .
Hi @supri03 ,
We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet .
In case if you have any resolution please do share that same with the community as it can be helpful to others .
Otherwise, will respond back with the more details and we will try to help .