- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark
- Subscribe
- Printer Friendly Page
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
DataFlow gen 2 not updating the data in my Warehouse
Hey Community,
I have this automated pipeline:
The DataFlow performs some transformations and dumps data into the Warehouse. And from the warehouse, I create my report. The pipeline ran successfully, however, the new columns I added in my dataflow are not reflected in my warehouse. Why is this happenning? And how can I make sure the data reflects? Cause I'm getting a lot of data and want it to see the warehouse and dashboard updated.
Thanks !
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
hello, I have noticed the same
My dataflow runs successfully but sometimes the new rows (or just some) are not being reflected in the warehouse table. First I copy tables to a lakehouse and then I do some transformations in the dataflow and push it to the warehouse. When I see that new lines are not reflected, I check if the information is in the lakehouse, and it is there in fact but for some reason it gets lost in the data flow. I have also checked if it is available in the datalfow (to make sure the transformations are not excluding them somehow) but they are there.
Is there something I can do to make sure the data is correctly updated?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Do you mean new columns or new rows?
When using a warehouse table as destination for your dataflow gen 2, I think you need to set up a mapping of columns from the dataflow gen 2 vs. the columns in the data warehouse table. This mapping is done when you set up the table destination in the dataflow gen 2.
I don't think this dynamically changes if you add more columns in your dataflow.
I think you need to add the new columns to the table in the data warehouse, and then edit the mapping of the columns in the dataflow gen 2 so that the columns in the dataflow gen 2 matches the columns in your data warehouse table.
When using Lakehouse I have experienced problems when adding columns in a lakehouse table (the problem wasn't actually in the Lakehouse, but in the SQL analytics endpoint and semantic model of the Lakehouse - the table had disappeared).
My solution was to create a new table from the Dataflow Gen2 which included the new columns . Maybe you will need to do the same thing in Warehouse, if you want to add columns to a table, but I haven't tested that. Maybe you don't need to do that.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hello @supri03 ,
We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet .
In case if you have any resolution please do share that same with the community as it can be helpful to others .
Otherwise, will respond back with the more details and we will try to help .
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi @supri03 ,
We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet .
In case if you have any resolution please do share that same with the community as it can be helpful to others .
Otherwise, will respond back with the more details and we will try to help .
Helpful resources
Subject | Author | Posted | |
---|---|---|---|
12-04-2024 07:19 PM | |||
09-03-2024 06:29 AM | |||
10-07-2024 01:25 AM | |||
09-13-2024 04:44 AM | |||
03-25-2024 10:20 AM |
User | Count |
---|---|
2 | |
1 | |
1 | |
1 | |
1 |