Forum Discussion
Helper IFew columns are missing from Lakehouse tables when fetched via Dataflow Gen2
Hi Nitesh,
Could you please also confirm that you have re-edited the mapping (in case you added columns after the initial mapping)? We do not yet have a mode where the mapping is automatically updated everytime you add a column. You have to explictly edit the Output Destinations settings.
Thanks
Community AdminCould you confirm if these columns are available in the Lakehouse and the issue is simply visualizing them on Dataflows Gen2? or is the issue that the columns do not exist in the lakehouse despite you defining those columns to be created by your original dataflow gen2 with the lakehouse as the output destination?
Microsoft EmployeeHi Nitesh,
Could you please also confirm that you have re-edited the mapping (in case you added columns after the initial mapping)? We do not yet have a mode where the mapping is automatically updated everytime you add a column. You have to explictly edit the Output Destinations settings.
Thanks
Dan44SidJay unfortunately that doesn't work. When I add a new column, despite entering the Data Destination settings within my dataflow and selecting "Refresh destination schema" and/or "Reset column mappings", my new column is not to be found. Only the current columns from the already published table in the lakehouse are showing. Funny enough I can see my column in the dropdown in Source but of course, if I change this to something which contridicts its current destination the dataflow will fail.
Like someone else commented, I am having to manually delete the table in the lakehouse and republish it - along with all of the configuration that goes into my dataset. It is very frustrating, especially since Microsoft has disabled the ability to refresh data for published Excel reports in the workspace, where the report is built through Power Pivot.
- miguel
Hi!
I believe the scenario that you're describing is a bit different from the one mentioned in this topic.
The scenario that you're describing sounds more of a dynamic query that can output more columns than when the first data destination was setup.
However, the scenario from this thread is around how the first time a data destination was set it had all columns, but some of them were not showing in the data destination after the refresh succeded for the first time.
Given that your scenario described is different, please go ahead and create a new topic / thread in this forum so we can have a better understanding of it.
- Dan44
Hi miguel thank you for your reply.
I'm not sure what you mean by a dynamic query. My issue boils down to I have a dataflow gen2 with the output destination to a lakehouse, and then if I insert ANY new column, I cannot add this to my current destination table. The only way to include this new column is to proceed to delete the current table in the lakehouse and republish. This appears because since the current table doesn't already have this new column present as a destination, it simply limits adding it (i.e. although I can see the new column in the dropdown for Source within the Data destination settings, there is no place to configure it to in Current destination).
Is this however expected behaviour?
Regards
NiteshSI used Dataflow Gen 2 to get my Tables from Dataverse. I did not add any new columns and did not change the mapping as well because the Type was fine for all the columns.
I think just choosing the destination folder solved this issue.
For me the problem was that the columns I added were untyped (any). Once I made sure they were typed as text, they started to become visibly in my table.