Forum Discussion
Dataflow Gen2 Dynamic Schema Not Updating
Thanks for the reply, Nono. I ensured the "Dynamic Schema on Publish" was selected and ran a manual refresh of the data flow. Neither action added the new column upon refresh.
Deleting the table and rerunning the dataflow works, but what is the point of the dynamic schema option if I have to still do that manually? It seems like the feature is not working as intended.
Are you able to consistently reproduce this behavior on a completely new Dataflow? If yes, could you share the repro steps so I can try it on my own? If you could use a sample public data source (such as OData NorthWind) that would ease things as well.
Regardless, I'd recommend raising a support ticket and reaching out to the support team so you can get direct help from our engineers to troubleshoot the issue. Below is the link to the support channel:
After creating a few new dataflows and updating existing ones with new columns, I can confirm that this behavior was erratic and not easily reproduced. Sometimes, the new columns would be added upon refresh, and sometimes, they wouldn't. Manually dropping the table and reapplying the data destination settings would fix the issue, but this leads to an unreliable experience.
What I did find this morning is when adding a new column and not specifying the column type (i.e, column type "Any"), the new column would not show part of the new schema. Maybe the root cause is the new column type is not being accepted by the Lakehouse. I suggest trying to change the column type to something supported by the Lakehouse. I'll update this thread if I run into this again and can actually test this.