When saving, Premium Dataflows almost consistently add two steps to my logic - Table.TransformColumnTypes and Table.ReplaceErrorValues. These appear without my request and cover multiple columns in the table. The UI will simply add them at the end of my logic. Although sometimes they cause no harm, there are instances where they consistently create problems.
For example, if using the enhanced compute engine, the performance-based recommendation is to have a staging or landing zone dataflow that simply pulls data from sources in their raw format (see "Using the enhanced compute engine" of the article "Premium features of dataflows"). Downstream dataflows then link to the staged dataflow tables rather than going to original sources. We have a large Smartsheet that undergoes structural (column insert/update or rename) modifications more than once a week. When I stage the smartsheet, my logic has only Source and Navigation steps, but then the dataflow UI Save operation adds the TransformColumnTypes and ReplaceErrorValues on every column in the table. This results in frequent daily failures when a column gets removed from or renamed in the smartsheet.
- Why are these two steps being added?
- Is there an off switch for this "feature?"
- If no off switch, what workarounds would you suggest? RemoveOtherColumns might be useful if I could spend the time restricting to only the columns to be used throughout the project. Any other suggestions?
By the way, when an error occurs due to these additions, I open the dataflow UI and delete the two steps. Upon saving, the dataflow UI then rebuilds the two steps with only the available columns.
Thank you in advance for your help.
