Forum Discussion
Dataflow Gen 2 query or table changes appear to cause errors or failures
If I'm understading your scenario correctly:
- You're setting the output destination of queries to the Lakehouse (tables)
- You're modufying the queries in ways that change the schema
- After those changes, refresh is failing
If so, the way to address this is to reconfigure the output destination after the schema changes. In the Query Settings pane of the Query Editor, you will see the Output Destination section at the bottom. Clicking on the "X" will remove the current settings and then you can reconfigure the destination (including specifying a new column mapping).
We are looking into a future mode that does not require explicit remapping for disruptive schema changes.
If I'm understading your scenario correctly:
- You're setting the output destination of queries to the Lakehouse (tables)
- You're modufying the queries in ways that change the schema
- After those changes, refresh is failing
If so, the way to address this is to reconfigure the output destination after the schema changes. In the Query Settings pane of the Query Editor, you will see the Output Destination section at the bottom. Clicking on the "X" will remove the current settings and then you can reconfigure the destination (including specifying a new column mapping).
We are looking into a future mode that does not require explicit remapping for disruptive schema changes.
Hey Sidjay,
Thanks for reply.
I will explicitly try this. Though pretty sure I tried it as one of my troubleshooting steps.
I note I've seen that the data flow gen 2 destination process has a step where it presents the in and out schema, which seems to confirm that the schema is ok, and have even seen it indicate new columns that have unchecked checkbox beside them (which I have checked).
But it appears that in fact this apparent confirmation doesn't yet actually behind the scenes update the schema mapping thus your recommendation to delete and recreate the Output Destination section by clicking X.