Forum Discussion
Dataflow Gen2 - Table not getting dropped and recreated
I'm not quite sure I understand.
"These are actually two different operations that the document aims to refresh."
"The documentation mentions that tables are deleted and recreated when the data flow is refreshed, which you can actually think of as an overwriting operation. The new table overwrites the original table."
Per my understanding, delete (drop) and recreate a table is a different concept than overwrite a table, especially in Delta lake world where old versions are kept when overwriting a table, but old versions are deleted when dropping a table.
So I would like to get clarification about this part of the documentation:
"Drop and recreate table: To allow for these schema changes, on every dataflow refresh the table is dropped and recreated."
Does the table actually get dropped and recreated?
Or it just gets replaced?
Ref.
https://docs.databricks.com/en/delta/drop-table.html#when-to-replace-a-table
If the table doesn't really get dropped, then I think this part of the documentation should be changed because the documentation says that the table gets dropped.
However, my understanding about what it means that a table gets dropped, may be wrong.
But so far, my understanding when you drop a table is that is deletes the table including all the version history. So it seems to me the documentation here is incorrect, because the version history actually seems to be retained after refreshing the dataflow.
Hi frithjof_v
We may need more time to discuss the problem.
Thank you for your discoveries and questions!
If you still have questions about the official documentation, you can provide product feedback in the Feedback section below the documentation.
Regards,
Nono Chen