I am not looking for a solution because this is something I believe is low level, but I do want to put it on record that it is happening. Perhaps others may have seen it happen. I'll illustrate by way of a sequence of steps:
1. A dataflow (DF1) is created to send data to a lakehouse table (Table1). Things work fine.
2. A minor change was made to DF1, but the number of columns and data types to send to Table1 stay the same. The minor change involves changing a conditional column's if statement. Instead of "if var1 = 5", it becomes "if var1 = 6".
3. After 2 successful refreshes, the data in Table1 does not change. Using SSMS and the SQL Endpoint shows that the data is still from step 1.
4. Table1 is renamed to Table1_OLD.
5. The data destination in DF1 is modified to create a new table named Table1. No other changes were made.
6. After a successful refresh, Table1 is still populated with correct data.
7. Verification shows that Table1 and Table1_OLD contains different data even though structually, they are identical.
I don't know it takes a rename followed by a re-create to get a correctly populated table, but it's scary, especially since the DF sucessfully refreshed.
