topic Data destination does not populate correctly in Dataflow

Data destination does not populate correctly

ebjim — Fri, 02 Feb 2024 20:14:41 GMT

I am not looking for a solution because this is something I believe is low level, but I do want to put it on record that it is happening. Perhaps others may have seen it happen. I'll illustrate by way of a sequence of steps:

1. A dataflow (DF1) is created to send data to a lakehouse table (Table1). Things work fine.

2. A minor change was made to DF1, but the number of columns and data types to send to Table1 stay the same. The minor change involves changing a conditional column's if statement. Instead of "if var1 = 5", it becomes "if var1 = 6".

3. After 2 successful refreshes, the data in Table1 does not change. Using SSMS and the SQL Endpoint shows that the data is still from step 1.

4. Table1 is renamed to Table1_OLD.

5. The data destination in DF1 is modified to create a new table named Table1. No other changes were made.

6. After a successful refresh, Table1 is still populated with correct data.

7. Verification shows that Table1 and Table1_OLD contains different data even though structually, they are identical.

I don't know it takes a rename followed by a re-create to get a correctly populated table, but it's scary, especially since the DF sucessfully refreshed.

Re: Data destination does not populate correctly

Anonymous — Sat, 03 Feb 2024 20:49:58 GMT

Hi ebjim
Thanks for using Fabric Community.
At this time, we are reaching out to the internal team to get some help on this.
We will update you once we hear back from them.
Thanks

Re: Data destination does not populate correctly

Anonymous — Mon, 05 Feb 2024 04:48:46 GMT

Hi ebjim
Thanks for your valuable feedback.

Appreciate if you could share the feedback on our feedback channel. Which would be open for the user community to upvote & comment on. This allows our product teams to effectively prioritize your request against our existing feature backlog and gives insight into the potential impact of implementing the suggested feature.

Please provide the link for the idea here so that others can upvote. Thanks.

Hope this helps. Please let me know if you have any further queries.