Dataflow Gen 2 Issue- No records to push to target, but target gets update with records

[Tarunire]

Hi all,

I am try the Dataflow Gen 2 to implement SCD2 logic with the target set to a warehouse. During the first run all the records are being pushed to target as expected.
During the second run the same set of records are again being pushed created a duplicate entry for the record even when there are no changes.
When manually viewing the target records in the dataflow, no records are being shown as expected. But upon publish/run the same set of records get pushed again to the target.
I am testing it out on sample data, I am attaching below some screenshots for the same.

Please anyone had a similar experience, it would be great if you could let me know how you resolved it.

Thank you

[Tarunire]

Hi @v-zhouwen-msft ,

Thanks for sharing the link.

But I am not facing issues with application of the logic in dataflow gen2, using a similar approach I was able to do it previously for one other table. I am facing challenge with this particular instance, the query that is present for populating the target table does not have any records when I do a debug and manually check the records that it fetchs. On running the dataflow, the target gets some records which shouldn't have populated as they are not present in the query intented for the target when I check it manually.

Regards
Tarun

[v-zhouwen-msft]

Hi @Tarunire ,

It looks like you are using Dataflow Gen 2 to implement SCD2 logic, see if the following article helps with your problem.Start with ‘The Dataflows Gen2 Approach’.

Implementing SCD Type 2 with Microsoft Fabric — PySpark / DataflowGen2 | by Nermeen Abdel-Aziz | Med...

Best Regards,
Wenbin Zhou
If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.