Forum Discussion

D4zk's avatar
D4zk
Frequent Visitor
4 months ago
Solved

Microsoft Fabric Copy Job – SCD Type 2 (SCD2) Issues

  During source data changes, the Copy Job does not manage correctly the SCD2 control colums: copyjob_valid_from, copyjob_valid_to, copyjob_is_current. As shown in the results, the first two rows c...
data engineering
  • v-saisrao-msft's avatar
    v-saisrao-msft
    3 months ago

    Hi D4zk,

    Since MERGE (SCD Type 1) is functioning while SCD Type 2 is not, this highlights the documented difference: SCD2 uses CDC-based historical tracking, including expiring rows and soft deletes, whereas MERGE updates only current data. SCD2 depends on CDC to handle inserts, updates, and deletes during incremental runs, so its behavior varies based on how those changes are applied. As your configuration has been validated, you can keep using MERGE for stability or consider submitting a support request for SCD2-specific issues.

    Change data capture (CDC) in Copy Job - Microsoft Fabric | Microsoft Learn

v-saisrao-msft's avatar
v-saisrao-msft
Community Support
4 months ago

Hi D4zk,

SCD Type 2 in Copy Job is currently available as a preview feature and depends on supported configurations. Make sure that Edit mapping is not enabled, because SCD Type 2 does not work when the destination schema is already created. Also, check that Change Data Capture is set up correctly, since inserts, updates, and deletes are handled through CDC during incremental runs. Because Copy Job uses these changes for incremental replication, any configuration issues can affect how updates and deletes are processed.

Change data capture (CDC) in Copy Job - Microsoft Fabric | Microsoft Learn

Connectors for Copy Job - Microsoft Fabric | Microsoft Learn

Slowly changing dimension type 2 - Microsoft Fabric | Microsoft Learn

 

Thank you.