Copy job is designed to simplify your data ingestion experience without compromise from any source to any destination. It supports multiple data delivery styles, including both batch and incremental copy, providing the flexibility to meet diverse needs.
We are excited to introduce the preview of native Change Data Capture (CDC) support in Copy job that enables efficient and automated replication of changed data including inserted, updated and deleted records from a source to a destination. This ensures your destination data stays up to date without manual effort, improving efficiency in data integration while reducing the load on your source system.
With CDC in Copy job, you can enjoy the following benefits
- Zero manual intervention: automatically captures incremental changes (inserts, updates, deletes) directly from the source.
- Automatic replication: keeps destination data continuously synchronized with source changes.
- Optimized performance: processes only changed data, reducing processing time and minimizing load on the source.
- Smarter incremental copy automatically detects CDC-enabled source tables and allows you to select either CDC-based or watermark-based incremental copy for each table.
How it works - CDC-based vs. Watermark-based Incremental Copy
- CDC-enabled tables: If your source database has CDC enabled, Copy job automatically captures and replicates inserts, updates, and deletes to the destination, applying the exact changes.
- Non-CDC-enabled tables: If CDC is not enabled on your source database, Copy Job detects changes by comparing an incremental column (e.g.,timestamp or ID) against the last run, then merges or appends the changed data to the destination based on your configuration.
Supported connectors:
Currently, Change Data Capture (CDC) in Copy job supports the following data
stores for both source and destination. Please stay tuned and we are adding more.
- Azure SQL DB
- On-premises SQL Server
- Azure SQL Managed Instance
Getting started
1. Ensure that Change Data Capture (CDC) is enabled on your database and tables in the supported source store.
Learn more in the Enable and Disable change data capture documentation.
2. Create a Copy job and connect it to your supported source database with CDC enabled. You’ll have clear visibility into which tables have CDC enabled, and you can then select those tables as your source.
3. After selecting a supported destination store, select 'Merge' (the default option) as the update method.
4. The Copy job will indicate that CDC is being used to capture inserts, updates, and deletions from your source and replicate them to your destination store.
5. After completing the Copy Job creation, the first run will perform an initial full copy. Subsequent runs will replicate only inserts, updates, and deletions from the source.
What’s next
The journey doesn’t stop here!
The Incremental Copy feature in Copy job will be Generally Available on July 1st. Along with this release, a new meter: Data Movement – Incremental Copy will take effect with a consumption rate of 3 CU. The Full/Batch Copy functionality will continue to emit usage on the existing meter: Data Movement with a consumption rate of 1.5 CU. You should start seeing usage related to this new meter in the Microsoft Cost Management experience starting the first week of July and these costs will be reflected in your July Azure invoice. You will also be able to see incremental copy Capacity Unit (CU) usage in the Fabric Capacity Metrics App under the operation name 'Data movement -Incremental copy' and compare it to your invoice.
Resources
Learn more about Copy job in: What is Copy job in Data Factory and Change data capture (CDC) in Copy Job.
Submit your feedback on Fabric Ideas and join the conversation on the Fabric Community. To get into the technical details, head over to the Fabric documentation.