Data Pipeline copy activity unexpected behavior

Hi schneiw ,

Thank you for reaching out to Microsoft Community.

Based on your use case and troubleshooting efforts, it seems that you're attempting to update only two fields (deleted_date and is_deleted) in a Lakehouse Delta table using the Copy Activity with the Upsert option in a Data Pipeline. Although the pipeline runs successfully and the data preview confirms that rows are flowing through, the changes are not reflected in the destination table. This could be due to several common and intermittent issues.

First, ensure that the destination table is a Delta Lake table, as Delta is required for Upsert operations to function correctly. If the table is not Delta-enabled, the update may silently fail without throwing an error. Also, verify that the table is not read-only or locked, and that the target columns (deleted_date, is_deleted) are nullable or have default values—otherwise, writes may be blocked or silently dropped due to schema constraints.

Next, confirm that you have correctly configured the Key Columns in the Upsert settings and that your data mapping aligns with the schema. Although you've tried both with and without column mapping, inconsistencies in column names or data types can still cause issues. Also, check the Copy Activity run output JSON, particularly for "rowsUpdated" or "rowsUpserted" counts. If those values are zero, it may indicate that the key values do not match any existing records, or that the data failed schema validation.
If you're working with Dataverse or connected environments, consider turning on logging (e.g., Dataverse Plugin Trace Logs) or enabling auditing for the target table and fields. This can help you trace incoming data and identify whether the upsert operation is hitting the backend as expected.

Finally, ensure that the Copy activity is not silently failing due to data mismatches. By default, it will fail or skip incompatible rows. You can configure it to skip and log incompatible rows to help isolate issues without stopping the pipeline.

Hope this helps.
Warm regards,
Chaithra E