In our Microsoft Fabric Lakehouse environment, we are working with a managed Delta table named "Customer_Orders", which includes a primary key column `cust_num` and a timestamp column `RecordDate`. Change Data Feed (CDF) has been enabled using the following configuration:
ALTER TABLE Customer_Orders SET TBLPROPERTIES (delta.enableChangeDataFeed = true)
Our current ingestion process involves extracting Delta Parquet files from SQL Server based on a rolling window defined by `MAX(RecordDate) - 3 days`. This approach occasionally results in duplicate rows being reintroduced into the Lakehouse table. To handle this, we perform a `MERGE` operation using `cust_num` and `RecordDate` as composite keys, with the following logic:
- WHEN MATCHED THEN SKIP** (no updates)
- WHEN NOT MATCHED THEN INSERT** (new row)
However, the customer expects Fabric Lakehouse to automatically detect changes across all columns—without requiring explicit key definitions in the `MERGE` clause—and to perform updates or inserts accordingly. This expectation appears to be misaligned with Fabric’s native capabilities.
We are seeking guidance from Microsoft to help clarify the out-of-the-box behavior of Lakehouse-managed Delta tables, particularly around change detection and merge semantics, so we can realign the customer’s understanding and ensure the solution is both technically sound and aligned with platform best practices.
