dragospopescu's avatar
dragospopescu
Icon for Advocate I rankAdvocate I
6 months ago
Status:
New

New MLV Constraint Type: Soft-Reject With Logging

Currently, Materialized Lake Views (MLV) in Microsoft Fabric offer two actions when a data quality constraint is violated: FAIL (which stops the refresh) and DROP (which silently removes the record). While these are useful for binary "all-or-nothing" or "clean-data-only" scenarios, they lack the flexibility required for robust production data engineering.

 

The Problem:

  • FAIL is too disruptive for large datasets where a single minor violation shouldn't halt an entire pipeline.

  • DROP results in data loss. While the lineage view shows the count of dropped records, there is no native, automated way to inspect or recover the specific rows that were excluded without manual auditing.

Proposed Solution:
I am suggesting a third option: ON MISMATCH QUARANTINE.

When this option is selected:
1. Violating records are excluded from the main Materialized Lake View (similar to DROP).

2. These records are automatically redirected into a system-managed "quarantine" table or a sidecar hidden table associated with the MLV.

3. The quarantine table should include metadata such as the Constraint Name violated and the Timestamp of the refresh.


Why this is needed:

This feature would allow data engineers to maintain "Medallion" architecture integrity while providing a clear path for data remediation. It turns MLVs into a true data quality gate where bad data is captured for review rather than simply deleted or used as a reason to crash the system.

1 Comment

  • Nijel's avatar
    Nijel
    Icon for Microsoft Employee rankMicrosoft Employee
    thanks for the feedback dragospopescu. This feature work is in progress. will update the progress once its delivered.

Recent ideas