Currently, Materialized Lake Views (MLV) in Microsoft Fabric offer two actions when a data quality constraint is violated: FAIL (which stops the refresh) and DROP (which silently removes the record). While these are useful for binary "all-or-nothing" or "clean-data-only" scenarios, they lack the flexibility required for robust production data engineering.
The Problem:
FAIL is too disruptive for large datasets where a single minor violation shouldn't halt an entire pipeline.
DROP results in data loss. While the lineage view shows the count of dropped records, there is no native, automated way to inspect or recover the specific rows that were excluded without manual auditing.
Proposed Solution:
I am suggesting a third option: ON MISMATCH QUARANTINE.
When this option is selected:
1. Violating records are excluded from the main Materialized Lake View (similar to DROP).
2. These records are automatically redirected into a system-managed "quarantine" table or a sidecar hidden table associated with the MLV.
3. The quarantine table should include metadata such as the Constraint Name violated and the Timestamp of the refresh.
Why this is needed:
This feature would allow data engineers to maintain "Medallion" architecture integrity while providing a clear path for data remediation. It turns MLVs into a true data quality gate where bad data is captured for review rather than simply deleted or used as a reason to crash the system.
1 Comment
- Nijel
Microsoft Employee
thanks for the feedback dragospopescu. This feature work is in progress. will update the progress once its delivered.
Recent ideas
Paste values in slicers for hierarchy data
Hierarchical slicers are widely used in enterprise reports (for example Product RCN > SKU Key, Region > Country > City, or GPP > Division > RCN > SKU). Users often need to bulk-filter by pasting valu...eddieisking20001 hour agoFrequent VisitorNew4Views0likes0CommentsPower Query Parameter create/use when connecting to a dataflow
In power query when you connect to a table in a dataflow there are no options to create or use a parameter for that dataflow. When connecting to SQL there is an option when setting up the connection ...ben_holmes3 hours agoNew MemberNeed Clarification203Views3likes2CommentsMake Copilot Approval status for semantic model available through API
In recent versions fabric allowed to mark a semantic model as approved for copilot to reduce AI adoption friction. However there is no specific way for external governance application to analyse whi...FrançoisD4 hours agoFrequent VisitorNew12Views2likes0CommentsClone a workspace, including its data, to a new workspace
Description: Today there's no way to fully clone a Fabric workspace: git sync and Deployment Pipelines move item definitions only, not the underlying data; Warehouse's zero-copy CREATE TABLE ... AS ...gclements5 hours agoHelper IINew14Views1like0CommentsAllow NotebookUtils getSecret() to authenticate with Workspace Identity
Current behavior In Microsoft Fabric, notebookutils.credentials.getSecret() authenticates against Azure Key Vault using the identity of the user who executes the notebook. This behavior appli...tmihara10 hours agoNew MemberNew8Views0likes0Comments