Currently, Materialized Lake Views (MLV) in Microsoft Fabric offer two actions when a data quality constraint is violated: FAIL (which stops the refresh) and DROP (which silently removes the record). While these are useful for binary "all-or-nothing" or "clean-data-only" scenarios, they lack the flexibility required for robust production data engineering.
The Problem:
FAIL is too disruptive for large datasets where a single minor violation shouldn't halt an entire pipeline.
DROP results in data loss. While the lineage view shows the count of dropped records, there is no native, automated way to inspect or recover the specific rows that were excluded without manual auditing.
Proposed Solution:
I am suggesting a third option: ON MISMATCH QUARANTINE.
When this option is selected:
1. Violating records are excluded from the main Materialized Lake View (similar to DROP).
2. These records are automatically redirected into a system-managed "quarantine" table or a sidecar hidden table associated with the MLV.
3. The quarantine table should include metadata such as the Constraint Name violated and the Timestamp of the refresh.
Why this is needed:
This feature would allow data engineers to maintain "Medallion" architecture integrity while providing a clear path for data remediation. It turns MLVs into a true data quality gate where bad data is captured for review rather than simply deleted or used as a reason to crash the system.
1 Comment
- Nijel
Microsoft Employee
thanks for the feedback dragospopescu. This feature work is in progress. will update the progress once its delivered.
Recent ideas
Allow us to rename fabric data agents published to m365
If we use deployment pipelines to promote fabric data agents between dev, test/UAT, and prod fabric workspaces, we need to keep the name of the fabric agent the same in each workspace. If we want to ...PeterDaniels26 minutes agoAdvocate IIINew176Views0likes2CommentsMake workspace and item session persistence optional
Description The new persistent session behavior in Microsoft Fabric should be optional rather than forced. Currently, Fabric remembers the workspaces and items that were open in my previous session...TeemuMultanen1 hour agoAdvocate INew292Views47likes2CommentsSupport Encrypted Sensitivity-Labeled Excel Files in Power Query
Description Currently, Power Query Online and Power Query in Excel are unable to access encrypted Excel files. Excel files with sensitivity types other than Public or Non-Business can be encrypted a...ewarstdhyjugkhi2 hours agoMicrosoft EmployeeNew16Views6likes0CommentsAuto-propagate Lakehouse (Direct Lake) permissions to users assigned to a Power BI App audience
Problem When a Power BI report uses a Live Connection to a Direct Lake Semantic Model, end users must have explicit ReadAll / ReadData permissions on the underlying Lakehouse (or its SQL analytics e...majofv4 hours agoNew MemberNew5Views1like0Comments