Currently, Materialized Lake Views (MLV) in Microsoft Fabric offer two actions when a data quality constraint is violated: FAIL (which stops the refresh) and DROP (which silently removes the record). While these are useful for binary "all-or-nothing" or "clean-data-only" scenarios, they lack the flexibility required for robust production data engineering.
The Problem:
FAIL is too disruptive for large datasets where a single minor violation shouldn't halt an entire pipeline.
DROP results in data loss. While the lineage view shows the count of dropped records, there is no native, automated way to inspect or recover the specific rows that were excluded without manual auditing.
Proposed Solution:
I am suggesting a third option: ON MISMATCH QUARANTINE.
When this option is selected:
1. Violating records are excluded from the main Materialized Lake View (similar to DROP).
2. These records are automatically redirected into a system-managed "quarantine" table or a sidecar hidden table associated with the MLV.
3. The quarantine table should include metadata such as the Constraint Name violated and the Timestamp of the refresh.
Why this is needed:
This feature would allow data engineers to maintain "Medallion" architecture integrity while providing a clear path for data remediation. It turns MLVs into a true data quality gate where bad data is captured for review rather than simply deleted or used as a reason to crash the system.
1 Comment
- Nijel
Microsoft Employee
thanks for the feedback dragospopescu. This feature work is in progress. will update the progress once its delivered.
Recent ideas
In TableEX visual when using Custom tooltip, it shows below it the automatic "-> Drillthrough >"
Using in the Power Bi's Service Cloud. In TableEX visual when using Custom tooltip, it shows below it the automatic "-> Drillthrough >" and selects & clicks the drillthrough page. In TableEX vis...LR310 minutes agoRegular VisitorNew2Views0likes0CommentsAllow Power BI subscription sender identity to be configured
The new Power BI subscription delivery platform changed existing subscriptions so that notification emails now show the subscription owner as the sender instead of mailto:[email protected]...bbussey2 hours agoRegular VisitorNew17Views4likes2CommentsSupport Workspaces folder pairing across Fabric Deployment Pipeline stages
Fabric Deployment Pipelines should support deploying items organized in workspace folders when equivalent folder structures exist across stages. Currently, deployments can be blocked when items are ...majofv2 hours agoNew MemberNew3Views0likes0CommentsReturn .png Attachments to Subscriptions
With the latest update to Power BI subscriptions, the separate .png report snapshot attachment has been removed. From Microsoft’s troubleshooting documentation, this appears to be an intentional chan...KDC_Group_PBI3 hours agoNew MemberNew84Views11likes3CommentsConfigure Subscription Image Generation Timestamp in Configured Timezone, not UTC
At the bottom of the image on subscription emails, the timestamp of the image is displayed in UTC time, which can confuse users, especially if timezone is PST and timestamp appears to be in the f...Davis-CMC3 hours agoRegular VisitorNew4Views0likes0Comments