Currently, Materialized Lake Views (MLV) in Microsoft Fabric offer two actions when a data quality constraint is violated: FAIL (which stops the refresh) and DROP (which silently removes the record). While these are useful for binary "all-or-nothing" or "clean-data-only" scenarios, they lack the flexibility required for robust production data engineering.
The Problem:
FAIL is too disruptive for large datasets where a single minor violation shouldn't halt an entire pipeline.
DROP results in data loss. While the lineage view shows the count of dropped records, there is no native, automated way to inspect or recover the specific rows that were excluded without manual auditing.
Proposed Solution:
I am suggesting a third option: ON MISMATCH QUARANTINE.
When this option is selected:
1. Violating records are excluded from the main Materialized Lake View (similar to DROP).
2. These records are automatically redirected into a system-managed "quarantine" table or a sidecar hidden table associated with the MLV.
3. The quarantine table should include metadata such as the Constraint Name violated and the Timestamp of the refresh.
Why this is needed:
This feature would allow data engineers to maintain "Medallion" architecture integrity while providing a clear path for data remediation. It turns MLVs into a true data quality gate where bad data is captured for review rather than simply deleted or used as a reason to crash the system.
1 Comment
- Nijel
Microsoft Employee
thanks for the feedback dragospopescu. This feature work is in progress. will update the progress once its delivered.
Recent ideas
Allow the target pipeline reference in Invoke Pipeline / ExecutePipeline to be set dynamically
Problem Today, the Invoke Pipeline activity (and legacy ExecutePipeline) requires picking the target pipeline statically at design time — a fixed Workspace/Pipeline selection in the UI, which serial...JONATHANHAUN9 hours agoNew MemberNew7Views0likes0CommentsReset Bookmark Navigator Selection to the Default Published View on Browser Refresh
In Power BI Service, when a user selects a bookmark through a Bookmark Navigator, the bookmark identifier is appended to the report URL. If the user refreshes the browser page, the report reloads in ...vivek1234410 hours agoMicrosoft EmployeeNew6Views0likes0CommentsSeparate Interactive and Background Compute Allocation to Prevent Throttling of Live Reports
Problem Statement When Fabric capacity utilization reaches or exceeds 100% in the last one-hour window, end users experience significant disruptions when accessing Power BI Fabric live reports. This...TilakArani10 hours agoNew MemberNew8Views0likes0CommentsAccessibility issue in the Share dialog of embedded Power BI reports
Accessibility issue in the Share dialog of embedded Power BI reports I would like to share an accessibility finding identified while testing an embedded Power BI report published as part of the cont...ugartema11 hours agoAdvocate INew111Views15likes1Comment- fbcideas_migusr13 hours agoNew MemberUnder Review386Views12likes4Comments