Currently, Materialized Lake Views (MLV) in Microsoft Fabric offer two actions when a data quality constraint is violated: FAIL (which stops the refresh) and DROP (which silently removes the record). While these are useful for binary "all-or-nothing" or "clean-data-only" scenarios, they lack the flexibility required for robust production data engineering.
The Problem:
FAIL is too disruptive for large datasets where a single minor violation shouldn't halt an entire pipeline.
DROP results in data loss. While the lineage view shows the count of dropped records, there is no native, automated way to inspect or recover the specific rows that were excluded without manual auditing.
Proposed Solution:
I am suggesting a third option: ON MISMATCH QUARANTINE.
When this option is selected:
1. Violating records are excluded from the main Materialized Lake View (similar to DROP).
2. These records are automatically redirected into a system-managed "quarantine" table or a sidecar hidden table associated with the MLV.
3. The quarantine table should include metadata such as the Constraint Name violated and the Timestamp of the refresh.
Why this is needed:
This feature would allow data engineers to maintain "Medallion" architecture integrity while providing a clear path for data remediation. It turns MLVs into a true data quality gate where bad data is captured for review rather than simply deleted or used as a reason to crash the system.
1 Comment
- Nijel
Microsoft Employee
thanks for the feedback dragospopescu. This feature work is in progress. will update the progress once its delivered.
Recent ideas
iOS Shortcuts Support for Direct Access to Power BI Reports
We are looking to deploy several Power BI reports to support emergency response activities. The intended users are primarily non-technical staff who do not use Power BI currently, but who would benef...Default81292 hours agoFrequent VisitorNew3Views0likes0CommentsSupport Fabric Workspace Identity for custom APIs
Currently, Fabric Workspace Identity cannot be used to authenticate against custom APIs protected by Microsoft Entra ID. Azure Managed Identities support requesting an access token for a specific AP...frithjof_v6 hours agoCommunity ChampionNew62Views5likes0CommentsAllow the target pipeline reference in Invoke Pipeline / ExecutePipeline to be set dynamically
Problem Today, the Invoke Pipeline activity (and legacy ExecutePipeline) requires picking the target pipeline statically at design time — a fixed Workspace/Pipeline selection in the UI, which serial...JONATHANHAUN20 hours agoNew MemberNew8Views0likes0CommentsReset Bookmark Navigator Selection to the Default Published View on Browser Refresh
In Power BI Service, when a user selects a bookmark through a Bookmark Navigator, the bookmark identifier is appended to the report URL. If the user refreshes the browser page, the report reloads in ...vivek1234421 hours agoMicrosoft EmployeeNew6Views0likes0CommentsSeparate Interactive and Background Compute Allocation to Prevent Throttling of Live Reports
Problem Statement When Fabric capacity utilization reaches or exceeds 100% in the last one-hour window, end users experience significant disruptions when accessing Power BI Fabric live reports. This...TilakArani21 hours agoNew MemberNew8Views0likes0Comments