Currently, Materialized Lake Views (MLV) in Microsoft Fabric offer two actions when a data quality constraint is violated: FAIL (which stops the refresh) and DROP (which silently removes the record). While these are useful for binary "all-or-nothing" or "clean-data-only" scenarios, they lack the flexibility required for robust production data engineering.
The Problem:
FAIL is too disruptive for large datasets where a single minor violation shouldn't halt an entire pipeline.
DROP results in data loss. While the lineage view shows the count of dropped records, there is no native, automated way to inspect or recover the specific rows that were excluded without manual auditing.
Proposed Solution:
I am suggesting a third option: ON MISMATCH QUARANTINE.
When this option is selected:
1. Violating records are excluded from the main Materialized Lake View (similar to DROP).
2. These records are automatically redirected into a system-managed "quarantine" table or a sidecar hidden table associated with the MLV.
3. The quarantine table should include metadata such as the Constraint Name violated and the Timestamp of the refresh.
Why this is needed:
This feature would allow data engineers to maintain "Medallion" architecture integrity while providing a clear path for data remediation. It turns MLVs into a true data quality gate where bad data is captured for review rather than simply deleted or used as a reason to crash the system.
1 Comment
- Nijel
Microsoft Employee
thanks for the feedback dragospopescu. This feature work is in progress. will update the progress once its delivered.
Recent ideas
Zoom Based Visibility for Location Labels in the Azure Maps Visual
Location labels can be helpful for maps but also create a lot of clutter when zoomed out. It would be great if zoom based visibility controls could be enabled for this feature.AllianceDA51 minutes agoAdvocate IINew6Views0likes0CommentsNeed an option to save PBIP files without data (the .abf files)
We need an option to save the PBIP without saving the ABF file, just like the PBIT file is saved without data. As it is now, we will violate company security policies that don't allow storing data ...heve4 hours agoNew MemberNew611Views39likes6CommentsiOS Shortcuts Support for Direct Access to Power BI Reports
We are looking to deploy several Power BI reports to support emergency response activities. The intended users are primarily non-technical staff who do not use Power BI currently, but who would benef...Default81297 hours agoFrequent VisitorNew3Views0likes0CommentsSupport Fabric Workspace Identity for custom APIs
Currently, Fabric Workspace Identity cannot be used to authenticate against custom APIs protected by Microsoft Entra ID. Azure Managed Identities support requesting an access token for a specific AP...frithjof_v10 hours agoCommunity ChampionNew64Views5likes0CommentsAllow the target pipeline reference in Invoke Pipeline / ExecutePipeline to be set dynamically
Problem Today, the Invoke Pipeline activity (and legacy ExecutePipeline) requires picking the target pipeline statically at design time — a fixed Workspace/Pipeline selection in the UI, which serial...JONATHANHAUN1 day agoNew MemberNew9Views0likes0Comments