Blog Post

Data Engineering Community Blog
5 MIN READ

Reinventing the Gold Layer Using Materialized Lake Views

pallavi_r's avatar
pallavi_r
Super User
7 months ago
1. Foundations of the Medallion Architecture As we know, modern analytics platforms must balance scale, reliability, and governance to process data from source to target. The Medallion Architecture...
Updated 7 months ago
Version 1.0
data engineering
pallavi_r's avatar
Super User
Joined February 20, 2024
View Profile
Data Engineering Community Blog
Get the latest Fabric Data Engineering tutorials, best practices, and in-depth articles, shared by Microsoft experts and community leaders. If you are interested in becoming a community blog author, send a private message to Natalie_iTalent. An overview of the community blogs can be found here.
AparnaRamakris's avatar
AparnaRamakris
Microsoft Employee
7 months ago

Hi amthog and pallavi_r ,

 

I have a slightly different perspective about the data quality check .All data that do not meet the constraints and are ignored are logged in the backend table .You may run queries on the log table to find out how many records were skipped ,which constraint validation led to skipping of these records as well from sys_dq_metrics

 

SELECT TOP (100) [Namespace],
			[MLVName],
			[MLVID],
			[RefreshPolicy],
			[RefreshDate],
			[RefreshTimestamp],
			[Message],
			[TotalRowsProcessed],
			[TotalRowsDropped],
			[TotalViolations],
			[ViolationsPerConstraint]
FROM [lakehouse].[dbo].[sys_dq_metrics]

According to me ,this is more than enough to backtrack on the bad record and make sufficient corrections on them from business if needed.Please read the blog here Materialized Lake Views -Case Study -Using NYC Tax... - Microsoft Fabric Community to understand more as I have a detailed case study using NYC Taxi data