Fabric Metadata & Data Quality Assessment Framework

Overview

A reusable Microsoft Fabric notebook for metadata discovery, single-pass PySpark data profiling, rule-based data quality assessment, and Data Quality Index (DQI) scoring.

Key Capabilities

Dual Execution Modes: Run on synthetic e-commerce data or target existing Delta tables in OneLake.

High-Performance Profiling: Single-pass PySpark aggregations compute distinct counts, null ratios, and boundaries in a single Spark action call.

Multi-Dimension Quality Auditing: Evaluates Completeness, Uniqueness, and Validity using custom weighted thresholds.

Executive Reporting: Generates an aggregate Data Quality Index (DQI) score and automated remediation guidance.

Use Cases

• Lakehouse data quality monitoring and health checks
• Data product readiness for Gold layer consumption
• Pre- and post-migration validation
• Enterprise data platform modernization

Full Notebook & Code: Download the complete .ipynb notebook file from the Github Repository