DataTrust Fabric
DataTrust is a Fabric extension designed to ensure trusted end-to-end data movement across lakehouses by automatically validating ingestion and transformation pipelines.
1. The Challenge: The Integrity Blind Spot
Current Microsoft Fabric workloads involve continuous data movement between lakehouses via complex pipelines. However, engineers currently face significant hurdles:
- Lack of Automation: Reliance on manual notebooks and ETL scripts without end-to-end integrity attestation.
- The Verification Gap: No native mechanism exists to verify if data was lost or unintentionally altered during execution.
Trust Issues: Reduced confidence in downstream datasets and heavy manual validation overhead.
Integrity vs. Quality: Unlike standard Data Quality checks (completeness, plausibility), this solution focuses on Data Integrity—ensuring the source and target lakehouses remain synchronized during execution.
2. The Solution: Automated Integrity Guardrails
The Data Trust Extension introduces an automated validation workload directly within the Microsoft Fabric environment.
Flexible Integration: Validation can be triggered as a native pipeline step, via a "Data Integrity" option in the Fabric UI, or by supplying a Spark job plan/notebook.
Core Capabilities:
- Automated Comparison: Compares source and target lakehouse tables automatically.
- Record Match Metrics: Instant verification of row counts and synchronization.
Field-Level Validation: Deep integrity checks at individual field level.
Discrepancy Insights: Actionable insights instead of simple pass/fail results to accelerate root-cause analysis.
3. Impact and Outcomes
By transitioning from assumed trust to proven trust, the extension delivers:
- Zero-Trust Movement: Continuous validation for every lakehouse-to-lakehouse transfer.
Auditable Pipelines: Automated integrity scorecards for every execution.
Efficiency: Faster troubleshooting and reduced manual validation overhead for data engineering teams.
4. Design
5. Demo Video
The following videos demonstrate the UI integration, source/target table selection, and execution of an integrity validation job within Microsoft Fabric.
The first part provides context. For a quick technical walkthrough, you may skip directly to the demo section.
Watch the Context:
Watch the Demo:
6. Fabric Portal Extension
DataTrust Fabric transforms ingestion from a blind transfer into a verified, deterministic, and reliable process across lakehouses.
