Introducing Fabric Data Quality Studio

A Unified Data Quality Management Experience for Microsoft Fabric

Discover. Profile. Validate. Monitor. Improve.

Modern data platforms have made it easier than ever to ingest, transform, and analyze massive volumes of data. With Microsoft Fabric, organizations can bring together data engineering, analytics, and business intelligence within a unified ecosystem.

But one fundamental challenge remains:

How do we establish trust in the data flowing through these platforms?

A pipeline may complete successfully. A Lakehouse table may contain millions of records. A semantic model may refresh without errors.

Yet none of these guarantee that the underlying data is complete, accurate, consistent, unique, or fit for business use.

A technically successful data pipeline is not necessarily delivering trustworthy data.

This challenge becomes more significant as organizations scale across multiple workspaces, Lakehouses, domains, and data engineering teams.

To explore a more structured approach to this problem, I've been building Fabric Data Quality Studio, a custom data quality management solution designed around the Microsoft Fabric ecosystem.

The objective is to bring data discovery, metadata visibility, profiling, quality-rule management, validation, duplicate detection, and quality monitoring into a cohesive experience.

Rather than treating data quality as an isolated engineering activity, the vision is to make it an integral part of the data platform lifecycle.

1. Data Asset Discovery and Registration

Establishing visibility before assessing quality.

Data quality management begins with understanding the assets that exist across the platform.

In enterprise environments, datasets are distributed across multiple Fabric workspaces, Lakehouses, schemas, and tables. Maintaining visibility into these assets can become challenging, particularly as platforms evolve.

Fabric Data Quality Studio introduces a centralized asset-registration experience.

The current interface supports:

• Discovering available Fabric workspaces and selecting Lakehouses for registration.
• Maintaining a centralized inventory of registered Lakehouses.
• Discovering schemas and tables from registered assets.
• Tracking metadata discovery and synchronization status.
• Displaying discovery failures and registration information.
• Searching and filtering assets across workspaces.

The purpose is to create a foundation where data quality operations can be associated with a known and discoverable data inventory.

This also provides a starting point for managing quality coverage as the number of data assets grows.

Key principle: You cannot systematically manage data quality without knowing what data exists.

2. Data Explorer — Understanding the Data Landscape

Once data assets are registered, the next step is understanding their structure and characteristics.

The Data Explorer provides a centralized interface for navigating registered Fabric Lakehouses.

Its workspace-oriented navigation allows users to explore the hierarchy:

Workspace → Lakehouse → Schema → Table

The experience is designed to consolidate information that data engineers and quality practitioners frequently need when investigating datasets.

This includes table discovery, metadata inspection, column information, data previews, and access to profiling and quality information as those capabilities become available.

A searchable schema and table navigator makes it easier to locate relevant assets without repeatedly switching between different platform experiences.

The broader goal is to establish a single place to understand both the structure of a dataset and its quality characteristics.

3. Data Profiling — Understanding What the Data Actually Contains

One of the most important stages of data quality assessment is profiling.

Before defining validation rules, engineers need to understand the existing characteristics of the data.

For example:

• What percentage of records contains null values?
• How many distinct values are present in a column?
• Are there unexpected patterns or outliers?
• Do actual values align with the intended data type or business meaning?
• Are duplicate values appearing where uniqueness is expected?
• How do value distributions change between datasets?

The profiling module is intended to provide this initial evidence and support data-driven validation decisions.

Instead of defining expectations entirely from assumptions, quality rules can be informed by observed data characteristics.

This capability is part of the product's evolving scope and is not presented here as a fully operational module.

4. Rule Library — Reusable and Standardized Data Quality Rules

Quality rules should be reusable assets, not scattered SQL statements.

In many enterprise data platforms, validation logic gradually becomes distributed across SQL scripts, Spark notebooks, pipelines, and application code.

This introduces several challenges:

• Similar validations are implemented differently by different teams.
• Changes to business rules require modifications across multiple solutions.
• Rule ownership and definitions become difficult to track.
• Reuse and version management are limited.
• Maintaining consistent quality standards becomes harder.

Fabric Data Quality Studio introduces a centralized Rule Library to address these challenges.

The current rule catalogue includes definitions such as:

Completeness

• Column count checks
• Conditional requirements

Validity

• Allowed-value validation
• Aggregate range validation
• Custom SQL assertions

Uniqueness

• Composite uniqueness checks

Consistency

• Cross-column comparisons

Timeliness

• Date recency validation

Rules are organized by quality dimension and evaluation scope, including column-level, table-level, cross-column, and composite validations.

The library also distinguishes built-in definitions from versioned custom definitions.

An important architectural consideration is that a rule definition and its execution capability are separate concerns. A rule being available in the catalogue does not automatically imply that it has been validated against every SQL or Spark execution path.

The long-term direction is to establish a reusable, governed collection of validation definitions that engineering teams can apply consistently across datasets.

5. Test Cases — Applying Rules to Business Data

A quality rule defines what should be validated.

A test case defines where and how that rule should be applied.

This separation is central to the design.

For example, consider a reusable rule for checking whether a column contains values from an approved list.

The same rule definition might be applied differently to:

• Customer status in a customer master dataset.
• Order status in a sales transaction dataset.
• Currency codes in a financial dataset.
• Product categories in a product master dataset.

The underlying validation concept remains the same, while the target dataset and expected values differ.

The Test Cases module is designed to manage these dataset-specific configurations.

It provides a foundation for associating reusable rule definitions with authorized data assets and maintaining saved configuration revisions.

The intended outcome is a clear distinction between validation logic, dataset-specific expectations, and execution results.

That separation becomes increasingly important when building scalable, repeatable data quality processes.

6. Test Suites — Organizing Quality Checks at Scale

Individual test cases are useful, but enterprise quality assessments frequently require groups of validations to be managed together.

Consider a customer dataset requiring checks for:

• Customer identifier completeness.
• Valid country and region codes.
• Duplicate customer identifiers.
• Date consistency.
• Required contact information.
• Valid status transitions.

Managing these checks independently can become difficult as the number of datasets grows.

Test Suites are intended to organize related test cases into logical validation groups.

The design direction is to support quality assessments at meaningful boundaries, such as individual tables, data domains, or defined processing stages.

This introduces a higher-level organizational model for data quality, allowing engineering teams to think in terms of validation coverage rather than isolated rules.

7. Duplicate Detection — Beyond Conventional Uniqueness Checks

Duplicate data is one of the most persistent quality challenges in enterprise systems.

A simple uniqueness constraint may detect identical identifiers, but real-world duplicate detection often requires more sophisticated matching strategies.

Customer, supplier, product, and transactional datasets can contain records representing the same logical entity with differences in formatting, spelling, or identifying attributes.

Fabric Data Quality Studio includes a dedicated Duplicate Detection module with a configurable matching-rule experience.

Its design brings together:

• Matching-rule selection and versioning.
• Target workspace, Lakehouse, schema, and table context.
• Detection execution.
• Records analyzed.
• Duplicate groups identified.
• Records belonging to duplicate groups.
• Excess duplicate counts.

This introduces a dedicated interface for managing and reviewing potential duplicate records rather than relying exclusively on isolated queries.

The matching and execution workflow is still being validated, and the current screenshots represent the interface rather than a completed detection result.

The broader objective is to evolve duplicate detection into a reusable capability within the data quality framework.

8. Data Quality Overview — Establishing Visibility and Coverage

Data quality cannot be effectively managed if results are fragmented across different engineering components.

The Data Quality Overview is intended to provide a consolidated view of data asset coverage and assessment activity.

The interface brings together indicators for:

Asset coverage

• Registered Lakehouses
• Discovered tables
• Profiled tables

Quality assessment

• Quality scores
• Failing tests
• Duplicate groups

Operational visibility

• Quality-related execution activity
• Data quality health
• Test case outcomes

An important distinction in the design is that metadata coverage, profiling coverage, validation outcomes, and overall quality scores represent different measurements.

For example, having complete metadata discovery does not establish that a dataset has passed its quality checks.

Likewise, a successfully completed execution does not necessarily mean every individual test passed.

The dashboard is being designed to preserve these distinctions rather than presenting misleading aggregate indicators.

The current implementation displays available catalogue information and explicitly identifies metrics for which sufficient evidence is not yet available.

9. Quality Execution, Scheduling, and Monitoring

The longer-term product direction extends beyond defining and organizing quality checks.

For data quality to become part of everyday data operations, validations must be executable, repeatable, observable, and actionable.

The planned operational capabilities include:

Execution management: Running configured validations against selected datasets and capturing execution context.

Scheduling: Enabling recurring quality checks as part of operational processes.

Result tracking: Preserving individual test outcomes separately from execution success or failure.

Quality scoring: Building interpretable quality measurements from sufficiently complete and comparable evidence.

Findings management: Providing a place to investigate detected issues and understand affected datasets.

These modules form part of the intended end-to-end experience, with additional implementation and validation work required.

The goal is to move toward a continuous data quality operating model rather than relying entirely on manual checks.

10. The Design Philosophy

Several design principles are shaping the solution.

Metadata-driven rather than dataset-specific

The framework should accommodate new data assets and reusable quality definitions without requiring a separate application implementation for every table.

Separation of concerns

Data assets, rule definitions, test configurations, executions, and outcomes should remain distinct concepts.

This supports maintainability, traceability, and future extensibility.

Native-context user experience

The interface is designed to sit within the Microsoft Fabric user experience, helping reduce the need to move between disconnected tools for common data quality activities.

Visibility into unavailable evidence

A missing quality result should not be interpreted as a passing result.

The product explicitly distinguishes available evidence from metrics that cannot yet be calculated reliably.

Practical enterprise usability

The experience is intended for data engineers and data platform practitioners who need clear navigation, reusable configurations, and meaningful operational information.

The goal is not simply to expose more technical metadata. It is to turn that information into something users can act on.

11. The Bigger Picture — Data Quality as a Platform Capability

The long-term vision is to make data quality a reusable capability of the data platform.

Imagine a Fabric environment where a new Lakehouse is registered, its structure is discovered, profiling provides evidence about its contents, reusable rules are applied through configured tests, and results become visible through a consistent monitoring experience.

As new data domains and Lakehouses are introduced, they could follow the same quality management approach.

Rather than every data engineering team implementing its own collection of validation scripts, the platform could provide shared definitions, consistent execution practices, and comparable quality evidence.

This is particularly relevant for organizations operating large Lakehouse environments with multiple data sources, engineering teams, and business domains.

The ultimate objective is not to eliminate every data quality issue. It is to make data quality visible, measurable, repeatable, and continuously improvable.

12. What's Next?

Fabric Data Quality Studio is an evolving solution, and the current product experience represents an important step toward that vision.

The next areas of focus include completing the profiling and validation workflows, strengthening execution capabilities, expanding duplicate-detection functionality, and establishing more comprehensive quality monitoring and scoring.

There is also an opportunity to explore how AI-assisted capabilities could support profiling interpretation, rule recommendations, and investigation of recurring quality issues in future iterations.

The foundation is a simple belief:

Data engineering delivers data. Data quality establishes the confidence required to use it.

And as platforms become increasingly automated and AI-driven, that confidence becomes even more important.

Sharing an early look at Fabric Data Quality Studio and the direction I'm exploring.

I'd be interested to hear from other Microsoft Fabric practitioners:

How are you currently managing data quality across your Fabric Lakehouses? Are you relying on custom SQL/Spark validations, centralized frameworks, or dedicated quality platforms?

#MicrosoftFabric #DataQuality #DataEngineering #DataGovernance #DataObservability #DataArchitecture #DataManagement #DataProfiling #DataValidation #DataDiscovery #DataQualityManagement #DataQualityFramework #Lakehouse #OneLake #MicrosoftAzure #DataPlatform #DataTrust #DataIntegrity #DataDeduplication #DataAnalytics #DataReliability #EnterpriseData #FabricCommunity