How should Microsoft Fabric data pipelines handle AI-driven validation ?

Hi codeautomation​,

I would separate deterministic data-quality rules from AI-assisted validation rather than treating AI as the primary gate.

For things such as:

• schema changes
• null checks
• duplicate detection
• allowed values
• referential integrity
• numeric/date ranges

I would keep the validation deterministic.

Then I would use AI for cases where the rule is harder to express precisely, for example:

• classifying unusual records
• detecting semantic inconsistencies
• validating free-text fields
• explaining anomalies
• extracting information from unstructured data

Fabric now has AI Functions that can be used from notebooks, Dataflow Gen2 and SQL for operations such as classification, extraction and similarity.

Architecturally, I would normally use:

Pipeline
-> Bronze/raw data
-> Notebook or validation step
-> deterministic checks
-> optional AI validation
-> write validation results
-> If Condition
-> Silver/curated data or quarantine

That also fits Microsoft’s medallion architecture guidance, where raw data is preserved in Bronze and validation/cleansing happens before data progresses into Silver.

For orchestration, a notebook can return a validation result to the pipeline, and the pipeline can branch using an If Condition. Fabric supports notebook exit values specifically for this kind of conditional orchestration.

I would also log the validation outcome rather than only returning pass/fail. For example:

• run ID
• rule name
• severity
• failed row count
• sample failures
• AI confidence where applicable
• timestamp

Then use severity levels such as:

Critical
-> fail pipeline / quarantine

Warning
-> continue but log

Informational
-> continue

For scale, I would avoid sending every row through an LLM. Run deterministic Spark checks across the full dataset first, then send only suspicious or ambiguous records to the AI layer.

That keeps the process cheaper, faster and easier to audit.

So my preference would be:

• Fabric Pipeline for orchestration
• Fabric Notebook/Spark for large-scale validation
• AI Functions only where semantic reasoning actually adds value
• Lakehouse tables for validation logs and quarantined records

AI-assisted drafting: AI was used to help structure and phrase this response. I reviewed and validated the technical content before posting.