How should Microsoft Fabric data pipelines handle AI-driven validation ?

Hi codeautomation​,

Thank you for reaching out to Microsoft Fabric Community.

Use the Fabric pipeline to orchestrate the process and a Fabric notebook to perform the validation. The notebook can check schema, nulls, duplicates, value ranges and anomalies before writing to the curated Lakehouse table.

Return the validation status to the pipeline and use an If Condition activity. Critical validation failures can trigger a Fail activity, while noncritical issues can be logged as warnings and written to a quarantine table. Retry only temporary technical failures, not data-quality failures. For large datasets, validate only new or changed partitions and use Spark-based checks rather than processing records individually.

Thanks and regards,
Anjan Kumar Chippa