Validation/optimisation options for AI features

Hi fabricuser123_ ,

First thing to assume: AI Functions results will never be 100% deterministic. Microsoft's own documentation includes the warning # This code uses AI. Always review output for mistakes in every code example — that's not decorative, it's a real acknowledgment of expected model behavior.

That said, there are two approaches depending on what you need:

Scenario A — Validate before trusting results

Microsoft has published evaluation notebooks that measure output quality using LLM-as-a-Judge: a larger model acts as evaluator and computes accuracy, precision, recall and F1 metrics. The workflow is: run the function with an executor model, evaluate with a judge model, identify which predictions need review, and refine labels or configuration. ( https://blog.fabric.microsoft.com/en-us/blog/unlock-insights-from-images-and-pdfs-with-multimodal-support-in-fabric-ai-functions-preview/ )

Use this before going to production to establish a quality baseline on your own dataset.

Scenario B — Control quality directly in production

A more robust pattern is adding a Data Quality layer after the AI Function:

• Layer 1: the AI Function generates the label → sentiment_ai column
• Layer 2: a DQ rule flags uncertain cases → needs_review = true when the model returns mixed, the text is very short, or contains negations/irony

There is a documented pattern where, if a second judge model disagrees with the assigned label, the pipeline conditionally triggers a relabeling step. This loop is orchestrated using Fabric Pipelines, which support conditional and iterative control flow. (https://learn.microsoft.com/en-us/fabric/data-science/tutorial-text-classification)

The human factor doesn't disappear — it gets focused on flagged cases, not 100% of the dataset. That's what makes the process sustainable at scale.

If this response was helpful, please give it a like and mark it as a solution — it encourages me to keep contributing and helps other users find answers faster 🙌