Skip to content

How AI product testing differs from traditional QA

By AI Testing Vendors · 2026-11-09

Traditional QA checks that a given input produces a known expected output, while an AI product can return different outputs for the same input and often has no single correct answer to compare against. AI testing therefore measures behaviour statistically on evaluation sets, checks relations between outputs, tests the data and the model as well as the code, and continues after release.

Which assumption of traditional QA breaks

Conventional test design assumes a test oracle: the developer knows the expected value and checks the output against it [1]. Machine learning systems are hard to test because they are built to answer questions for which no previous answer exists [1]. The NIST AI Risk Management Framework lists the same gap among AI-specific risks: the ground truth "may either not exist or not be available" [7]. It also names the difficulty of "determining what to test" [7].

The practical effect is that a pass or fail per test case stops being enough. A QA team moving into AI work needs new answers to three questions: what counts as correct, how often the system must be correct, and how to tell when that rate changes. The post on how to choose a vendor to test an AI product turns these into questions for an external team.

Why the same input gives different outputs

Generative models sample their output, so repeated calls with the same prompt can differ. One study ran 829 code generation problems from three benchmarks through a commercial chat model, sending each prompt five times [2]. The share of tasks where no two requests produced equal test output was 75.76% on CodeContests, 51.00% on APPS and 47.56% on HumanEval [2]. Setting the temperature to 0 reduced this variation but did not remove it [2].

A single passing run therefore proves little about the next run. Each test case needs several runs, and the result is a pass rate with its spread rather than a single verdict. Prompt and model regression testing applies this to every change of prompt, model version or retrieval index.

Record the conditions of every run next to its result: model version, sampling settings, prompt version and retrieval index. Without that record, a change in the pass rate cannot be traced to a cause. The same record lets a vendor and your own team reproduce each other's numbers.

What replaces a fixed assert

An evaluation set replaces the single expected value: a fixed collection of inputs, each with an approved answer or a scoring rubric. A frozen version of it serves as a golden set for regression runs. The test outcome becomes a metric over the set with a threshold, such as the share of answers a reviewer marks as correct and grounded.

golden set: A fixed set of inputs with approved outputs used for regression checks.

Labels are the expensive part. In ML testing, answers are typically confirmed by hand, and finding the oracle is labour-intensive because it needs domain knowledge [1]. Teams that grade with an LLM-as-judge reduce that cost. One study measured over 80% agreement between strong model judges and human preferences, alongside position, verbosity and self-enhancement biases [6].

LLM-as-judge: Using one language model to grade another model's answers against a rubric.

Vendors in this directory state evaluation work for LLM applications under these services:

Services stated by entries testing LLM applications
CheckDirectory serviceEntries stating it for LLM applications
Hallucination and factuality checks, answer quality rubrics, LLM-as-judge with human reviewLLM output evaluation34
Retrieval quality, grounding, citation accuracy, chunking regressionsRAG pipeline validation14
Golden sets, non-deterministic assertions, repeated runs, version diffsPrompt and model regression testing14

The service page for LLM output evaluation and the post on how to evaluate LLM applications cover metrics and rubrics in detail.

How metamorphic testing works without an oracle

Metamorphic testing checks how outputs relate across related inputs, so it needs no known correct answer. Chen, Cheung and Yiu proposed it in a 1998 technical report as a way to derive new test cases from successful ones [3]. They noted that the technique can be used in the absence of test oracles [3]. A textbook example: sin(x) must equal sin(pi minus x), so a difference signals a fault without computing either value [1].

metamorphic testing: Checking that related inputs produce outputs related in a predictable way.

The method carries over to machine learning. Xie and colleagues applied it to classifier implementations and detected real faults in a popular open-source classification program [4]. They also reported that checking cross-validation results alone was not enough to detect faults in a supervised classifier [4].

For language models, CheckList adds two test types inspired by metamorphic tests [5]. An invariance test applies label-preserving changes to an input and expects the same prediction; a directional expectation test expects the output to change in a known direction [5]. In the authors' user study, practitioners using the method created twice as many tests and found almost three times as many bugs [5].

Relations that fit an LLM product include these. A paraphrased question should return the same facts. An irrelevant document added to the retrieved context should not change the answer. Swapping a name or gender in an otherwise identical case should not change a decision, which links this method to bias audits for AI systems.

Which risks sit in the data and the model

In traditional software, new data does not usually change the core behaviour of the system; in a machine learning system it can [1]. NIST names data that does not represent the intended context, harmful bias, and training sets that become stale or detached from their original context [7]. Article 15 of the EU AI Act names data poisoning, model poisoning and adversarial examples as attacks that providers of high-risk systems must address [9].

None of these shows up in a code review. They need tests on the data itself and on the trained model. Test data also has to stay separate from training and tuning data, or the scores measure recall of seen examples instead of behaviour on new ones. In this directory, 25 entries state training data and labeling QA and 31 entries state ML model validation.

Why testing continues after release

NIST states that AI systems should be tested before deployment and regularly while in operation [7]. It adds that they may need more frequent maintenance because of data, model or concept drift [7]. The ML Test Score rubric lists 28 specific tests and monitoring needs drawn from production ML systems [8].

drift: Change in input data or model behaviour after release that lowers quality.

For high-risk systems, Article 72 of the EU AI Act requires a post-market monitoring system that collects and analyses performance data "throughout their lifetime" [9]. In this directory, 31 entries state production monitoring and drift testing. The post on monitoring AI in production and drift covers signals and retest triggers.

What conventional QA still covers

The model is one component inside a product that still has APIs, a user interface, authentication, logging and integrations. Those parts keep their deterministic tests, and a failure there looks the same as in any other software release. Functional, integration, performance and security testing of that surrounding code stays in the plan.

What changes is the organisation of the work around the model. NIST describes test, evaluation, verification and validation as tasks performed throughout the AI lifecycle, from design and data collection to operation [7]. It also states that, ideally, the people doing verification and validation are distinct from those doing test and evaluation [7]. For a QA lead, that means evaluation work starts at data collection, not at the release candidate, and that a separate reviewer signs off on the evaluation design.

How to adapt a QA plan for an AI product

The six changes below turn a conventional test plan into one that fits an AI product.

  1. 01

    Replace asserts with thresholds

    For each risk, define a metric, a pass threshold and the number of cases that the result must rest on.

  2. 02

    Repeat every run

    Run each case several times and report the pass rate and its spread, not a single verdict.

  3. 03

    Freeze a golden set

    Version the evaluation set together with the prompt, model and retrieval index it was scored against.

  4. 04

    Add metamorphic relations

    Test paraphrases, irrelevant context and attribute swaps where the expected relation between outputs is known.

  5. 05

    Test the data

    Check representativeness for the intended users, label quality and overlap between training and test data.

  6. 06

    Keep testing in production

    Monitor inputs and outputs, and write down which changes trigger a rerun of the full evaluation set.

The LLM application testing checklist applies these steps to chat and retrieval products.

Sources

Related pages

Request a listing or correction