Skip to content

How to choose a vendor to test an AI product

By AI Testing Vendors · 2026-11-09

Write down the product type, the risks and the metrics before contacting anyone, then shortlist vendors whose own AI pages state the matching services and send each of them the same written questions. Run a short paid pilot against acceptance criteria fixed in advance, and sign only when data handling, ownership of test assets and reporting are written into the contract.

What to write down before contacting vendors

Four facts come before any vendor call: the product type, its intended purpose, the failures that would hurt users or the business, and the data testers will need. A vendor can only plan and price against a scope that names these. The NIST AI Risk Management Framework states that AI systems should be tested before deployment and regularly while in operation [1]. Plan for both phases now, even if the contract covers only pre-release work.

Next, turn each failure into a metric with a threshold. For high-risk systems, the EU AI Act requires testing against "prior defined metrics and probabilistic thresholds" suited to the intended purpose, before the system is placed on the market [2]. Products outside that category gain the same benefit from the habit: proposals that can be compared on one scale.

For LLM applications, the OWASP Top 10 for LLM Applications 2025 is a public risk list to start from [3]. Its entries include prompt injection, excessive agency, system prompt leakage, and vector and embedding weaknesses [3]. Mark which entries apply to your architecture before a vendor does it for you. The post on how AI product testing differs from traditional QA explains why these risks need methods beyond fixed test scripts.

prompt injection: Input crafted to override a model's instructions or extract hidden data.

When regulation changes the scope

A provider of a high-risk AI system under the EU AI Act must run a quality management system and put the system through a conformity assessment before release [2]. That quality management system must include "examination, test and validation procedures" before, during and after development, with their frequency [2]. Article 15 names attacks the provider has to address, including data poisoning, model poisoning, adversarial examples and confidentiality attacks [2]. A vendor can produce test evidence for these duties, but the legal obligation stays with the provider.

conformity assessment: The EU AI Act procedure showing a high-risk system meets legal requirements.

The guide EU AI Act testing services maps articles to testing tasks. The service page for compliance and conformity assessment support lists the vendors that state it.

How to build the shortlist

Filter on product type, then on service, then on location. This directory records the AI product types each vendor's own AI page names, with the date the page was read.

Published entries stating each AI product type
AI product typePublished entriesShare
LLM applications6169%
AI agents4349%
Predictive ML models2731%
Voice AI1719%
Computer vision1315%
Recommender and ranking systems1113%
Autonomous and embedded AI56%
Generative media22%
Document AI11%

Inside one product type, check which services vendors state. For LLM applications the split is:

Services stated by entries testing LLM applications
CheckDirectory serviceEntries stating it for LLM applications
Hallucination and factuality checks, answer quality rubrics, LLM-as-judge with human reviewLLM output evaluation34
Retrieval quality, grounding, citation accuracy, chunking regressionsRAG pipeline validation14
Golden sets, non-deterministic assertions, repeated runs, version diffsPrompt and model regression testing14
Jailbreaks, harmful output elicitation, OWASP Top 10 for LLM Applications, MITRE ATLASRed teaming and adversarial testing45
Prompt injection, data exfiltration, agent and tool abuse, supply chain; OWASP LLM Top 10AI security penetration testing39

A service marked "not stated" on an entry means the vendor's AI page does not name it. Treat that as a question for the request for proposal, not as a reason to exclude. Product type pages such as LLM applications and AI agents list the entries for each type.

Location comes last because it decides where your data travels. That choice feeds directly into the contract terms covered below.

Which questions to put in the request for proposal

Send every shortlisted vendor the same questions and ask for written answers. Each question maps to the directory filter that answers its opening part, so the shortlist and the request use one vocabulary.

Questions to ask a vendor and the directory facet that answers the first filter
Question to ask the vendorFilter in this directory
Which of these AI product types have you tested for clients, with a public case study URL for each?AI product type
How do you build the evaluation set: source, size, labelling, and who keeps it afterwards?LLM output evaluation
Do you test retrieval and grounding separately from answer quality?RAG pipeline validation
Which reference list do your adversarial tests follow, and which OWASP entries are in scope?Red teaming and adversarial testing
How do you rerun the suite after a prompt or model change, and what does a rerun report contain?Prompt and model regression testing
Which group metrics do you compute in a bias test, and is the auditor independent of the model developer?Bias and fairness audit
Which documents do you deliver for EU AI Act Articles 9, 15 and 17?Compliance and conformity assessment support
What do you monitor after release, and what triggers a retest?Production monitoring and drift testing
Who owns test sets, attack prompts, scripts and evaluation code after the engagement?Not a directory field; ask in writing

How to read a proposal

Read every proposal against the same five areas.

  • Test design. Each risk on your list maps to test cases and a metric. Generic test categories with no link to your risks cannot be checked.
  • Evaluation sets. The proposal names the source, size and labelling process of the test data, plus a fixed golden set for regression runs. It also says who keeps the set when the contract ends.
  • Human review. Some proposals score answers with an LLM-as-judge. One study found strong model judges reached over 80% agreement with human preferences, and also found position, verbosity and self-enhancement biases [6]. A proposal using a model judge states how its scores are calibrated against human reviewers.
  • Adversarial coverage. NIST describes AI red teaming as a structured exercise to find flaws such as inaccurate, harmful or discriminatory outputs [5]. NIST adds that result quality depends on the background and expertise of the red team [5]. The proposal names the team's roles, the reference list and the share of manual work.
  • Reporting. The NIST AI RMF calls for performance assessment with associated measures of uncertainty and formalized reporting of results [1]. A usable report gives sample sizes, uncertainty ranges and reproduction steps for every failure.
golden set: A fixed set of inputs with approved outputs used for regression checks. LLM-as-judge: Using one language model to grade another model's answers against a rubric. red teaming: Deliberate attempts to make a system produce harmful or unintended output.

How to run a paid pilot

A pilot tests the shortlisted vendors on a small slice of the real work before a long contract. The criteria are written before the pilot starts, so the result does not depend on which report reads better.

  1. 01

    Pick one slice

    Choose one feature or user flow that carries a real risk and is small enough to finish within the window you set.

  2. 02

    Fix acceptance criteria

    Write the metrics, thresholds and deliverables in advance and send them to every vendor in the same words.

  3. 03

    Give equal inputs

    Give each vendor the same build, documentation and test data under the same data processing terms.

  4. 04

    Ask for raw results

    Require the evaluation set, scores per item and reproduction steps, not only a summary deck.

  5. 05

    Reproduce findings

    Have your own team rerun a sample of the reported failures in its own environment.

  6. 06

    Score and record

    Score each pilot against the written criteria and file the scoring with the contract decision.

The engagement models vendors name on their AI pages show which formats are on offer. A fixed-scope project or a one-time audit is the closest match to a pilot.

Engagement models named on vendors' AI pages
Engagement modelEntries naming it on the AI page
Staff augmentation3
Dedicated team3
Managed testing9
Fixed-scope project8
One-time audit or assessment25
Retainer or subscription3
None named51

Timing and pricing models are covered in how long AI testing takes and AI testing cost.

What to put in the contract

Data handling. If testers process personal data on your behalf, GDPR Article 28 requires a written contract with the processor [4]. Under it, the processor acts only on documented instructions, deletes or returns the data when the service ends, and allows audits by the controller [4]. A processor may not engage another processor without written authorisation [4], so ask for the list of subcontractors and crowd testers.

Data for bias tests. Article 10(5) of the EU AI Act lets providers of high-risk systems process special categories of personal data for bias detection only under listed conditions [2]. Those conditions include that the data are not transmitted to or accessed by other parties, and are deleted once the bias is corrected [2]. Agree with counsel how an external tester works within these conditions, for example inside your own environment.

Test assets. None of the standards cited here settles who owns evaluation sets, attack prompts, scripts and evaluation code, so the contract has to. Write down what transfers to you, what the vendor keeps as pre-existing material, and what licence you hold to rerun it. The NIST AI RMF lists third-party intellectual property among the risks that policies should address [1].

Certificates. ISO/IEC 42001:2023 specifies requirements for an AI management system within an organization [7]. Ask which sites and delivery teams a vendor's certificate covers.

Red flags in vendor answers

  • Scores reported without sample sizes or uncertainty [1].
  • Model-graded results with no calibration against human reviewers [6].
  • Adversarial testing with no named reference list and no description of the team [5].
  • A promise of zero hallucinations or of guaranteed compliance, when the legal obligation and the conformity assessment stay with the provider [2].
  • The team that built or sells the model offering to audit it; NIST notes that independent review can mitigate conflicts of interest [1]. The post on bias audits for AI systems covers a legal definition of auditor independence.
  • Client names with no public case study URL.
  • No written answer on who owns the test assets.
hallucination: Model output that is fluent but not supported by facts or sources.

Sources

Related pages

Request a listing or correction