How to monitor an AI system in production for drift

Why does an AI system need monitoring after release?

A model is tested on data collected before release, and production data keeps changing after that date. Rabanser et al. note that machine learning systems "tend to fail silently" when inputs change [7]. The NIST AI RMF states that AI systems may need corrective maintenance "due to data, model, or concept drift" [2]. Monitoring is the test that keeps running after the release test ends.

What is the difference between data drift and concept drift?

Lu et al. define concept drift as "a phenomenon in which the statistical properties of a target domain change over time in an arbitrary way" [1]. Formally, drift occurs when the joint distribution of inputs X and labels y changes, written Pt(X, y) ≠ Pt+1(X, y) [1]. The review splits this into three sources [1].

drift: Change in input data or model behaviour after release that lowers quality.

Source in Lu et al. [1]	What changes	Name used in this article
Source I	Input distribution P(X); P(y given X) unchanged. Lu et al. call it virtual drift	Data drift
Source II	Relationship P(y given X); inputs unchanged. Lu et al. call it actual drift	Concept drift
Source III	Both at once	Data and concept drift

The difference matters for testing. Source I "does not affect the decision boundary", so the model may still be correct on the new inputs [1]. Source II changes the decision boundary and leads to "learning accuracy decreasing" [1]. Input checks detect data drift without labels, while concept drift needs fresh labels or graded outputs to see.

How does the population stability index work?

The population stability index (PSI) is "a statistic that measures how much a variable has shifted over time" [5]. The variable is split into bins, and PSI sums, over the bins, the difference between baseline and current shares multiplied by the natural log of their ratio [3]. Khademi et al. show that PSI is a symmetric version of the Kullback-Leibler divergence [3].

Common interpretation bands, attributed by du Pisanie et al. to Lewis, read below 0.1 as no substantial change, 0.1 to under 0.25 as a small change, and 0.25 or more as a substantial change [4]. Yurdakul notes that such benchmarks are used "without reference to Type I or Type II error rates" [5]. Du Pisanie et al. add that rules of thumb are "difficult to justify from a statistical perspective" [4]. A test plan should therefore fix the binning and the bands per feature before release and record them with the baseline.

When is a Kolmogorov-Smirnov test the better check?

The Kolmogorov-Smirnov (K-S) test compares empirical distribution functions and uses "the maximum distance between these two curves" as its statistic [6]. For drift, the two curves are the baseline sample and the current window [7]. It needs no binning, which removes one design choice that PSI depends on.

The NIST/SEMATECH e-Handbook lists limits: the test "only applies to continuous distributions" and is more sensitive near the centre of the distribution than at the tails [6]. For categorical features, Rabanser et al. use Pearson's chi-squared test instead [7]. When many features are tested at once, they apply a Bonferroni correction, rejecting stability only if the smallest p-value is below α/K for K tests [7]. Du Pisanie et al. warn that the large samples common in production "cast doubt" on reading p-values alone [4].

Check	Input type	Output	Main limit
PSI [3], [4]	Binned numeric or categorical	Index read against bands	Bands have no stated error rates [5]
Two-sample K-S [6], [7]	Continuous	Statistic and p-value	Less sensitive at the tails [6]
Chi-squared [7]	Categorical	Statistic and p-value	Needs an adequate sample size for its approximation [6]

Rabanser et al. found that, across the shifts they studied, two-sample testing on representations from a pre-trained classifier performed best [7]. For text and image inputs, the same approach means running the tests on embeddings or model scores rather than on raw text or pixels [7]. The model validation service covers the baseline these tests compare against, and predictive ML entries list who states it.

How is LLM output evaluated online?

Distribution checks on prompts do not show whether answers are still correct. Online evaluation draws a random sample of live conversations at a fixed rate, grades each against the same rubric used before release, and tracks the pass rate per day or per release. The grader can be an LLM-as-judge with a human reviewing a subset.

LLM-as-judge: Using one language model to grade another model's answers against a rubric.

Zheng et al. found that strong LLM judges reached "over 80% agreement" with human preferences, the same level as agreement between humans [8]. The same paper documents "position, verbosity, and self-enhancement biases" in LLM judges [8]. A monitoring plan therefore keeps a human-graded calibration set and rechecks judge agreement whenever the judge model or rubric changes.

A pass rate from a sample is an estimate with an interval. The NIST/SEMATECH e-Handbook describes the Wilson method for confidence intervals on proportions [9]. Reporting the interval stops a small daily sample from triggering an alert on noise. The post how to evaluate LLM applications covers rubric design and judge setup in more detail.

How does an incident become a regression test?

The incident-to-regression loop turns each confirmed production failure into a permanent test case. The failing input, the expected behaviour and the model and prompt versions are added to the golden set. The fix ships only when the new case passes and the rest of the set still passes.

golden set: A fixed set of inputs with approved outputs used for regression checks.

This loop is what prompt and model regression testing covers before each release. Lu et al. note that drift detectors "require a minimum number of new data" before they signal, so some incidents reach the team through user reports before any alert fires [1]. User reports therefore feed the same regression set as automated alerts. The post how AI product testing differs from traditional QA compares this loop with conventional regression testing.

How does monitoring map to the NIST AI RMF?

The AI RMF Core has four functions: GOVERN, MAP, MEASURE and MANAGE [2]. MEASURE "employs quantitative, qualitative, or mixed-method tools" to "analyze, assess, benchmark, and monitor AI risk" [2]. MANAGE covers risk treatment, including plans "to respond to, recover from, and communicate about incidents or events" [2].

Monitoring activity	AI RMF subcategory [2]
Distribution checks and online grading in production	MEASURE 2.4: functionality and behaviour monitored in production
Tracking performance in deployed contexts	MEASURE 3.1
User problem reports feeding evaluation metrics	MEASURE 3.3
Post-deployment monitoring plan	MANAGE 4.1
Rollback or deactivation when outcomes diverge	MANAGE 2.4
Incident tracking, response and recovery	MANAGE 4.3

A monitoring report that cites these subcategories can be reused as evidence in a compliance and conformity file.

Which entries state monitoring and drift testing?

31 published entries state production monitoring and drift testing, and 17 state prompt and model regression testing. For LLM applications, the split is below.

Counts reflect what each vendor's AI page states, as described in the methodology.

Production monitoring checklist

1. 01

   Freeze a baseline

   Store the release-time sample of inputs, model scores and graded outputs, with model and prompt versions.

2. 02

   Choose a check per feature

   PSI with fixed bins and bands, two-sample K-S for continuous features, chi-squared for categorical ones [3], [4], [6], [7].

3. 03

   Correct for many tests

   When testing K features, use a Bonferroni threshold of α/K or a stated alternative [7].

4. 04

   Watch embeddings and scores

   For text and images, run the shift tests on embeddings or model scores rather than raw inputs [7].

5. 05

   Grade a live sample

   Sample conversations at a fixed rate and grade them with the release rubric; report the pass rate with a Wilson interval [9].

6. 06

   Calibrate the judge

   Keep a human-graded set and recheck LLM judge agreement after any judge or rubric change [8].

7. 07

   Route user reports

   Feed user problem reports into the same metrics as automated alerts, as MEASURE 3.3 describes [2].

8. 08

   Close the loop

   Add each confirmed incident to the golden set and block release until it passes; record it under MANAGE 4.3 [2].

9. 09

   Define rollback

   Name who can roll back or switch off the model and at what threshold, as MANAGE 2.4 describes [2].

Sources

[1] Lu, J., Liu, A., Dong, F., Gu, F., Gama, J., Zhang, G. "Learning under Concept Drift: A Review", IEEE Transactions on Knowledge and Data Engineering (arXiv:2004.05785), 2018, https://arxiv.org/abs/2004.05785

[2] NIST AI 100-1, Artificial Intelligence Risk Management Framework (AI RMF 1.0), NIST, 2023, https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.100-1.pdf

[3] Khademi, A., Hopka, M., Upadhyay, D. "Model Monitoring and Robustness of In-Use Machine Learning Models: Quantifying Data Distribution Shifts Using Population Stability Index", arXiv:2302.00775, 2023, https://arxiv.org/abs/2302.00775

[4] du Pisanie, J., Allison, J., Budde, C., Visagie, J. "A critical review of existing and new population stability testing procedures in credit risk scoring", arXiv:2303.01227, 2023, https://arxiv.org/abs/2303.01227

[5] Yurdakul, B. "Statistical Properties of Population Stability Index", PhD dissertation, Western Michigan University, 2018, https://scholarworks.wmich.edu/dissertations/3208/

[6] 1.3.5.16 Kolmogorov-Smirnov Goodness-of-Fit Test, NIST/SEMATECH e-Handbook of Statistical Methods, NIST, 2012, https://www.itl.nist.gov/div898/handbook/eda/section3/eda35g.htm

[7] Rabanser, S., Günnemann, S., Lipton, Z. C. "Failing Loudly: An Empirical Study of Methods for Detecting Dataset Shift", NeurIPS (arXiv:1810.11953), 2019, https://arxiv.org/abs/1810.11953

[8] Zheng, L. et al. "Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena", NeurIPS Datasets and Benchmarks Track (arXiv:2306.05685), 2023, https://arxiv.org/abs/2306.05685

[9] 7.2.4.1 Confidence intervals, NIST/SEMATECH e-Handbook of Statistical Methods, NIST, 2012, https://www.itl.nist.gov/div898/handbook/prc/section2/prc241.htm