Bias audits for AI systems: what they measure and how they run

What a bias audit measures

A bias audit compares the outcomes of an AI system, and often its errors, across groups defined by characteristics such as sex, race or ethnicity. The output is a set of rates per group, the ratios or differences between them, and the data they rest on. Most audits cover systems that score or classify people, such as predictive ML models used in hiring, lending or insurance.

bias audit: Measurement of whether a model's outcomes differ across demographic groups.

NIST groups bias in AI into three categories: systemic, statistical and human [5]. A bias audit mostly measures the statistical part, in data and outputs. NIST also states that it is not possible to achieve zero risk of bias in an AI system [5]. An audit report therefore describes measured differences on stated data; it does not certify that bias is absent.

What NYC Local Law 144 requires

Local Law 144 of 2021 bars employers and employment agencies in New York City from using an automated employment decision tool unless it has had a bias audit within one year of use [1]. Information about the audit must be public, and candidates and employees must receive notices [1]. The city's Department of Consumer and Worker Protection began enforcing the law and its rule on July 5, 2023 [1].

The audit must be done by an independent auditor [2]. Under the rule, an auditor is not independent if it was involved in using, developing or distributing the tool [2]. The same applies if, during the audit, it has an employment relationship with the employer or the tool vendor, or a direct or material indirect financial interest in either [2].

The audit calculates a selection rate and an impact ratio for each category [2]. The impact ratio is the selection rate for a category divided by the selection rate of the most selected category [2]. For tools that score rather than select, the rule uses a scoring rate: the share of a category scoring above the sample's median score [2].

Calculations run separately for sex, for race and ethnicity, and for intersectional categories combining the two [2]. The audit reports how many people fell into an unknown category [2]. An auditor may exclude a category that makes up less than 2% of the audit data, with a written justification [2]. Test data may replace historical data only when there is not enough historical data for a statistically significant audit [2].

Before using the tool, the employer publishes the date of the most recent audit, a summary of results and the tool's distribution date [2]. The summary stays online for at least 6 months after the tool's latest use [2]. Notice to candidates can be given at least 10 business days before use, for example on the careers page or in the job posting [2].

How the impact ratio relates to the four-fifths rule

The rule's own worked example shows the arithmetic [2]. Men are selected at 48% and women at 47%, so the impact ratio for women is 0.979. In the race and ethnicity table, a group selected at 44% against a top rate of 52% gets 0.84 [2]. One intersectional group in the example has 17 applicants, 7 selected and an impact ratio of 0.789 [2].

The adopted NYC rule requires these ratios to be calculated and published; it sets no pass mark [2]. For comparison, the federal Uniform Guidelines on Employee Selection Procedures of 1978 set a reference point. Under 29 CFR 1607.4(D), a selection rate below four-fifths, or 80 percent, of the highest group's rate will generally be regarded by federal enforcement agencies as evidence of adverse impact [3].

The same paragraph limits the rule in both directions [3]. Smaller differences may still count as adverse impact when they are significant in statistical and practical terms [3]. Larger differences may not count when they rest on small numbers and are not statistically significant [3]. The group with 17 applicants in the example above falls below four-fifths, which is the case where a significance test belongs in the report.

What EU AI Act Article 10 adds

The EU AI Act lists AI systems for recruitment and selection, including tools that filter applications and evaluate candidates, as high-risk in Annex III [4]. For high-risk systems, Article 10 makes bias part of data governance for training, validation and testing data [4]. Providers must examine data for possible biases that could harm health and safety, affect fundamental rights or lead to prohibited discrimination [4]. They must also take measures to detect, prevent and mitigate the biases found [4].

Article 10(5) lets providers process special categories of personal data for bias detection and correction only under strict conditions [4]. The processing must be necessary because other data, including synthetic or anonymised data, would not do [4]. The data may not be transmitted to or accessed by other parties and must be deleted once the bias is corrected [4]. Records must state why the processing was strictly necessary [4].

The two regimes differ in focus. The NYC rule asks for an outcome audit by an independent party with public results; Article 10 asks the provider to govern its data and keep records. The guide on EU AI Act testing services maps the other articles, and compliance and conformity assessment support lists vendors stating that work.

Which fairness metrics an audit can use

• Demographic parity. The decision is independent of the protected attribute, so positive rates are equal across groups [6]. The NYC impact ratio compares exactly these rates [2]. Hardt, Price and Srebro note that parity can hold while qualified people are accepted in one group and unqualified people in another [6].
• Equalized odds. True positive rates and false positive rates are equal across groups [6]. This needs ground truth labels, such as later job performance or loan repayment.
• Equal opportunity. A relaxed form that equalises only true positive rates, the rate at which qualified people receive the positive outcome [6].
• Predictive parity. The positive predictive value is the same across groups: a positive prediction is equally reliable for each group [8].
• Calibration within groups. A score means the same probability of the outcome in every group [7].

Why the metrics cannot all hold at once

Chouldechova shows that when the base rate of the outcome differs between groups, a score with predictive parity cannot also have equal false positive and false negative rates [8]. Kleinberg, Mullainathan and Raghavan prove that calibration within groups and balance for the positive and negative classes cannot hold together except in highly constrained special cases [7].

The choice of metric is therefore a decision about which error matters more in the use case, not a technical default. An audit report names the metrics used, explains why they fit the decision being audited, and shows the others where the data allow. The post on how AI product testing differs from traditional QA covers related test methods such as attribute swaps.

How a bias audit runs

1. 01

   Fix the scope

   Name the decision, the tool version, the population and the jurisdictions, such as New York City, the EU or both.

2. 02

   Choose groups and metrics

   List the categories the law requires, including intersectional ones for NYC, and the fairness metrics chosen, with the reason for each.

3. 03

   Collect and document data

   Use historical data where it exists; document any test data, unknown categories and the legal basis for special category data.

4. 04

   Compute with sample sizes

   Calculate rates, impact ratios and error rates per group, each with its count and a significance test.

5. 05

   Review exclusions and causes

   Justify any small category excluded, then trace large gaps to features, labels or thresholds.

6. 06

   Report and schedule

   Publish what the law requires, record the tool's distribution date, and set the next audit date.

The evidence pack that comes out of the audit holds:

• a statement of auditor independence against the three NYC criteria [2];
• the source and explanation of the data, and how any test data was generated [2];
• counts, selection or scoring rates and impact ratios for every category, plus the unknown category count [2];
• the fairness metrics chosen, the reason, and the trade-offs accepted;
• for EU high-risk systems, the Article 10 bias examination, the mitigation measures, and the records on special category data [4];
• the tool version and date audited.

Who states bias audits in this directory

In this directory, 44 entries state bias and fairness audits on their own AI pages. The table splits them by company type next to compliance support:

A listing records what a vendor's page states, not whether the vendor meets the NYC independence test for your tool. Check the three criteria in writing; the post on how to choose a vendor to test an AI product covers the rest of the selection. The AI assurance and audit specialist page lists that company type.

Sources

• [1] Automated Employment Decision Tools (AEDT), NYC Department of Consumer and Worker Protection, 2023, nyc.gov/site/dca/about/automated-employment-decision-tools.page
• [2] Notice of Adoption of Final Rule: Automated Employment Decision Tools, 6 RCNY Chapter 5 Subchapter T, NYC Department of Consumer and Worker Protection, 2023, rules.cityofnewyork.us adopted rule text (PDF)
• [3] 29 CFR 1607.4, Information on impact (Uniform Guidelines on Employee Selection Procedures), Electronic Code of Federal Regulations, 1978 (amended 1981), ecfr.gov/current/title-29/subtitle-B/chapter-XIV/part-1607/section-1607.4
• [4] Regulation (EU) 2024/1689 (Artificial Intelligence Act), Article 10 and Annex III, Official Journal of the European Union, 2024, eur-lex.europa.eu/legal-content/EN/TXT/HTML/?uri=OJ:L_202401689
• [5] Schwartz et al., Towards a Standard for Identifying and Managing Bias in Artificial Intelligence, NIST Special Publication 1270, National Institute of Standards and Technology, 2022, nvlpubs.nist.gov/nistpubs/SpecialPublications/NIST.SP.1270.pdf
• [6] Hardt, Price and Srebro, Equality of Opportunity in Supervised Learning, arXiv, 2016, arxiv.org/abs/1610.02413
• [7] Kleinberg, Mullainathan and Raghavan, Inherent Trade-Offs in the Fair Determination of Risk Scores, Innovations in Theoretical Computer Science (ITCS), 2017, arxiv.org/abs/1609.05807
• [8] Chouldechova, Fair prediction with disparate impact: A study of bias in recidivism prediction instruments, arXiv, 2017, arxiv.org/abs/1703.00056