AI Fairness Audit Checklist

Fairness in AI is a tested property, not a stated value. Systems that influence credit, employment, insurance, healthcare, or access to services can produce disparate outcomes across groups through data, proxies, or thresholds, and regulators increasingly expect organizations to have looked. This checklist covers how to look and what to document. It complements ai bias and fairness, the responsible AI implementation whitepaper, and ai explainability requirements. This is general guidance, not legal advice.

Who should use this checklist?

Model owners and engineers, compliance and legal partners, and governance bodies responsible for systems that affect people.

Is the audit scoped?

1. The decision or output under audit and its consequence are defined.
2. Affected groups and protected characteristics are identified with legal input for the jurisdiction and sector.
3. Proxies for protected characteristics in the data are considered.
4. Regulatory requirements for testing and documentation are identified.
5. The audit owner and independence from the build team are set proportionate to consequence.

Reference: ai in regulated industries.

Are metrics chosen and documented?

Consideration	Documented?
Fairness definitions relevant to the decision context
Metrics selected, with legal and domain rationale
Known conflicts between metrics acknowledged
Thresholds for acceptable disparity and their basis
Accuracy and business metrics reported alongside

Has the data been assessed?

1. Representation of groups in training and evaluation data compared to the population served.
2. Proxy variables identified through correlation and domain review.
3. Label bias considered: do historical labels encode past discrimination?
4. Missing data patterns across groups examined.
5. Sensitive attributes available for testing under access controls, or a documented method where they are not.

Reference: the ai training data checklist.

Has pre-launch testing been done?

1. Outcomes across groups measured on evaluation data with the chosen metrics.
2. Intersectional groups examined where sample sizes allow.
3. Error types (false positives and false negatives) compared across groups.
4. Threshold sensitivity analyzed.
5. Explanations checked for reliance on proxies.
6. Results compared to thresholds and documented.

Reference: the AI evaluation and testing whitepaper.

Is remediation planned and verified?

1. Root cause of disparities identified: data, features, model, thresholds, process.
2. Remediation chosen and applied: data improvement, feature changes, model or threshold changes, human review for affected cases, process redesign.
3. Re-testing after remediation.
4. Trade-offs between fairness metrics and accuracy documented and approved by governance.
5. Residual disparities recorded with rationale.

Is production monitored?

1. Group outcome metrics tracked on a schedule proportionate to consequence.
2. Drift in group distributions and outcomes alerted.
3. Complaints and appeals analyzed for patterns.
4. Re-audit triggers: model changes, data changes, regulatory changes, time.

Reference: the AI observability whitepaper.

Are human oversight and recourse in place?

1. Human review for consequential decisions where required or where disparities warrant.
2. Adverse-action explanations where regulation requires.
3. Appeal and correction processes for affected individuals.
4. Reviewers trained on fairness considerations.

Reference: ai human oversight requirements.

Is the audit documented?

1. Methodology: scope, groups, metrics, data, tests, thresholds.
2. Results before and after remediation.
3. Decisions and trade-offs with approvers.
4. Monitoring plan and re-audit schedule.
5. Documentation stored with the model registry record and available to governance and regulators.

Reference: how to build a model registry and what is a model card.

Is governance engaged?

1. The governance body reviews audit results for high-consequence systems.
2. Risk tiering reflects fairness exposure.
3. Policy defines when audits are required and by whom.
4. Vendor systems are audited or their audits reviewed.

Reference: the ai governance checklist.

How should findings be handled?

Disparities beyond thresholds in consequential decisions block launch until remediated and re-tested or until human review is placed on affected decisions with governance approval. Documentation gaps block sign-off. Monitoring gaps block autonomy increases.

Worked example: a lead prioritization model

A lender's model prioritizes loan applications for underwriter review. The audit scopes the decision as consequential, identifies protected groups with counsel, and finds a location feature acting as a proxy. Selection rates and error rates are compared across groups on evaluation data, and one group shows a materially higher false-negative rate. The proxy feature is removed, the model retrained, and testing repeated; the residual disparity falls within threshold and is documented with the accuracy trade-off approved by governance. Group outcomes are added to monthly monitoring, and the methodology is stored with the registry version for regulatory review.

How FISTA Solutions approaches fairness

FISTA Solutions builds fairness testing into evaluation for systems that affect people: scoped with legal input, metrics chosen and documented, data assessed for representation and proxies, outcomes tested across groups before launch and monitored after, remediation re-tested, and methodology documented in the registry. The AI enablement practice provides the evaluation and monitoring platform, AI agents that touch consequential decisions keep humans in the decision, and forward deployed engineers work with your compliance and legal teams. The record behind the approach is 150+ projects with 99.9% uptime.

This checklist is general guidance, not legal advice. To scope a fairness audit, message FISTA on WhatsApp, or read the AI controls for financial services whitepaper for a regulated-sector application.