AI Training Data Checklist

Every model, whether a classifier trained in-house or a language model evaluated against a golden set, inherits the flaws in its data: unrepresentative samples, inconsistent labels, leaked future information, and personal data nobody had the right to use. This checklist covers what to verify before data is used for training or evaluation. It complements what is training data, what is data labeling, and what is a golden dataset.

Who should use this checklist?

Data scientists and engineers assembling datasets, domain experts labeling them, and governance functions reviewing model development.

Is provenance documented and are rights clear?

1. Source of every record or document is recorded.
2. Usage rights are confirmed: internal data policy, licenses, contracts, consent where required.
3. Third-party and scraped data has a documented legal basis and license compliance.
4. Synthetic data is labeled as such with its generation method.
5. Collection dates and conditions are recorded.

Reference: ai data governance.

Is the data representative?

Check	Evidence
Population compared to production population on key attributes	Distribution comparison
Conditions covered: seasons, channels, regions, device types, document layouts	Coverage analysis
Rare but important classes present in sufficient numbers	Class counts
Known biases in collection identified and addressed	Bias assessment
Drift between collection period and deployment considered	Analysis

Reference: ai bias and fairness.

Is labeling quality measured?

1. Guidelines with definitions and examples exist and are versioned.
2. Annotators are qualified for the domain.
3. Overlap sample labeled by multiple annotators; inter-annotator agreement computed and acceptable.
4. Disagreements adjudicated and guidelines refined.
5. Audits of labels on a schedule.
6. Label provenance: who labeled what, when, under which guideline version.

Reference: the AI evaluation and testing whitepaper.

Is leakage prevented?

1. Splits by time where the model predicts the future; by entity where records cluster.
2. No duplicates or near-duplicates across training, validation, and test sets.
3. Features computed as of the prediction time; no post-outcome information.
4. Test set never used for tuning; a held-out set is reserved.
5. Performance validated on a later period.

Reference: how to build a feature store.

Is personal and sensitive data handled per policy?

1. Classification of the dataset by sensitivity.
2. Legal basis and minimization for personal data; anonymization or pseudonymization where feasible.
3. Sensitive attributes handled deliberately: excluded, or retained only for fairness testing under controls.
4. Access control and retention for the dataset.
5. Memorization risk considered for generative models trained on the data.

Reference: ai data privacy compliance and what is differential privacy.

Is the data clean and consistent?

1. Schema validated; types and formats consistent.
2. Missing values understood and handled deliberately.
3. Outliers and errors investigated, not silently dropped.
4. Duplicates removed.
5. Preprocessing documented and reproducible.

Reference: how to prepare data for ai.

Are splits and versions fixed?

1. Dataset versions immutable with identifiers.
2. Splits recorded and reproducible.
3. Lineage from raw sources through preprocessing to the versioned dataset.
4. Model versions reference dataset versions in the registry.

Reference: how to build a model registry.

Is the dataset documented?

1. A datasheet or data card covering purpose, composition, collection, preprocessing, uses, limitations, and maintenance.
2. Known gaps and biases stated.
3. Contact and ownership.
4. Change history.

Reference: what is a model card.

Is security addressed?

1. Poisoning risk assessed for data from external or untrusted sources.
2. Integrity checks on datasets.
3. Access logging.

Reference: what is data poisoning.

How should gaps be handled?

Rights, privacy, and leakage gaps block use of the data. Representativeness and labeling gaps block production use of models trained on it. Documentation and versioning gaps are closed before the model is registered.

Worked example: a claims triage golden set

A claims team assembles a golden set of historical claims for a triage agent. Provenance is recorded per claim with the policy basis for internal use. Representation is checked against the current claim mix by line, region, and severity, and rare high-severity classes are oversampled deliberately and labeled as such. Two experienced adjusters label an overlapping sample; agreement is measured, disagreements on complexity definitions are adjudicated, and the guideline is revised before the rest is labeled. Splits are by time so the test period follows the training period, and duplicate claims from resubmissions are removed across splits. Claimant personal data is pseudonymized, and the dataset is versioned and documented with a datasheet before the first evaluation run.

How FISTA Solutions handles training and evaluation data

FISTA Solutions applies this checklist to every dataset it assembles or evaluates against: documented provenance and rights, representativeness checked against production, expert labeling with measured agreement, time- and entity-aware splits with held-out sets, privacy handling, reproducible versions linked in the registry, and datasheets for reviewers. The AI enablement practice delivers the data and evaluation platform, AI agents are evaluated on golden datasets built this way, and forward deployed engineers work with your domain experts on labeling. The record behind the approach is 150+ projects with 99.9% uptime.

To assess a dataset against this checklist, message FISTA on WhatsApp, or read the ai data readiness checklist for the upstream source assessment.