Psychometric equivalence specification

Section 01The claim, stated so it can be tested

Claim. For any two candidates i and j of identical ability θ, sitting two different generated forms of the same examination, the expected score difference attributable to form is smaller than the measurement error of the instrument by at least an order of magnitude.

This is deliberately a statement about expected scores on the ability scale, not about surface similarity. Two papers that look different are not necessarily unequal, and two that look identical are not necessarily equal. Only the item parameters settle it.

Section 02The measurement model

Every item is described by a three-parameter logistic model. The probability that a candidate of ability θ answers item k correctly is:

Two items are psychometrically equivalent when their (a, b, c) triples coincide within tolerance and their expected response times coincide. Everything else about an item — its wording, its names, its numbers — is free to move.

Equivalence of difficulty is not enough on its own. An item that is equally hard but less discriminating measures less, and a paper built from such variants would be quietly less reliable. So the same tolerance is applied to information:

Section 03Radicals and incidentals

The generation rule comes from item-generation theory, not from prompt engineering. Every feature of an item is classified as a radical — a feature that drives difficulty — or an incidental — a feature that does not. Generation may only move incidentals.

An item template is therefore not a prompt. It is a typed object: declared parameters with ranges and units, a closed-form solution, a misconception map for distractors, and an explicit list of transforms that provably preserve the answer. The model's job is confined to producing fluent natural language over that structure.

Section 04Cold-start: parameters for an item nobody has ever sat

The obvious objection: IRT parameters are normally estimated from response data, and a variant generated four seconds ago has none. Four mechanisms answer this, in the order they are applied.

1. Family-level priors (hierarchical IRT). Variants of one template are not independent items; they are siblings drawn from an item family. The family has its own parameter distribution, estimated from every sibling ever administered. A new variant inherits the family posterior as its prior — so it starts with an informative estimate, not a guess. Formally, b_k ~ N(μ_family, σ²_family), and the family's σ is itself an object of study: a family with a wide σ is a badly specified template and is sent back to its author.
2. Feature-based prediction (LLTM-style). Difficulty is regressed on the item's declared radicals — number of solution steps, operation type, unit conversions, distractor structure, readability. Because generation cannot move radicals, the predicted difficulty of a variant is by construction the predicted difficulty of its parent. This model is fitted on the whole calibrated bank, not on one family, and its residual standard error is reported with every release.
3. Seeding on a non-scoring cohort. New families enter live papers as unscored seed items first — the standard practice for pre-testing — accumulating real response data before any candidate's result depends on them. A family is not released for scoring until its parameters are estimated from at least 1,200 responses per variant class.
4. Online re-estimation and retrospective correction. During delivery, responses update the family posterior continuously. If a variant's realised difficulty drifts outside tolerance mid-administration, it is withdrawn from further delivery and every candidate who already received it is re-scored on the corrected parameters — upward or downward — before results are released.

Section 05Where ±0.05 logits comes from

The tolerance is not a round number chosen for comfort. It is derived from the effect a difficulty shift can have on a candidate's expected score, and bounded so that effect stays far below the test's own measurement error.

The steepest point of the ICC bounds the damage a difficulty error can do:

Aggregated over a 60-item paper, three quantities matter, and the specification bounds all three:

^† SEM = SD × √(1 − reliability). The comparison is the point: a candidate's score already carries about ±3.2 marks of irreducible measurement noise. The form they were given contributes under a tenth of one mark of that.

Two rules follow, and both are enforced by the calibrator rather than by policy:

• Per-item: reject any variant with |Δb| > 0.05, |Δa| > 0.01, |Δc| > 0 or |Δt̄| > 4 s.
• Per-form: reject any assembled form whose mean signed difficulty deviation exceeds 0.008 logits. This is what stops many small permitted errors from stacking in one direction.

Section 06Form-level equivalence

Item-level parity is necessary but not sufficient; candidates sit forms, not items. The test characteristic curve — expected raw score as a function of ability — is the form-level statement of fairness, and it is checked for every assembled form before release.

Assembly constraints applied to every form, in addition to the parity rules:

• Target information function. Every form must match the reference form's information curve within 3% at each of nine ability anchor points, so precision is equal where cut-scores fall.
• Content blueprint. Topic, sub-topic and cognitive-level counts are fixed by the blueprint and cannot be traded against difficulty.
• Answer-key balance. Key positions are balanced across the cohort so option order carries no signal.
• Time model. The sum of expected solve times must fall within ±90 seconds of the reference form.

Section 07Linking and scale maintenance

Scores are reported on a stable scale across shifts, sessions and years. Three mechanisms hold that scale in place.

Where shifts must be compared — as in any multi-session national examination — equating is performed on the ability metric, not by percentile normalisation of raw scores. This matters: percentile normalisation across shifts of unequal ability composition is the source of much of the public distrust in existing multi-shift examinations, and IRT linking removes the need for it.

Section 08Exposure control

Generation does not by itself solve exposure. If one template's variants are drawn far more often than another's, that template becomes worth memorising. Exposure is therefore managed explicitly.

• Per-template exposure ceiling. No template may exceed a target proportion of the cohort, enforced by the sampler at draw time.
• No repeats within a cohort. Two candidates in the same administration never receive the identical parameter tuple from the same template.
• Family retirement. Cumulative exposure across administrations is tracked; families are retired on a schedule and replaced from the authoring pipeline.
• Late-arrival parity. A candidate starting twenty minutes late is drawn from the same live pool under the same ceilings — there is no leftover paper and no advantage to arriving late.

Section 09Differential item functioning

Parity in aggregate can still hide unfairness for a subgroup — a generated variant set in an unfamiliar context may be harder for rural candidates at the same ability, or a Hindi rendering may not be of equal difficulty to its English source. DIF analysis is run after every administration and its results are published, not filed.

• Method. Mantel–Haenszel with ETS A/B/C classification for flagging, plus IRT-based Δb comparison for magnitude, matched on ability rather than raw score.
• Contrasts. Language of delivery, urban/rural, gender, device tier, and state. Device tier is included because it is the contrast most specific to remote delivery and the one a centre-based exam never had to consider.
• Consequence. A family showing C-level DIF is withdrawn and its authors notified. Candidates who received the affected variant are re-scored without it.
• Translation. Hindi and English renderings are calibrated as separate items in the same family, never assumed equal because they are translations of one another.

Section 10Validation protocol and limitations

Before an examination

1. Family seeding on a non-scoring cohort, minimum 1,200 responses per variant class.
2. Solver verification of every template transform, including adversarial parameter values at range boundaries.
3. Cognitive-lab review of a random variant sample by subject experts, blind to which is the parent item.
4. Simulated cohort run: 100,000 synthetic candidates across the ability range, checking recovery of true θ under randomised form assignment.

After an examination

1. Concurrent calibration of all delivered variants; comparison of realised against predicted parameters.
2. Regression of score on variant identity, conditioning on θ. The coefficient must be indistinguishable from zero. This is the falsification test in §1.
3. Full DIF sweep across the five contrasts, published with the results.
4. Publication of every delivered variant and its parameters, as answer keys are published today.

Limitations, stated plainly