Scoring people fairly: notes from a ranking system we shipped

We built a selection system that ranks candidates for a fully funded education programme. Roughly a thousand applicants, a few dozen places, and a review panel that has to be able to defend every decision. The engineering was not the hard part. Deciding what the score should mean was.

The brief

Select on potential rather than on current attainment, for applicants aged sixteen to twenty-two across a country with very uneven access to preparation. A grant covering tuition and accommodation is at stake, so a wrong ranking is not a minor inconvenience. It changes someone's trajectory.

The obvious system is a weighted formula over grades, olympiad results, and the essay. We built something close to that first, and it was quietly unfair.

The first design was wrong

A weighted attainment score measures a snapshot. A seventeen-year-old in a large city with tutors and competition access arrives at that snapshot from a different starting point than a seventeen-year-old in a rural district with neither. Ranking on the snapshot mostly ranks the starting point, and does it with the authority of a number.

The system should not measure where a candidate is. It should measure how far they have come from where they started, and how fast they are still moving.

That sentence changed the features, not the algorithm. The model stayed a gradient-boosted tree with five-fold cross-validation; it was the inputs that had to be rebuilt.

Measuring distance travelled

What we ended up feeding it, in three groups:

• Structured intake fields collected through a plain state-machine chat bot: no model in the loop, so the questions are identical for everyone and the answers are comparable.
• Context-relative signals: attainment expressed against the opportunity available locally, so founding a project and organising forty people where nothing existed reads as strongly as it should.
• Leadership scenarios: a short set of branching situations with a response timer, scored against a published leadership-behaviour instrument rather than our own intuition.

A candidate with a 4.2 average who built something from nothing in a small town can now outrank a 4.8 average whose activities were all provided. That is not the model being generous. It is the model finally measuring the thing the programme said it cared about.

Text that a model wrote

Any essay prompt in 2025 is answered partly by language models, and detectors are not reliable enough to make a decision on. So we stopped trying to detect and changed the question instead.

Prompts anchor on episodic memory: a specific occasion, who else was there, what you decided in the moment, what you would do differently. A model writes fluent, structurally perfect prose about such a moment and it reads as generic, because the specifics are not available to it. We score for concreteness rather than polish. It is not airtight. It is considerably better than a detector we cannot justify to a rejected applicant.

The panel decides

Concretely, this means the review interface shows, for every candidate: the score, the signals that moved it most in each direction, the raw intake answers, and a free-text field for the reviewer's own reasoning. Overrides are expected, recorded, and fed back into evaluation: a pattern of overrides in one direction is evidence the features are wrong, and we want to see it.

It also means the system is defensible. When someone asks why a candidate placed where they did, the answer is a list of signals and a named human decision, not "the algorithm".

What we would keep

Three things transferred to how we approach every scoring system since:

1. Write the fairness question before the feature list. "What would make this unfair?" is a design input, not a compliance review at the end.
2. Keep the intake deterministic. If the questions differ per applicant, nothing downstream is comparable, however good the model is.
3. Instrument the humans, not just the model. Override patterns were the most useful accuracy signal we had, and they cost almost nothing to capture.

If you are building something that ranks people, we are glad to talk about it: info@rdrightnow.com. It is the category of work we are most careful about, and the one where a second opinion is worth most.

---

Questions or disagreement are welcome: info@rdrightnow.com. We would rather be corrected than quoted.