What vetting actually predicts: assessment scores against on-the-job performance

An assessment is a claim about the future. Ours says: this engineer will do good work on a team they have not met, in a codebase they have not seen. That is a bold claim to make on the strength of a few hours of evidence, and it deserves to be checked against what actually happened.

This report is that check. We took every engineer placed through Engagetal since we started operating in August 2025 who had reached a six-month review by April 2026, 412 people, and compared their assessment scores against how they were actually rated by the people they worked for.

The headline is honest but not flattering

The composite vetting score correlates with six-month performance at r = 0.61. In the context of personnel selection that is a strong result; published meta-analyses put structured interviews around 0.42 and unstructured interviews closer to 0.20. We are doing better than the industry default.

We are also explaining about 37% of the variance in outcomes, which means 63% of what determines whether an engineer succeeds is something we cannot see before they start. Team fit, manager quality, the state of the codebase, what is happening in someone's life that year. None of this is in our data, and all of it matters. Anyone selling a hiring signal that claims to be more decisive than this is either measuring something narrower or not checking.

Writing predicts more than puzzles

Figure 1 breaks the composite into its parts. The strongest single predictor was written reasoning. A component where candidates are given an ambiguous technical problem and asked to write a short design memo explaining their approach and its trade-offs. It correlated at r = 0.54 raw and held at 0.33 when every other component was controlled for.

System design and the work sample followed. Timed algorithmic puzzles came second-to-last raw, and collapsed to a partial correlation of 0.04 once the other components were in the model. In plain terms: the algorithm score told us almost nothing that the design memo and the work sample had not already told us.

This is not a novel finding. It echoes what a lot of engineering leaders have argued from experience for a decade, but we had been running the puzzles anyway, on the reasonable-sounding grounds that they were cheap and standardised. Cheap and standardised is not the same as informative. We removed timed algorithmic puzzles from the assessment in March 2026. Candidates get 45 minutes of their life back and our predictive power is unchanged.

Sixty-three percent of what determines whether an engineer succeeds is something we cannot see before they start.

Section 1: The headline is honest but not flattering

Where our bar is wrong

Figure 2 is the chart that changed how we think about the threshold. Engineers scoring 70-79, the band immediately below our launch bar of 80, landed in the top performance quartile 33% of the time. That is not far off the 41% we see from the 80-89 band, and it is dramatically better than the 9% from 60-69.

There is a sharp discontinuity in outcomes between the 60s and the 70s, and a much softer one between the 70s and the 80s. Our bar was sitting in the soft part. We were rejecting a group of engineers a third of whom would have been top-quartile performers, and we were doing it for a gain in precision that the data does not really support.

We now admit into the pool at 72 with a flag for additional reference checking between 72 and 79, rather than a hard cut at 80. We will report on whether that was the right call in next year's edition, including if it turns out badly.

Prestige adds nothing

We tested university tier, previous employer brand, and referral source as additional predictors on top of the work-sample evidence. All three added no meaningful power, the model's adjusted R² moved by less than 0.01 in every case.

They do correlate with performance on their own, which is why people believe in them. But that correlation runs almost entirely through the work-sample evidence: engineers from well-known companies tend to write better design memos, and once you have read the memo, knowing where they worked adds nothing. We do not collect employer prestige as a scored input, and this is the evidence for why.

Table 1: Assessment components, predictive power and current weighting.

What we are still getting wrong

Two things bother us about this analysis and we would rather say them than have someone else find them.

First, manager ratings are a flawed criterion. They are unblinded, they correlate with likeability, and they are known to carry demographic bias. We use a composite that includes peer review and delivery consistency to dilute this, but we have not solved it. Second, we can only measure engineers we placed. The engineers we rejected might have done brilliantly, and we would never know. The classic range-restriction problem, which almost certainly means our true correlations are somewhat higher than reported and our confidence in the bar is somewhat lower than it looks.