Not all skills prove the same way: the four evidence shapes

Proof is not one thing

It is tempting to talk about "proving a skill" as if it were a single, uniform act — run the assessment, get the evidence, done. Assessment science says otherwise. The evidence that proves one skill is the wrong instrument for another, and the difference is not a matter of difficulty settings. It is structural. Skills differ in the shape of the evidence they produce, and an assessment that ignores that is measuring the wrong thing confidently.

The clearest way to see this is to put two sales skills side by side. Objection-handling and negotiation sound like neighbours — both are conversational, both are commercial, both live in the same deal. But they prove out in completely different ways, and building an honest measure of each forces you to confront the whole landscape of what evidence can and cannot do.

Objection-handling: proof in a single episode

Objection-handling is episodic. It happens in a bounded moment — a prospect raises a concern, and the seller either addresses it well or does not. You can see the whole thing in one call, sometimes in one exchange. The evidence is self-contained: the objection, the response, the outcome of that moment. Score enough of these episodes and you can be confident about the skill in a couple of weeks, because each episode is a near-complete observation on its own.

This is the friendliest evidence shape there is. It is frequent, it is single-source, and the causal link between the behaviour and its immediate result is short and legible. If you want a skill to prove out quickly and cleanly, pick one that lives in single episodes.

Negotiation: proof only over the whole arc

Negotiation is the opposite. It is longitudinal, and its proof is entangled with an outcome that arrives much later. A single negotiation exchange tells you very little — the concession that looks weak in the moment may be the setup that closes the deal a month later; the firm stance that looks strong may be what kills it. You cannot score negotiation from a snapshot. You have to watch the whole deal arc and then look at how it landed.

And here is the part that makes it genuinely hard: even with the full arc and the outcome, the causal attribution is fragile.

Did the deal close because of how they negotiated — or despite it, because the product was the only real option, or because a competitor stumbled, or because procurement had already decided? The outcome is real. Its cause is contested.

Negotiation needs a deal arc plus the commercial outcome plus corroboration, gathered over something like a quarter, and even then the honest confidence is lower than for objection-handling. This is not a flaw in the measurement. It is the true shape of the evidence, and pretending otherwise — scoring negotiation off a single role-play as if it were objection-handling — produces a number that looks the same and means far less.

The four evidence shapes

Generalise from those two and a taxonomy falls out. Almost every skill sits in one of four evidence shapes, and the shape predicts how hard it is to prove, how long it takes, and how much you should trust the result.

Episodic-behavioural is the easy end: build there first if you can. Longitudinal-outcome is hard because of the time lag and attribution problem. Procedural-compliance is medium — it is objective, but only if a trustworthy record exists. Rare-artifact is hardest of all: the evidence is the most valuable and the least available, because you cannot summon a genuine crisis to observe how someone leads through one.

What assessment science adds on top

Knowing the shape is necessary but not sufficient. Turning evidence into a defensible level takes the parts of assessment science that vendors rarely talk about, because they are unglamorous and expensive.

Calibration is the real bottleneck

A rubric that scores a skill is only as good as the human-labelled examples it was calibrated against — on the order of fifty to a hundred gold-standard cases per skill, per industry, graded by subject-matter experts across the full range of levels. That labelled set, not the model or the compute, is the scarce resource. It is what lets you claim a score means what it says.

Inter-rater agreement is the ceiling

Here is a humbling number: if two expert humans grading the same evidence only agree with each other seventy per cent of the time, no automated system can be more reliable than that. Human agreement is the ceiling on machine accuracy. For skills where experts genuinely disagree, the honest label is "indicative only" — useful for coaching, never for a high-stakes gate.

The rubric is not universal

The same behaviour can be excellent in one context and a violation in another. A software seller who pushes hard to expand a deal is demonstrating strong commercial skill. A pharmaceutical medical liaison who does the same thing has committed a compliance breach — they are not allowed to promote off-label. A rubric that scores "assertive selling" the same way in both industries is not calibrated; it is dangerous.

Evidence is a leading indicator, not an oracle

Put these together and the right posture becomes clear. A skill proof — especially for the harder evidence shapes — is a corroborated, human-reviewable indicator, not an infallible verdict. It aggregates signal, states its confidence, and knows what it does not know. Two failure modes matter most: over-claiming from partial evidence, and treating the absence of evidence as proof of a deficiency. Someone who has not been observed negotiating a hard deal has not been shown to be bad at it — they have simply not been measured yet. An honest system says "unmeasured," not "weak."

What this means if you are buying or building

• Ask what evidence shape the vendor is actually capturing. A single role-play scoring negotiation is capturing an episodic proxy for a longitudinal skill. That gap is where confidence quietly leaks away.
• Expect different timelines for different skills. Objection-handling can prove out in weeks; negotiation needs a quarter and an outcome. A tool that promises the same confidence for both, at the same speed, is overselling one of them.
• Demand a confidence level, and honesty about attribution. Especially for outcome-linked skills, the right answer includes "and here is why we are only this sure."
• Insist on per-context calibration. The rubric that scores your industry must be calibrated for your industry, not borrowed from a generic template.
• Keep humans on the high-stakes shapes. The harder the evidence shape and the bigger the decision, the more a person needs to stay in the loop.