Accuracy, Validity, Completeness, and Usability Are Different Scores

The question

When an evaluation report says a model scored 87%, what was measured?

Usually the answer is accuracy, and usually accuracy is standing in for three other things it cannot represent: whether the test measured the intended capability (validity), whether the output addressed everything the task required (completeness), and whether anyone downstream could actually use the result (usability). These are four different scores answering four different questions. Reporting one while implying the others is not a rounding error. It is a category error that survives review because every individual number looks precise.

A 2024 review of LLM evaluation dimensions in healthcare found that accuracy and correctness appeared as criteria in 99.1% of reviewed studies, while completeness appeared in 18.5% and effectiveness/usability in 7.4% . The field measures what is easy to measure and then treats the easy measurement as the whole picture.

This post separates the four scores, states what each actually supports, and shows the trade-offs that make collapsing them into a single number a decision rather than a simplification.

The short answer

None of these is derivable from the others. A response can be accurate but incomplete, complete but invalidly measured, valid but unusable, or usable but inaccurate. The rest of this post is why.

Accuracy measures agreement, not truth

Accuracy is a match rate: the proportion of outputs that agree with a reference. That definition is clean, and it contains a hidden dependency — accuracy is only as good as the reference it is matched against.

A benchmark that relies on a single crowd-worker to label each item as true or false “risks embedding that worker’s errors and biases into the ground truth. If 5% of labels are incorrect, model scores will be systematically distorted” . The accuracy number is still precise. It is precise about the wrong target.

There is a second dependency. On multiple-choice benchmarks, random selection produces a non-trivial baseline accuracy, so a raw accuracy figure without a chance baseline overstates capability . The same arithmetic score means different things depending on the task format, and the format is rarely reported alongside the score.

The practical rule: report accuracy with its reference construction, its chance baseline, and its label quality, or do not report it as accuracy. A bare percentage is an agreement rate with an unstated artifact.

Validity is the meta-score, and it is not reliability

Validity asks whether the measurement measures the construct it claims to measure. It is the score that sits above the other three, and it is the one most often replaced by its weaker cousin, reliability.

The distinction matters because reliability evidence and validity evidence are different things. Agreement-based metrics, including most LLM-as-judge annotation benchmarks, are reliability evidence — they show that raters agree. “Validity requires an independent argument” . Two annotators agreeing on a label does not establish that the label measures the capability. It establishes that the label is consistently assigned.

The empirical record on validity is uncomfortable. An audit of ten popular agentic benchmarks found validity flaws in seven and reporting gaps in all ten . A study of 55 papers found that approximately 82% apply structurally mismatched, incomplete, or absent inter-rater reliability metrics . The same analysis formalizes the compounding:

Vtotal≤V1×V2×V3Vtotal≤V1×V2×V3

where V1V1 is validity at task generation, V2V2 at simulation, and V3V3 at judgment. A pipeline retaining 70% of valid signal at task generation, 80% at simulation, and 65% at judgment is at most 36% valid against the intended construct, with a bound spanning 0.22–0.54 across empirical estimates .

Read that against an accuracy number. A benchmark can report 87% accuracy and carry a construct validity bound of 0.22–0.54. The accuracy is a measurement of agreement. The validity bound is a statement about what the agreement means. Reporting the first without the second is not a summary. It is a substitution.

Does this mean accuracy is useless? No. It means accuracy is an outcome measurement, not a capability measurement. The field has known this since Cronbach and Meehl formalized construct validity in 1955 — a test is valid for a specific inference in a specific context, not in general. A benchmark score supports the inference "the model performs well on this task distribution as operationalized by this harness." It does not automatically support the inference "the model has the capability the benchmark is named after." The gap between those two inferences is the validity argument, and the argument is currently skipped in most reporting.

The 2026 survey literature frames a related structural limit as an evaluation trilemma: authenticity, scalability, and contamination resistance cannot all be maximized simultaneously in a single framework . A large, cheap, uncontaminated benchmark is hard to build; a realistic, scalable benchmark is hard to keep uncontaminated; a contaminated-resistant benchmark is hard to scale. Validity is not a property you add at the end. It is a constraint on which trade-offs the evaluation can afford.

Completeness is a coverage score, and it trades against conciseness

Completeness asks whether every required element of the task was addressed. It is not a subset of accuracy, and the two diverge in both directions.

A response that thoroughly answers one part of a multi-part question while ignoring the others “scores low on completeness even if what it does cover is accurate” . In the reverse direction, a complete response can contain inaccurate elements; coverage and correctness are independent axes.

Completeness also has an explicit trade-off that most evaluation frameworks acknowledge and most reports omit: completeness trades off against conciseness . A maximally complete answer to a simple question is verbose. A maximally concise answer is incomplete. The right operating point depends on query complexity, which means a completeness score without a stated complexity target is not interpretable.

The operationalization follows from the definition. Completeness is elements present divided by elements required, where the required set is defined before the evaluation, not inferred from the response . If the required set is defined after seeing the output, the score becomes a description of what the model did rather than a measure of what it should have done.

Usability is the score that decides whether the other three matter

Usability asks whether the output can be used for its intended purpose by its intended user. It is the most outcome-adjacent of the four scores, and the least measured.

The evidence for its neglect is quantitative. In the healthcare evaluation review, effectiveness/usability appeared in 7.4% of studies, against 99.1% for accuracy . In a capital-markets reliability framework that does include usability-adjacent dimensions, the scoring is explicit about the gap between “plausible” and “bankable” — decision usefulness and reviewability are scored as separate dimensions from factual accuracy .

Usability also has a concrete interaction with completeness that is easy to miss in aggregate scores. In retrieval-augmented systems, utilization and completeness are complementary diagnostics: high utilization (0.9) with low completeness (0.3) “indicates the system retrieves accurate but incomplete information” . Neither score alone identifies the failure. The pair does.

The general point: usability is not a soft metric appended to a hard one. It is the score that determines whether an accurate, valid, complete output changes any decision. An output that no one can act on has a usability score of zero regardless of its accuracy.

The failure cases these distinctions prevent

Accurate but invalid. A benchmark reports 85% accuracy. The construct validity bound is 0.3–0.5. The accuracy is a real agreement rate against a reference whose relationship to the intended capability is unestablished. Reporting the first without the second converts a bounded measurement into an unbounded claim.

Accurate but incomplete. A response correctly answers one part of a three-part question. Accuracy on the answered part is 100%. Completeness is 33%. A single score that weights accuracy heavily will report success on a response that failed the user’s actual request.

Complete but unusable. A response covers every required element, at a length and structure that no downstream process can consume. Completeness is high. Usability is low. The failure is invisible in a completeness score and visible immediately to the person who has to use the output.

Valid test, wrong target. A test is internally valid — it reliably measures what it measures, with good inter-rater agreement. What it measures is a proxy that diverges from the capability users need. This is the hardest failure to catch, because every local metric passes. Only the validity argument, made independently of the test, can catch it.

The compounding version. Task generation loses validity, simulation loses validity, judgment loses validity. The pipeline retains 22–54% of valid signal while reporting a single clean score . The score is not wrong. It is a measurement of a different quantity than the one the decision assumes.

The reporting discipline

The practical output of these distinctions is a reporting minimum. A score without the following is not citable:

json

{
  "primary_score": 0.87,
  "score_type": "accuracy",
  "reference_construction": "gold labels, single annotator, 5% estimated label error",
  "chance_baseline": 0.25,
  "construct_validity_argument": "present / absent",
  "validity_bound_estimate": [0.22, 0.54],
  "completeness": {
    "required_elements": 12,
    "covered_elements": 9,
    "score": 0.75,
    "complexity_target": "multi-part query, moderate"
  },
  "usability": {
    "task_success_rate": 0.0,
    "human_review_agreement": 0.0,
    "utilization": 0.0
  },
  "tradeoffs_stated": ["completeness vs conciseness", "accuracy vs validity bound"],
  "evidence_level": "documented method"
}

The fields that carry the most diagnostic weight are the ones most reports omit: chance_baseline, construct_validity_argument, and validity_bound_estimate. The first corrects accuracy for task format. The second states whether anyone made the independent validity argument. The third bounds what the accuracy can mean even when the argument is present.

completeness and usability are nested rather than flat because they are structurally different from accuracy: completeness is a coverage ratio against a pre-defined required set, and usability is an outcome measurement against a use case. Treating all four as scalar fields in one row would reproduce the collapsing error the post is about.

What this post does not prove

It does not prove that accuracy is a bad metric. It is a good metric for the question it answers — agreement with a reference — and that question is often the right first question.

It does not prove that every evaluation needs all four scores. A screening test for a low-stakes internal tool can legitimately report accuracy alone, provided it does not present that number as a capability claim.

It does not prove that validity can be established conclusively. The construct validity literature is clear that validity is an argument, not a proof, and the argument is always provisional and context-specific.

What it does establish is the specific error the post is aimed at: four scores answering four questions, reported as one, with the most decision-relevant ones omitted first. Accuracy survives the omission because it is easy. Validity, completeness, and usability are the ones that require a definition before the test and a sentence after it. The definition and the sentence are the work.

A working demo is evidence. It is not a production system. A high accuracy score is evidence of agreement. It is not evidence of validity, completeness, or usability — and those are the three that decide whether the agreement matters.

Reproduction notes

• Dimension frequency figures referenced: accuracy/correctness 99.1%, completeness 18.5%, effectiveness/usability 7.4%, from a review of LLM evaluation criteria in healthcare (Nature, 2024) .

• Accuracy reference-quality and chance-baseline caveats referenced from benchmark validity literature .

• Validity vs. reliability distinction and the “validity requires an independent argument” framing referenced from evaluation methodology literature .

• Compounding validity bound Vtotal≤V1×V2×V3Vtotal≤V1×V2×V3 and the 0.22–0.54 empirical range referenced from the measurement-without-validity analysis .

• Benchmark validity audit figures (7 of 10 with validity flaws, 10 of 10 with reporting gaps) and 82% IRR mismatch figure referenced from the same analysis .

• Evaluation trilemma (authenticity, scalability, contamination resistance) referenced from the 2026 LLM evaluation survey taxonomy .

• Completeness–conciseness trade-off and multi-part question example referenced from LLM evaluation criteria guidance .

• Utilization–completeness diagnostic pair referenced from retrieval evaluation guidance .

• Capital-markets reliability dimensions (factual accuracy, evidence traceability, decision usefulness, reviewability) referenced for the usability-adjacent scoring precedent .

The next post in this series applies these distinctions to a concrete benchmark artifact: what a single benchmark row can and cannot support when the validity argument, the completeness definition, and the usability measurement are all absent from the table.

For AI tool features, rules and availability, refer to their official sites.