The 88% That Describes a Population, Not a Death

Two numbers from one Ugandan dataset — 88% population-level accuracy and 40.5% individual agreement — explain what CSMF accuracy actually measures, and why the metric's authors asked for a second number alongside it.

verbal autopsyInterVAvalidationcause of death

A comparison published in April in PLOS Global Public Health ran two computer-coded methods over 1,564 adult verbal autopsies from two Ugandan surveillance cohorts — 1,037 deaths in Rakai, 527 in Kyamulibwa — and scored them against physician review. Read as descriptions of the population, the algorithms did well: the InterVA model reached 88% (95% CI 85–90) and InSilicoVA 83% (79–87). Read as descriptions of individual deaths, the same two methods matched the physicians 40.5% and 38.2% of the time.

Both figures are correct. They measure different things, and the distance between them is where most arguments about verbal autopsy “accuracy” go wrong.

A metric built around a decision

The population-level figure is called CSMF accuracy — cause-specific mortality fraction accuracy — and it was defined in 2011 by Christopher Murray, Rafael Lozano, Abraham Flaxman, Alireza Vahdatpour and Alan Lopez in Population Health Metrics, in the middle of the Population Health Metrics Research Consortium’s validation programme. The consortium had a practical problem: validation studies were multiplying across methods, and they were being compared with statistics that did not measure the same thing.

The paper’s central move is a normative argument, made explicitly. When a method misallocates deaths between causes, what harm follows? If programme budgets and political attention scale to the size of a problem, then the harm tracks the absolute error in a cause fraction, not the relative one. Underestimating a cause by a hundred deaths misdirects resources whether that cause was large or small. So the metric sums absolute differences between predicted and reference cause fractions, normalises the total against the worst possible assignment, and reports a number between zero and one.

They were also explicit about what they were not using. Cohen’s kappa, then common in the literature, was rejected on the grounds that it summarises individual assignment rather than cause fractions, and that “CSMF errors of near zero are possible with kappa values that are less than 0.1”. Instead the paper asks for a pair of numbers: chance-corrected concordance, computed cause by cause, for how well individuals are classified; and CSMF accuracy for how well the distribution is reproduced. “Different users are likely to attach different importance to these dimensions,” they wrote.

The arithmetic has a shape

Because the sum is of absolute differences, one percentage point of error costs the same whether it moves a cause from 15% to 16% or doubles a cause from 1% to 2%. The consequence is structural: the metric is dominated by the large causes, and a method that mishandles a rare cause can still post a respectable score.

The same 2011 paper shows why this is not merely hypothetical. Plotting estimated against true fractions cause by cause, most methods produce slopes below one — they overestimate small causes and underestimate large ones — with intercepts that stay above zero even when a cause is absent from the data. A method, in other words, tends to leave a faint smear of deaths across the whole cause list.

Composition matters too. The number depends on which causes happen to be frequent in the test dataset, so Murray and colleagues recommended resampling cause compositions from an uninformative Dirichlet distribution and reporting the median CSMF accuracy across those draws, plus a full classification matrix on demand. A single figure computed from a single dataset — the form in which the statistic is most often quoted — is the form the metric’s designers argued against.

Where individual assignment goes to hide

The Ugandan comparison gives the divergence a clean shape. At the fourteen-category WHO cause list, the ranking even reverses: InSilicoVA matched physician review 44.4% of the time against InterVA-5’s 41.0%. Which method looks better depends on how finely you slice the causes, and neither number tells you whether the deaths that were misassigned were misassigned harmfully.

A 2026 analysis of CHAMPS data makes the harm visible. Pramanik and colleagues estimated misclassification matrices for three computer-coded algorithms against minimally invasive tissue sampling and expert panel determinations across eight jurisdictions, and found sensitivities ranging from 62–85% for prematurity down to 12–22% for sepsis, meningitis and neonatal infection. In a passage that deserves to be quoted more often: of 61 neonatal deaths that the CHAMPS panel attributed to sepsis, meningitis or infection, 37 met the Expert Algorithm’s own clinical criteria for sepsis, and two were assigned it. Between 31% and 44% of those deaths were labelled prematurity instead. A cause fraction can look approximately right in aggregate while the named cause is close to invisible at the level of the death — the full analysis is worth reading alongside the metric.

The reference standard is a measurement too

Every one of these numbers is a comparison against something. In Uganda that something was physician review, which the authors describe plainly as “taken as the gold standard in the absence of hospital-based medical certification” — an assumption, not a finding.

How much the assumption costs is an empirical question, and autopsy series answer it. In Manaus, a retrospective review of 101 adult HIV deaths that came to autopsy found ante-mortem and post-mortem diagnoses agreed on the underlying cause in 36.7% of cases; 47.6% were Goldman class I, meaning a major diagnosis was missed that could have changed treatment. That is the clinical chart against the tissue. Verbal autopsy is compared against the chart.

Murray and colleagues reached the same conclusion from the other direction, in a sentence that has aged well: in the absence of rigorous gold standards, chance-corrected concordance and CSMF accuracy “will remain only measures of similarity between two imperfect assessments of cause of death”.

Keeping the pair together

None of this makes the 88% wrong. For the use it was built for — national cause fractions feeding burden-of-disease estimates, programme planning, the relative size of malaria against meningitis — a distribution-level metric is the right instrument, and the Ugandan comparison of the two paths is a model of reporting both levels honestly.

But case-level work asks a different question. Maternal and perinatal death reviews, child death review panels, any process that ends with a recommendation about a specific death, is consuming individual assignments, and for those decisions the relevant number is chance-corrected concordance, not CSMF accuracy.

Three questions are worth asking of any accuracy figure quoted in isolation. Which reference standard produced it, and what is known about that standard’s own error? How many causes were in the list — the Ugandan ranking flipped at fourteen categories? And was the number a median across resampled cause compositions, or the output of one convenient dataset?

The answers do not usually invalidate the claim. They tell you which claim is being made.

Sources

  1. Murray CJL, Lozano R, Flaxman AD, Vahdatpour A, Lopez AD. Robust metrics for assessing the performance of different verbal autopsy cause assignment methods in validation studies. Population Health Metrics 2011;9:28
    pmc.ncbi.nlm.nih.gov
  2. Nabukalu D, Lutalo T, Mugisha J, et al. A comparison of verbal autopsy assignment methods to obtain adult cause-specific mortality in two longitudinal studies in Rakai and Kalungu districts of South Central, Uganda. PLOS Global Public Health 2026;6(4):e0006223
    pmc.ncbi.nlm.nih.gov
  3. Pramanik S, Wilson E, Kalter HD, et al. Country-specific estimates of misclassification rates of computer-coded verbal autopsy algorithms. BMJ Global Health 2026;11(3):e021747
    pmc.ncbi.nlm.nih.gov
  4. Byass P, Hussain-Alkhateeb L, D'Ambruoso L, et al. An integrated approach to processing WHO-2016 verbal autopsy data: the InterVA-5 model. BMC Medicine 2019;17:102
    pmc.ncbi.nlm.nih.gov
  5. Martins YF, Almeida TVR, Reis MFD, et al. Clinicopathological diagnostic concordance among autopsied adults living with HIV in the Brazilian Amazon: a retrospective cross-sectional study. PLOS One 2026;21(10):e0361097
    pmc.ncbi.nlm.nih.gov

← Back to the news digest