October 18, 2026 interva.net Editorial

Told How Wrong They Are

Three computer-coded verbal autopsy algorithms named neonatal sepsis correctly in only 12–22% of cases. A new calibration framework turns that failure into an explicit, country-specific correction — and in Mozambique it moved the leading cause of newborn death from prematurity to infection.

verbal autopsyInterVAvalidationdata quality

Take the pooled sensitivity of three widely used computer-coded verbal autopsy algorithms for neonatal sepsis, meningitis and infection, as measured against autopsies and clinical records from eight countries: it runs between 12% and 22%. Read that as a failure rate and the sentence becomes uncomfortable — for every ten newborn deaths caused by infection, the algorithms named the right cause once or twice. The rest were filed under prematurity, pneumonia, intrapartum events, congenital malformation.

Now hold that beside a second number from the same paper. When those algorithms were applied to 1,192 neonatal deaths recorded by community surveillance in Mozambique, and then corrected using error estimates specific to that country, the ensemble of all three put 58% of deaths on sepsis, meningitis and infection (95% credible interval 47 to 68). Before the correction, the same data put roughly a quarter of deaths there and made prematurity the leading cause.

Both figures come from a study by Sandipan Pramanik, Emily Wilson, Henry Kalter and colleagues at Johns Hopkins, the US Centers for Disease Control and Prevention, Emory and Mozambique’s Instituto Nacional de Saúde, published in BMJ Global Health in March and funded by the Gates Foundation and a National Institutes of Health career award. It is a modelling study built on surveillance data, not a field trial or a validation exercise in the usual sense, and it does not test whether any algorithm is “better”. What it does is make a specific claim about what a verbal autopsy number is worth — and where the error inside it can be made to work for the estimate rather than against it.

A reference standard made of tissue and a panel

The paired data come from CHAMPS, the Child Health and Mortality Prevention Surveillance network, which investigates deaths of children under five and stillbirths at sites in Bangladesh, Ethiopia, Kenya, Mali, Mozambique, Sierra Leone and South Africa. Its teams are notified of a death within 24 hours, seek consent from parents or guardians, and then assemble three things: pre-mortem clinical and laboratory records, a verbal autopsy interview with the caregivers, and minimally invasive tissue sampling with microbiological and histopathological testing, a process that typically takes months. A multidisciplinary panel then decides the cause, including the chain of conditions leading to death. The study takes the panel’s primary cause as its reference — the authors call it a “gold standard”, though its own uncertainty is exactly why they treat the comparison as a matrix rather than a verdict.

Against that reference, the team estimated how each algorithm confuses each cause with every other, for neonates aged 0–27 days and children aged 1–59 months, using 1,379 neonatal records and 1,080 child deaths collected between December 2016 and June 2023. The three algorithms are an odd trio by design: the Expert Algorithm (a deterministic rule hierarchy, the same family of coding used in several national programmes), InSilicoVA and InterVA, both Bayesian. Two age groups, three algorithms, seven named countries plus a pooled “other” category.

The yardstick nobody quotes

Verbal autopsy has always been judged by two different rulers, and the gap between them is where most public argument about the method goes wrong. The first is individual accuracy: did this algorithm name the right cause for this death? On that ruler the numbers in this paper are poor. Pooled sensitivity was highest for prematurity, at 62–85% depending on the algorithm, and lowest for sepsis and for the residual “other” category, at 5–10%. Between 31% and 44% of infection deaths were labelled prematurity; between 12% and 39% of congenital malformation deaths met the same fate.

The second ruler is population accuracy: does the set of assigned causes add up to the right distribution of deaths? A method can be mediocre on the first and serviceable on the second, provided its mistakes are patterned rather than random and the resulting confusion matrix can be inverted. That is not a loophole; it is the arithmetic that made verbal autopsy useful in the first place, and it is the reason method comparisons have reported cause-specific mortality fraction accuracy alongside individual sensitivity since at least the 2014 comparison by Murray and colleagues.

What this paper adds is a diagnosis of why the correction has been too crude. The existing calibration framework pooled CHAMPS data across countries, implicitly assuming that an algorithm misreads symptoms the same way everywhere. It does not. Replacing the pooled model with country-specific estimates cut the average absolute error against observed rates by 35%, 38% and 34% for the three algorithms among neonates, and by 19%, 24% and 13% among children. For the Expert Algorithm, sensitivity to neonatal sepsis ranged from 6% in South Africa to 31% in Kenya. Of 61 sepsis deaths in the CHAMPS data, 37 met the algorithm’s own clinical criteria for sepsis, and two were assigned it: the other 35 went to causes sitting higher in its rule hierarchy — 13 to prematurity, 12 to pneumonia, 8 to intrapartum events.

Some of the country variation is not algorithmic at all. The rule that catches intrapartum-related deaths depends on a single interview item, id10106: how many minutes after birth the baby first cried. That answer was missing for 7 of 20 relevant deaths in Kenya and 11 of 17 in South Africa, pushing those deaths down the hierarchy into prematurity. An interviewer’s skipped field becomes, at the far end of the pipeline, a national cause-of-death profile. It is the same point made by the four real-time data-quality indicators we covered earlier this month, arriving from the opposite direction: the arithmetic of a VA estimate is only as good as the record underneath it.

The Mozambique test

The demonstration uses COMSA, the Countrywide Mortality Surveillance for Action project, which collects pregnancy and mortality data through routine community surveillance in 700 clusters of about 300 households each. Deaths are notified in the community and followed by caregiver interviews using an integrated verbal and social autopsy questionnaire based on the 2016 WHO instrument — the generation before the current 2022 WHO verbal autopsy instrument, but close enough in structure to share the same symptom vocabulary. The analysis covers 1,192 neonatal and 2,812 child records from January 2018 to December 2023.

Uncorrected, the algorithms disagreed about what kills newborns in Mozambique. The Expert Algorithm and InSilicoVA put infection first; InterVA and the ensemble put prematurity first. That kind of disagreement is familiar to anyone who has run two coders over the same file. After calibration with Mozambique-specific error estimates, the ensemble moved to 58% for sepsis, meningitis and infection, 13% for intrapartum-related events and 12% for prematurity — against 26%, 25% and 29% uncorrected. Among children aged 1–59 months, calibration raised malaria and lowered pneumonia and diarrhoea.

The corrections run in the direction the CHAMPS matrices predict: algorithms were missing infection deaths and over-attributing them to prematurity, so a correction that knows this pulls the distribution back. Two things follow. First, the leading cause of newborn death in a national surveillance dataset can depend on whether the analyst calibrated. Second, uncalibrated estimates are not merely wrong but falsely confident — they assume perfect classification, so their intervals are narrower than the evidence allows. Calibration widens them, which is the honest direction.

What the correction assumes

The framework rests on a claim that cannot be tested in the data it is applied to: transportability. To correct Mozambican VA-only data with error rates estimated from Mozambican CHAMPS deaths, you must assume that the misclassification measured in a small, intensively investigated set of hospital-referred deaths applies to the far larger set of community deaths nobody investigated. The authors say plainly that this is unverifiable without reference causes, and that it presumes symptom–cause relationships behave consistently.

Two further assumptions are stated with similar candour. The error structure is treated as stable over time, though disease prevalence, access to care and the mechanics of VA implementation all move. And the whole exercise is single-cause in a domain where comorbidity is the rule: many under-five deaths involve interacting conditions, and a VA interview may legitimately be describing an intermediate cause in a chain rather than the underlying one. That limitation, the authors suggest, points toward multi-cause frameworks that assign probabilistic weights instead of picking one label.

There is also a boundary on scope. Because CHAMPS covers stillbirths and under-fives, misclassification matrices exist only for neonates and children. Adult and maternal verbal autopsy — the bulk of what national CRVS systems code — has no equivalent inventory yet, and the same country-specific correction cannot be run there.

Where the error becomes a parameter

The practical weight of the paper is in what shipped with it: the misclassification matrices are public, and the correction is packaged as vacalibration, an R package on CRAN (version 2.2, published 20 March 2026) that has been folded into openVA, the toolkit most programmes use to run InterVA and InSilicoVA over a VA file. A ministry analyst can now calibrate a national dataset without access to tissue samples or to CHAMPS.

That is a real change in what a verbal autopsy estimate looks like when it leaves the analyst’s desk. Instead of a distribution presented as measurement, it becomes a distribution plus a stated error model plus wider intervals — a claim that can be argued with. It does not turn verbal autopsy into a certification system, and it does not help the individual family whose relative’s death is coded as prematurity when the true cause was infection; the individual record remains, in this framework, an input to an aggregate, not a finding in its own right. What it does is retire the pretence that the aggregate is error-free. For a method whose entire justification is that most deaths in the world go uncertified, admitting the size and shape of its own error is not a concession. It is the only basis on which the numbers can be defended.

interva.net is an independent observer: it is not affiliated with the InterVA project, Umeå University or the World Health Organization, and nothing here is medical or policy advice.

Sources

← Back to the news digest