Never
- numeric scores, points, grades or star ratings of any kind
- adding, averaging or multiplying rungs or links into one aggregate
- combining cost-effectiveness figures across units (deaths averted, DALYs, WELLBYs, income) or across evaluators
- computing a cost-effectiveness figure; only quote an evaluator's published figure
- assigning a historical verdict word (worked, mixed, didnt, harm) to a live charity
- filling an 'unknown' with the agent's own estimate
- letting a strong link upgrade the rung of a weaker link
- choosing moral weights for the donor
How sure: the rungs
docs/brand/voice.md section 5 (confidence ladder). The order is a ladder of verbs, not a numeric scale.
| Rung id | Rung | How to phrase it |
|---|---|---|
verified | Directly verified outcome | plain statement of fact, cited |
trials | Multiple randomised trials or systematic review | Trials show / A systematic review found |
one_study | One good study or quasi-experiment | One study found / Economists estimate |
own_estimate | An organisation's own estimate | by its own count |
evaluator_model | Evaluator cost-effectiveness model | [Evaluator]'s estimate ([period]): modeled, not guaranteed |
contested | Contested or mixed evidence | The link is contested |
our_view | Our opinion | In our view |
unknown | Unknown | Not yet rated / Unknown |
The links
L0 · Defined
Worked at what, compared with what, for whom, by when?
- Allowed rungs: none: this link is a gate
- If missing: status 'undefined'; output 'Not established: the claim does not say compared with what.' and stop
- Rules it carries: No counterfactual, no claim.
L1 · Happened
Was it delivered, and did the measured outcome move, counting the missing and checking the measure was not gamed?
- Allowed rungs:
verified,trials,own_estimate,unknown - When unknown, say: “Reported by the organisation; not independently checked.”
- Rules it carries: Count the missing.
L2 · Caused
Would it have happened anyway?
- Allowed rungs:
trials,one_study,contested,unknown - Rules it carries: Design beats adjectives. Ask who was chosen, and when.
L3 · Sized
How big is the effect, honestly?
- Allowed rungs:
trials,one_study,contested,unknown - Rules it carries: Read the interval, not the point. Small, noisy and surprising means shrink it. Ask how many studies were run. Ask for the range, not just the average. Decide the slices before you look.
L4 · Travels
Will it work here, run by this organisation, at this scale, now?
- Allowed rungs:
evaluator_model,one_study,unknown - When unknown, say: “A strong hint, not a promise: tested in [place] by [implementer]; no evidence yet for this setting or scale.”
- Rules it carries: Ask whether it travels.
L5 · Worth it
What good per dollar, in a unit someone values, for how long?
- Allowed rungs:
evaluator_model,unknown - When unknown, say: “Not yet rated.”
- Rules it carries: Count every link. Mind the clock.
L6 · Adds
Will my marginal dollar change what happens?
- Allowed rungs:
evaluator_model,own_estimate,unknown - When unknown, say: “Whether more money adds now is unknown; an evaluator's fund can direct it to where it does.”
- Rules it carries: Credit can’t add up to more than the result.
L7 · Checked
Was a prediction written down, and has anyone looked again?
- Allowed rungs:
pending,confirmed,contradicted,unverifiable,no follow-up scheduled - Rules it carries: Write it down, then check.
Combining the links
- Method: report, do not aggregate.
- Weakest link: the first link in chain order whose rung is unknown, contested or own_estimate; if none, the link whose stated range is widest relative to its estimate.
- No borrowing: no link's rung may be raised by another link's evidence.
Donor-voice summary template: [L2–L3 in the rung's verb] [the effect, with interval] ([evaluator label, verbatim]). [L4 sentence]. [Evaluator]'s estimate ([period]): [figure], modeled, not guaranteed. [L6 sentence]. Weakest link: [link name]: [open question]. [L7 status].
Worked examples
Worked example A: treated bed nets (real figures, programme level only; says nothing about any listed charity)
- L0 All-cause child mortality; nets compared with no nets; children in the trial settings; per year of protection.
- L1 Delivered and measured within randomised trials. (Multiple randomised trials or systematic review)
- L2 A Cochrane review found that insecticide-treated nets reduce child mortality from all causes by 17% compared to no nets (high-certainty evidence, in Cochrane's words). (Multiple randomised trials or systematic review)
- L3 Rate ratio 0.83 (95% CI 0.77 to 0.89), 5 trials, 200,833 participants: a saving of 5.6 lives (95% CI 3.6 to 7.6) each year for every 1000 children protected. (Multiple randomised trials or systematic review)
- L4 The trials come from specific settings and decades; carrying them to a district today needs an external-validity argument, which evaluators supply charity by charity. (Unknown)
- L5 Not applicable without a charity and an evaluator's published figure. (Unknown)
- L6 Not applicable without a charity. (Unknown)
- L7 Not applicable at programme level.
Weakest link: L4. A Cochrane review of randomised trials found treated nets cut child deaths by 17 percent (high-certainty evidence). Weakest link for any particular gift: whether that effect carries to today's settings and what one more gift adds; that is what evaluators model.
Worked example B: a before-and-after claim
Illustrative: made-up numbers, not any charity's results. Claim: “Our programme improved wellbeing for 80% of participants.”
- L0 Outcome stated; comparator missing: before and after is not a comparator.
Weakest link: L0. Not established: compared with what? People chosen at a hard moment would be expected to improve on their own (regression to the mean), and the 80% is among those who answered. If half answered, the honest range is between 40% and 90% improved, by the charity's own count, and whether the programme caused it stays unknown.
Worked example C: a famous figure nobody could check (DCP2 deworming, real, cited; history, not advice about any listed charity)
- L0 Cost per disability-adjusted life year (DALY) of treating intestinal worms, as printed in the second edition of Disease Control Priorities (DCP2).
- L1 Not what this example tests: the error sat in the arithmetic, not in whether treatment was delivered.
- L2 Not what this example tests. GiveWell said its finding should not be taken as a general argument against deworming as a promising intervention.
- L3 Not what this example tests.
- L4 Not what this example tests.
- L5 Printed as $3.41 per DALY, among the most cost-effective health programmes of all, and widely cited. But the full calculation was not public, so no one outside could check it: presented as if checked, checkable by nobody. (Unknown)
- L6 Not applicable: no charity and no gift.
- L7 Contradicted. In 2011, after months of correspondence with six scholars, GiveWell obtained the spreadsheet and found five separate errors; corrected, $326.43 per DALY, about a hundred times less cost-effective. The authors confirmed the major errors.
Weakest link: L5. Not established at the time, and later contradicted: a figure that looked like the strongest link was the weakest, because no one could check it. It took L7, a re-check years later, to catch a hundredfold error. All figures are modeled, not guaranteed, and history, not current advice.
Checked in CI by scripts/check_verdict_chain.py: every rule on /how-we-know maps to exactly one link; no score, points or rating keys; no numeric fields; rungs match our voice guide.