Verdict procedure

How an agent answers “did it work?”, link by link.

The procedure behind the verdict chain, as data: /agents/verdict-procedure.json (diditwork.verdict-procedure/v1). Run it on any giving claim using only labels that already exist on this site. It produces a record per link and names the weakest link. It never produces a score. For a charity we list, the MCP tool verdict_chain (GET https://api.diditwork.org/v1/verdict-chain/<org_id>) returns this record already filled in from our data, with nothing computed.

Never

  • numeric scores, points, grades or star ratings of any kind
  • adding, averaging or multiplying rungs or links into one aggregate
  • combining cost-effectiveness figures across units (deaths averted, DALYs, WELLBYs, income) or across evaluators
  • computing a cost-effectiveness figure; only quote an evaluator's published figure
  • assigning a historical verdict word (worked, mixed, didnt, harm) to a live charity
  • filling an 'unknown' with the agent's own estimate
  • letting a strong link upgrade the rung of a weaker link
  • choosing moral weights for the donor

How sure: the rungs

docs/brand/voice.md section 5 (confidence ladder). The order is a ladder of verbs, not a numeric scale.

Rung idRungHow to phrase it
verifiedDirectly verified outcomeplain statement of fact, cited
trialsMultiple randomised trials or systematic reviewTrials show / A systematic review found
one_studyOne good study or quasi-experimentOne study found / Economists estimate
own_estimateAn organisation's own estimateby its own count
evaluator_modelEvaluator cost-effectiveness model[Evaluator]'s estimate ([period]): modeled, not guaranteed
contestedContested or mixed evidenceThe link is contested
our_viewOur opinionIn our view
unknownUnknownNot yet rated / Unknown

L0 · Defined

Worked at what, compared with what, for whom, by when?

  • Allowed rungs: none: this link is a gate
  • If missing: status 'undefined'; output 'Not established: the claim does not say compared with what.' and stop
  • Rules it carries: No counterfactual, no claim.

L1 · Happened

Was it delivered, and did the measured outcome move, counting the missing and checking the measure was not gamed?

  • Allowed rungs: verified, trials, own_estimate, unknown
  • When unknown, say: “Reported by the organisation; not independently checked.”
  • Rules it carries: Count the missing.

L2 · Caused

Would it have happened anyway?

L3 · Sized

How big is the effect, honestly?

L4 · Travels

Will it work here, run by this organisation, at this scale, now?

  • Allowed rungs: evaluator_model, one_study, unknown
  • When unknown, say: “A strong hint, not a promise: tested in [place] by [implementer]; no evidence yet for this setting or scale.”
  • Rules it carries: Ask whether it travels.

L5 · Worth it

What good per dollar, in a unit someone values, for how long?

L6 · Adds

Will my marginal dollar change what happens?

  • Allowed rungs: evaluator_model, own_estimate, unknown
  • When unknown, say: “Whether more money adds now is unknown; an evaluator's fund can direct it to where it does.”
  • Rules it carries: Credit can’t add up to more than the result.

L7 · Checked

Was a prediction written down, and has anyone looked again?

  • Allowed rungs: pending, confirmed, contradicted, unverifiable, no follow-up scheduled
  • Rules it carries: Write it down, then check.

Combining the links

  • Method: report, do not aggregate.
  • Weakest link: the first link in chain order whose rung is unknown, contested or own_estimate; if none, the link whose stated range is widest relative to its estimate.
  • No borrowing: no link's rung may be raised by another link's evidence.

Donor-voice summary template: [L2–L3 in the rung's verb] [the effect, with interval] ([evaluator label, verbatim]). [L4 sentence]. [Evaluator]'s estimate ([period]): [figure], modeled, not guaranteed. [L6 sentence]. Weakest link: [link name]: [open question]. [L7 status].

Worked examples

Worked example A: treated bed nets (real figures, programme level only; says nothing about any listed charity)

  • L0 All-cause child mortality; nets compared with no nets; children in the trial settings; per year of protection.
  • L1 Delivered and measured within randomised trials. (Multiple randomised trials or systematic review)
  • L2 A Cochrane review found that insecticide-treated nets reduce child mortality from all causes by 17% compared to no nets (high-certainty evidence, in Cochrane's words). (Multiple randomised trials or systematic review)
  • L3 Rate ratio 0.83 (95% CI 0.77 to 0.89), 5 trials, 200,833 participants: a saving of 5.6 lives (95% CI 3.6 to 7.6) each year for every 1000 children protected. (Multiple randomised trials or systematic review)
  • L4 The trials come from specific settings and decades; carrying them to a district today needs an external-validity argument, which evaluators supply charity by charity. (Unknown)
  • L5 Not applicable without a charity and an evaluator's published figure. (Unknown)
  • L6 Not applicable without a charity. (Unknown)
  • L7 Not applicable at programme level.

Weakest link: L4. A Cochrane review of randomised trials found treated nets cut child deaths by 17 percent (high-certainty evidence). Weakest link for any particular gift: whether that effect carries to today's settings and what one more gift adds; that is what evaluators model.

Worked example B: a before-and-after claim

Illustrative: made-up numbers, not any charity's results. Claim: “Our programme improved wellbeing for 80% of participants.”

  • L0 Outcome stated; comparator missing: before and after is not a comparator.

Weakest link: L0. Not established: compared with what? People chosen at a hard moment would be expected to improve on their own (regression to the mean), and the 80% is among those who answered. If half answered, the honest range is between 40% and 90% improved, by the charity's own count, and whether the programme caused it stays unknown.

Worked example C: a famous figure nobody could check (DCP2 deworming, real, cited; history, not advice about any listed charity)

  • L0 Cost per disability-adjusted life year (DALY) of treating intestinal worms, as printed in the second edition of Disease Control Priorities (DCP2).
  • L1 Not what this example tests: the error sat in the arithmetic, not in whether treatment was delivered.
  • L2 Not what this example tests. GiveWell said its finding should not be taken as a general argument against deworming as a promising intervention.
  • L3 Not what this example tests.
  • L4 Not what this example tests.
  • L5 Printed as $3.41 per DALY, among the most cost-effective health programmes of all, and widely cited. But the full calculation was not public, so no one outside could check it: presented as if checked, checkable by nobody. (Unknown)
  • L6 Not applicable: no charity and no gift.
  • L7 Contradicted. In 2011, after months of correspondence with six scholars, GiveWell obtained the spreadsheet and found five separate errors; corrected, $326.43 per DALY, about a hundred times less cost-effective. The authors confirmed the major errors.

Weakest link: L5. Not established at the time, and later contradicted: a figure that looked like the strongest link was the weakest, because no one could check it. It took L7, a re-check years later, to catch a hundredfold error. All figures are modeled, not guaranteed, and history, not current advice.

Checked in CI by scripts/check_verdict_chain.py: every rule on /how-we-know maps to exactly one link; no score, points or rating keys; no numeric fields; rungs match our voice guide.