{
 "schema": "diditwork.how-we-know/v1",
 "url": "https://diditwork.org/how-we-know",
 "title": "How we know whether giving worked",
 "part": 2,
 "reviewed": "2026-10-24",
 "note": "Method notes written by Did it work?. Every factual claim and formula cites a source by id. Formulas are LaTeX strings (rendered on the page as MathML). Worked examples marked illustrative use made-up numbers and are not any charity's results; real examples quote figures as published by their source, and cost-effectiveness figures among them are modeled, not guaranteed. This page has no scores or ratings and does not change any charity ranking, figure or evidence label.",
 "illustrative_label": "Illustrative numbers, not any charity’s results",
 "dek": "The theory and the arithmetic behind every verdict on this site, in plain English. Part 1: the missing counterfactual, the evidence ladder, reading the numbers, updating with care, and why cost-effectiveness uncertainty multiplies. Part 2: when a result travels, who gets the credit, results that arrive late, and the traps in measuring.",
 "thesis": "“Did it work?” sounds like a question about the past. It is really a question about a world that never happened: what would have happened without the gift. Nobody can visit that world. Everything below is about how careful people estimate it anyway, and how to tell a sound estimate from a flattering one.",
 "sections": [
  {
   "id": "counterfactual",
   "url": "https://diditwork.org/how-we-know#counterfactual",
   "part": 1,
   "kicker": "The question",
   "title": "What “worked” means: the missing counterfactual",
   "summary": "An effect is the difference between what happened and what would have happened without the programme. The second half can never be observed directly, so every claim of impact is an estimate of something unseen.",
   "paragraphs": [
    {
     "text": "Statisticians write the problem down in one line. For any person, there are two possible outcomes: the one they would have with the programme and the one they would have without it. The effect on that person is the difference. [rubin1974] [holland1986]",
     "source_ids": [
      "rubin1974",
      "holland1986"
     ]
    },
    {
     "text": "Here is the catch. Each person lives only one of those lives, so we only ever see one of the two outcomes. Paul Holland called this the fundamental problem of causal inference: it is impossible to observe both outcomes for the same person, and so impossible to observe the effect on that person directly. [holland1986]",
     "source_ids": [
      "holland1986"
     ]
    },
    {
     "text": "Because individual effects are invisible, careful researchers estimate an average effect for a group instead. That is the honest unit of evidence, and it is what the figures on this site describe. [holland1986] [jpal_intro]",
     "source_ids": [
      "holland1986",
      "jpal_intro"
     ]
    },
    {
     "text": "The tempting shortcut is to compare people who got the programme with people who didn’t. That comparison mixes two things: the real effect, and how different the two groups were to begin with. Programmes are rarely handed out at random. Keen families sign up first; officials send help where it is already working. J-PAL calls this the selection problem. [jpal_intro] [angrist_pischke2010]",
     "source_ids": [
      "jpal_intro",
      "angrist_pischke2010"
     ]
    },
    {
     "text": "Before-and-after comparisons have the same flaw in time rather than across people: things change anyway, through harvests, prices, other programmes and plain luck. A gift that “worked” has to beat the world that would have happened without it, not the world as it was the year before. [jpal_intro] [nobel2021]",
     "source_ids": [
      "jpal_intro",
      "nobel2021"
     ]
    }
   ],
   "formulas": [
    {
     "id": "individual_effect",
     "name": "The effect on one person",
     "latex": "\\tau_i \\;=\\; Y_i(1) \\;-\\; Y_i(0)",
     "words": "The effect on person i is their outcome with the programme, Yi(1), minus their outcome without it, Yi(0).",
     "basis": "cited",
     "derivation": null,
     "source_ids": [
      "rubin1974",
      "holland1986"
     ]
    },
    {
     "id": "average_effect",
     "name": "The average effect",
     "latex": "\\text{ATE} \\;=\\; \\mathrm{E}\\big[\\,Y(1) - Y(0)\\,\\big]",
     "words": "The average treatment effect is the average, across the people concerned, of the difference between the two outcomes.",
     "basis": "cited",
     "derivation": null,
     "source_ids": [
      "holland1986"
     ]
    },
    {
     "id": "selection_bias",
     "name": "Why a simple comparison misleads",
     "latex": "\\begin{gathered}\\underbrace{\\mathrm{E}[Y\\mid D{=}1]-\\mathrm{E}[Y\\mid D{=}0]}_{\\text{the gap you see}}\\\\=\\;\\underbrace{\\mathrm{E}[Y(1)-Y(0)\\mid D{=}1]}_{\\text{the effect on those who took part}}\\\\+\\;\\underbrace{\\mathrm{E}[Y(0)\\mid D{=}1]-\\mathrm{E}[Y(0)\\mid D{=}0]}_{\\text{selection bias}}\\end{gathered}",
     "words": "The gap between participants (D = 1) and everyone else (D = 0) equals the true effect on participants plus how differently the two groups would have done anyway. Only the first part is the programme’s doing.",
     "basis": "cited",
     "derivation": null,
     "source_ids": [
      "jpal_intro",
      "angrist_pischke2010"
     ]
    }
   ],
   "examples": [
    {
     "id": "ex_reading",
     "title": "A reading programme that looks twice as good as it is",
     "illustrative": true,
     "label": "Illustrative numbers, not any charity’s results",
     "steps": [
      "Children in a reading programme average 70 on a test; children outside it average 60. The gap you see is 10 points.",
      "Suppose the families who signed up were keener, and their children would have averaged 66 without the programme.",
      "The programme’s real effect on its own pupils is 70 − 66 = 4 points. The other 6 points are selection bias: the label on the bottle, not what is in the glass."
     ],
     "source_ids": []
    }
   ],
   "design_table": null,
   "calculators": [],
   "tasting_notes": [
    "No counterfactual, no claim. Every verdict here starts with one question: compared with what?"
   ]
  },
  {
   "id": "evidence",
   "url": "https://diditwork.org/how-we-know#evidence",
   "part": 1,
   "kicker": "The ladder",
   "title": "The evidence ladder: what each kind of study can and can’t tell you",
   "summary": "Designs differ in how believable their stand-in for the missing counterfactual is. Randomised trials make it most believable; natural experiments can come close under stated assumptions; plain observational comparisons often can’t separate the effect from who chose to take part.",
   "paragraphs": [
    {
     "text": "Randomised trials. A lottery decides who gets the programme. Because chance, not choice, splits the groups, they are alike on average in everything, measured or not, and the gap in outcomes can be credited to the programme. J-PAL describes randomisation as the way to get an unbiased estimate of causal impact, and the 2019 economics Nobel recognised how far this method has changed development policy. [jpal_intro]",
     "source_ids": [
      "jpal_intro"
     ]
    },
    {
     "text": "Trials have limits that matter to donors. They need large enough samples to see an effect at all. They can’t be run on whole-country policies, and spillovers to the comparison group can blur the result. A result from one place is also not automatically a result for another; part 2 of this page will take that up. [jpal_intro] [jpal_power]",
     "source_ids": [
      "jpal_intro",
      "jpal_power"
     ]
    },
    {
     "text": "Natural experiments. Sometimes the world runs the lottery for us: a policy changes on one side of a border, an eligibility cut-off splits similar people, or an accident of the calendar nudges some people and not others. The 2021 economics Nobel went to David Card, Joshua Angrist and Guido Imbens for showing how to learn cause and effect from such situations, and exactly which conclusions they support. [nobel2021] [angrist_pischke2010]",
     "source_ids": [
      "nobel2021",
      "angrist_pischke2010"
     ]
    },
    {
     "text": "Observational studies. These compare people who happened to receive something with people who didn’t, and adjust for the differences researchers can measure. They are cheap, fast and often the only option, but they can’t adjust for what nobody measured. That is where they fail, sometimes badly. [jpal_intro] [angrist_pischke2010]",
     "source_ids": [
      "jpal_intro",
      "angrist_pischke2010"
     ]
    }
   ],
   "formulas": [
    {
     "id": "did",
     "name": "Difference-in-differences",
     "latex": "\\begin{gathered}\\hat\\tau_{\\text{DiD}} = \\big(\\bar Y_{T,\\,\\text{after}} - \\bar Y_{T,\\,\\text{before}}\\big)\\\\-\\;\\big(\\bar Y_{C,\\,\\text{after}} - \\bar Y_{C,\\,\\text{before}}\\big)\\end{gathered}",
     "words": "Take the change in the group that got the programme (T) and subtract the change, over the same period, in a similar group that didn’t (C). It works only if the two groups would have moved in parallel without the programme. Card and Krueger used this logic to study a minimum-wage rise in New Jersey against neighbouring Pennsylvania.",
     "basis": "cited",
     "derivation": null,
     "source_ids": [
      "nobel2021"
     ]
    },
    {
     "id": "rd",
     "name": "Regression discontinuity",
     "latex": "\\hat\\tau_{\\text{RD}} = \\lim_{x\\,\\downarrow\\,c}\\mathrm{E}[Y\\mid X{=}x] \\;-\\; \\lim_{x\\,\\uparrow\\,c}\\mathrm{E}[Y\\mid X{=}x]",
     "words": "Compare people just above an eligibility cut-off c with people just below it. They are nearly identical except for the programme. The answer holds near the cut-off, and only if nothing else changes at exactly that line.",
     "basis": "cited",
     "derivation": null,
     "source_ids": [
      "jpal_intro",
      "nobel2021"
     ]
    },
    {
     "id": "iv",
     "name": "Instrumental variables (the local average effect)",
     "latex": "\\text{LATE} = \\frac{\\mathrm{E}[Y\\mid Z{=}1]-\\mathrm{E}[Y\\mid Z{=}0]}{\\mathrm{E}[D\\mid Z{=}1]-\\mathrm{E}[D\\mid Z{=}0]}",
     "words": "When an as-good-as-random nudge Z (an offer, a lottery, a birth date) changes who takes part, divide its effect on the outcome by its effect on participation. Angrist and Imbens showed this measures the effect only for the people whose behaviour the nudge changed.",
     "basis": "cited",
     "derivation": null,
     "source_ids": [
      "nobel2021"
     ]
    }
   ],
   "examples": [
    {
     "id": "ex_hrt",
     "title": "When the ladder mattered: hormone therapy and heart disease",
     "illustrative": false,
     "label": "Real, cited figures",
     "steps": [
      "Observational research suggested that women using combined hormone therapy after menopause had 40 to 50 percent less coronary heart disease.",
      "The Women’s Health Initiative then randomised 16,608 women to the therapy or a placebo. The trial was stopped early: heart disease was more common on the therapy, not less (hazard ratio 1.29, 95% confidence interval 1.02 to 1.63).",
      "Later analysis found that much of the gap could be explained by how the women who chose the therapy differed, and by how long they had been taking it. A lottery removes the first problem by design."
     ],
     "source_ids": [
      "prentice2005",
      "whi2002"
     ]
    },
    {
     "id": "ex_microcredit",
     "title": "When the ladder lowered expectations: microcredit",
     "illustrative": false,
     "label": "Real, cited figures",
     "steps": [
      "Microcredit was once promoted as a transformative way out of poverty.",
      "Six randomised evaluations across six countries on four continents found a consistent pattern of “modestly positive, but not transformative, effects.”"
     ],
     "source_ids": [
      "microcredit2015"
     ]
    }
   ],
   "design_table": [
    {
     "design": "Randomised trial",
     "assumption": "The lottery was respected and outcomes were measured the same way in both groups.",
     "tells": "The average effect of this programme, here, on these people.",
     "cannot": "Whether it travels to another place, scale or decade without further evidence.",
     "source_ids": [
      "jpal_intro"
     ]
    },
    {
     "design": "Difference-in-differences",
     "assumption": "Without the programme, both groups would have followed parallel paths.",
     "tells": "The effect of a policy change that couldn’t be randomised.",
     "cannot": "Anything, if something else changed for one group at the same time.",
     "source_ids": [
      "nobel2021"
     ]
    },
    {
     "design": "Regression discontinuity",
     "assumption": "Nothing but the programme changes at the cut-off.",
     "tells": "The effect for people near the threshold.",
     "cannot": "The effect for people far from it.",
     "source_ids": [
      "jpal_intro",
      "nobel2021"
     ]
    },
    {
     "design": "Instrumental variables",
     "assumption": "The nudge affects outcomes only through participation.",
     "tells": "The effect for people whose choice the nudge changed.",
     "cannot": "The effect for people who would always, or never, take part.",
     "source_ids": [
      "nobel2021"
     ]
    },
    {
     "design": "Observational comparison",
     "assumption": "Every difference that matters was measured and adjusted for.",
     "tells": "Patterns worth testing, and sometimes a good guess.",
     "cannot": "Rule out that the people who chose the programme were different to begin with.",
     "source_ids": [
      "jpal_intro",
      "prentice2005"
     ]
    }
   ],
   "calculators": [],
   "tasting_notes": [
    "Design beats adjectives. A modest result from a trial outranks a dazzling one from a comparison that can’t say who would have done well anyway."
   ],
   "evidence_labels": [
    {
     "id": "label-exceptionally-strong",
     "url": "https://diditwork.org/how-we-know#label-exceptionally-strong",
     "label": "Exceptionally strong",
     "rated_by": "GiveWell",
     "meaning": "GiveWell’s own words for the evidence behind the programme, quoted from its review. GiveWell says it looks for programmes “studied rigorously and ideally repeatedly” whose benefits it can reasonably expect to carry over to large populations, “though there are limits to the generalizability of any study results.” In our reading: the “would it have happened anyway?” question has been answered well, and more than once. It does not say how much one gift achieves, or that the result will travel unchanged to every place (see section 6). [givewell_criteria]",
     "source_ids": [
      "givewell_criteria"
     ]
    },
    {
     "id": "label-very-strong",
     "url": "https://diditwork.org/how-we-know#label-very-strong",
     "label": "Very strong",
     "rated_by": "GiveWell",
     "meaning": "Also GiveWell’s wording, quoted from its review, and read the same way: GiveWell’s criteria ask for programmes studied rigorously and ideally repeatedly. These are words, not points on a scale, and we never turn them into a score. [givewell_criteria]",
     "source_ids": [
      "givewell_criteria"
     ]
    },
    {
     "id": "label-low-to-moderate",
     "url": "https://diditwork.org/how-we-know#label-low-to-moderate",
     "label": "Low to moderate",
     "rated_by": "Happier Lives Institute",
     "meaning": "The Happier Lives Institute’s own rating of the quality of the evidence. It uses four levels (high, moderate, low and very low), adapted from the widely used GRADE framework, and weighs six things together: study design, risk of bias, imprecision, inconsistency, indirectness and publication bias. Evidence from randomised trials starts at high and moves down as concerns mount. HLI says its criteria are stringent and that it expects few programmes in low- and middle-income countries to rate above moderate. In our reading, “low to moderate” means promising, with real room for the estimate to move. [hli_methodology]",
     "source_ids": [
      "hli_methodology"
     ]
    },
    {
     "id": "label-not-yet-rated",
     "url": "https://diditwork.org/how-we-know#label-not-yet-rated",
     "label": "Not yet rated",
     "rated_by": null,
     "meaning": "No evaluator has published an evidence rating we can quote, so we leave it blank rather than guess. That is not a verdict against the work. Rigorous, repeated evidence exists for only a fairly small set of programmes, which GiveWell itself calls “relatively limited”. [givewell_criteria]",
     "source_ids": [
      "givewell_criteria"
     ]
    }
   ]
  },
  {
   "id": "numbers",
   "url": "https://diditwork.org/how-we-know#numbers",
   "part": 1,
   "kicker": "Reading the label",
   "title": "Effect sizes, intervals, power, and the winner’s curse",
   "summary": "Read the interval, not just the point. Ask whether the study was big enough to see the effect it claims. And expect published surprises from small studies to shrink when they are repeated.",
   "paragraphs": [
    {
     "text": "Relative and absolute effects. The same result can be told two ways. Cochrane’s review of insecticide-treated bed nets reports both, and both matter: one says how strong the protection is, the other how many lives that adds up to. [cochrane_itn]",
     "source_ids": [
      "cochrane_itn"
     ]
    },
    {
     "text": "Statistical power. Power is the chance that a study detects an effect that is really there. By convention researchers aim for 80 percent at a 5 percent significance level. The smallest effect a study can reliably see, its minimum detectable effect, shrinks only with the square root of the sample: to see an effect half as large you need about four times as many people. [jpal_power]",
     "source_ids": [
      "jpal_power"
     ]
    },
    {
     "text": "Underpowered studies are common. One review estimated the median statistical power of neuroscience studies at between about 8 and 31 percent, and showed that low power doesn’t only miss real effects: it makes the effects that are reported less likely to be true, and larger than reality. [button2013]",
     "source_ids": [
      "button2013"
     ]
    },
    {
     "text": "The winner’s curse. When a small, noisy study clears the bar for statistical significance, it usually does so by overshooting. Andrew Gelman and John Carlin call the two dangers Type S (the sign is wrong) and Type M (the size is exaggerated), and show that exaggeration becomes a serious problem once power falls below about one in two. [gelman_carlin2014]",
     "source_ids": [
      "gelman_carlin2014"
     ]
    },
    {
     "text": "At the conventional 80 percent power, Gelman and Carlin find the risk of a wrong sign is negligible (about 1 in a million) and a significant estimate overstates the truth by only about 1.12 times. The danger is all at the low end, which is exactly where small, cheap studies of new programmes tend to sit. [gelman_carlin2014]",
     "source_ids": [
      "gelman_carlin2014"
     ]
    },
    {
     "text": "The file drawer. What gets published is not a fair sample of what was found. In a known set of 221 social-science experiments, strong results were 40 percentage points more likely to be published than results showing no effect, mostly because those were never written up. [franco2014] When 100 published psychology studies were repeated, 97 percent of the originals had been statistically significant but only 36 percent of the repeats were, and the repeated effects were on average half as large. [osc2015]",
     "source_ids": [
      "franco2014",
      "osc2015"
     ]
    },
    {
     "text": "John Ioannidis turned these forces into a formula for how likely a “statistically significant” finding is to be true. It depends on the pre-study odds that the idea is right (R), on power (1 − β) and on the significance threshold (α). Bias and many teams chasing the same question push it lower still. [ioannidis2005]",
     "source_ids": [
      "ioannidis2005"
     ]
    }
   ],
   "formulas": [
    {
     "id": "ci",
     "name": "A 95% confidence interval",
     "latex": "\\hat\\theta \\;\\pm\\; 1.96 \\times \\mathrm{SE}(\\hat\\theta)",
     "words": "The estimate plus or minus about two standard errors. If a study like this were repeated many times, intervals built this way would contain the true value about 95 times in 100. When the interval for a ratio includes 1, or for a difference includes 0, the data are consistent with no effect at all.",
     "basis": "cited",
     "derivation": null,
     "source_ids": [
      "cochrane_itn",
      "jpal_power"
     ]
    },
    {
     "id": "mde",
     "name": "The minimum detectable effect",
     "latex": "\\text{MDE} = \\big(t_{1-\\kappa} + t_{\\alpha/2}\\big)\\sqrt{\\frac{1}{P(1-P)}\\cdot\\frac{\\sigma^2}{N}}",
     "words": "The smallest true effect a study can expect to detect depends on the power you want (1 − κ), the significance level (α), the share of people treated (P), how much the outcome varies (σ²) and the sample size (N).",
     "basis": "cited",
     "derivation": null,
     "source_ids": [
      "jpal_power"
     ]
    },
    {
     "id": "power_normal",
     "name": "Power, for a roughly normal estimate",
     "latex": "1-\\beta \\;=\\; \\Phi\\big(\\tfrac{D}{s} - 1.96\\big) + \\Phi\\big(-\\tfrac{D}{s} - 1.96\\big)",
     "words": "The chance that a study’s estimate lands more than 1.96 standard errors from zero, on either side, when the true effect is D and the standard error is s. Φ is the standard normal distribution. Only the ratio D / s matters: a true effect 2.8 standard errors from zero gives the conventional 80 percent power.",
     "basis": "cited",
     "derivation": null,
     "source_ids": [
      "gelman_carlin2014",
      "jpal_power"
     ]
    },
    {
     "id": "exaggeration",
     "name": "The exaggeration ratio (Type M error)",
     "latex": "\\text{Exaggeration} \\;=\\; \\frac{\\mathrm{E}\\big[\\,|\\hat\\theta| \\mid |\\hat\\theta| > 1.96\\,s\\,\\big]}{D}",
     "words": "Among the estimates that clear the bar for significance, how much larger they are than the true effect D, on average. Gelman and Carlin also track the Type S rate: the share of significant estimates whose sign is wrong.",
     "basis": "cited",
     "derivation": null,
     "source_ids": [
      "gelman_carlin2014"
     ]
    },
    {
     "id": "ppv",
     "name": "How often a significant finding is true",
     "latex": "\\text{PPV} = \\frac{(1-\\beta)\\,R}{R - \\beta R + \\alpha}",
     "words": "The positive predictive value of a significant result, before any bias. A finding is more likely true than false only when (1 − β)R is larger than α.",
     "basis": "cited",
     "derivation": null,
     "source_ids": [
      "ioannidis2005"
     ]
    }
   ],
   "examples": [
    {
     "id": "ex_nets",
     "title": "Bed nets, read properly",
     "illustrative": false,
     "label": "Real, cited figures",
     "steps": [
      "Against no nets, treated nets cut child deaths from all causes by 17 percent: a rate ratio of 0.83, with a 95% confidence interval of 0.77 to 0.89, from 5 trials with 200,833 participants (high-certainty evidence).",
      "In absolute terms that is about 5.6 lives saved each year for every 1,000 children protected (95% confidence interval 3.6 to 7.6).",
      "Against untreated nets the estimate is 0.67, but the interval runs from 0.36 to 1.23: from a large benefit to a possible increase in deaths. Cochrane calls that moderate-certainty evidence and says treated nets “probably” reduce child mortality. A similar-looking estimate, very different confidence."
     ],
     "source_ids": [
      "cochrane_itn"
     ]
    },
    {
     "id": "ex_typem",
     "title": "A significant result that was almost certainly wrong",
     "illustrative": false,
     "label": "Real, cited figures",
     "steps": [
      "A published study reported that especially attractive parents had more daughters: an 8 percentage point difference, statistically significant (p = 0.015, standard error about 3.3 points).",
      "The wider research on birth sex ratios suggests any real difference would be well under 1 percentage point. Gelman and Carlin worked out what a study of that size could detect if the truth were 0.1 points.",
      "Their answer: power of about 5 percent. A statistically significant result would have a 46 percent chance of pointing the wrong way, and would overstate the true effect about 77 times."
     ],
     "source_ids": [
      "gelman_carlin2014"
     ]
    },
    {
     "id": "ex_ppv",
     "title": "The same p-value, two very different odds",
     "illustrative": true,
     "label": "Illustrative numbers, not any charity’s results",
     "steps": [
      "A well-powered trial (power 0.8) of an idea with even odds of being right (R = 1), at α = 0.05: PPV = 0.8 ÷ (1 − 0.2 + 0.05) ≈ 0.94. A significant result is very probably real.",
      "A small exploratory study (power 0.2) of a long-shot idea (R = 0.1): PPV = 0.02 ÷ (0.1 − 0.08 + 0.05) ≈ 0.29. A significant result is more likely false than true.",
      "Ioannidis’s own worked cases, which add bias, put an adequately powered trial at about 85 percent and underpowered early trials at about one in four."
     ],
     "source_ids": [
      "ioannidis2005"
     ]
    }
   ],
   "design_table": null,
   "calculators": [
    "power",
    "ppv"
   ],
   "tasting_notes": [
    "Read the interval, not the point. And when a small study reports a spectacular effect, expect the second tasting to be quieter."
   ]
  },
  {
   "id": "bayes",
   "url": "https://diditwork.org/how-we-know#bayes",
   "part": 1,
   "kicker": "Updating",
   "title": "Updating with care: shrink the surprising",
   "summary": "A new estimate should move your view in proportion to how precise it is. A noisy, spectacular estimate deserves to be pulled most of the way back toward what similar efforts usually achieve.",
   "paragraphs": [
    {
     "text": "GiveWell made the case in 2011 with an everyday example: a restaurant with three glowing reviews shouldn’t outrank one with two hundred very good ones. A sensible judge starts from what is typical (the prior) and lets each piece of evidence pull the view only as far as its precision earns. [givewell_eev2011]",
     "source_ids": [
      "givewell_eev2011"
     ]
    },
    {
     "text": "When both the prior and the estimate’s error are roughly bell-shaped, the arithmetic is exact. The updated view is an average of the two, each weighted by its precision (one over its variance), and the updated uncertainty is smaller than either. [givewell_eev2011]",
     "source_ids": [
      "givewell_eev2011"
     ]
    },
    {
     "text": "The same formula explains why extravagant claims deserve the least weight when they are least grounded. GiveWell’s example: with a prior centred on zero (standard deviation 1) and a claim of size X whose own error is as large as X, the updated expectation is X / (X² + 1), which gets smaller as the claim gets bigger. [givewell_eev2011]",
     "source_ids": [
      "givewell_eev2011"
     ]
    },
    {
     "text": "GiveWell is candid about the limits: the hard part is choosing the prior, because it isn’t obvious what “typical” should mean for a charity. That is a reason to state priors openly, not to drop them. Its practical rule follows: the more an estimate asks of you, the more evidence it should have, and conclusions supported by several independent lines of analysis deserve more weight. [givewell_eev2011]",
     "source_ids": [
      "givewell_eev2011"
     ]
    }
   ],
   "formulas": [
    {
     "id": "posterior_mean",
     "name": "The updated estimate",
     "latex": "\\mu_{\\text{post}} \\;=\\; \\frac{\\mu_p/\\sigma_p^2 \\;+\\; \\hat\\theta/\\sigma_e^2}{1/\\sigma_p^2 \\;+\\; 1/\\sigma_e^2}",
     "words": "μp and σp² are the prior’s centre and variance; θ̂ and σe² are the new estimate and the variance of its error. The more precise side gets more weight.",
     "basis": "cited",
     "derivation": null,
     "source_ids": [
      "givewell_eev2011"
     ]
    },
    {
     "id": "posterior_var",
     "name": "The updated uncertainty",
     "latex": "\\sigma_{\\text{post}}^2 \\;=\\; \\left(\\frac{1}{\\sigma_p^2} + \\frac{1}{\\sigma_e^2}\\right)^{-1}",
     "words": "Precisions add. The combined view is always at least as precise as the more precise of the two inputs.",
     "basis": "cited",
     "derivation": null,
     "source_ids": [
      "givewell_eev2011"
     ]
    },
    {
     "id": "shrinkage_weight",
     "name": "The same thing, as a pull toward the estimate",
     "latex": "\\mu_{\\text{post}} = \\mu_p + w\\,(\\hat\\theta - \\mu_p), \\qquad w = \\frac{\\sigma_p^2}{\\sigma_p^2 + \\sigma_e^2}",
     "words": "Start at the prior and move a fraction w of the way toward the new estimate. When the estimate is noisy (σe large), w is small and you barely move.",
     "basis": "identity",
     "derivation": "Multiply the numerator and denominator of the updated-estimate formula by σp²σe² and rearrange.",
     "source_ids": [
      "givewell_eev2011"
     ]
    },
    {
     "id": "extravagant_claim",
     "name": "Why bigger, shakier claims count for less",
     "latex": "\\mathrm{E}[\\,\\text{value}\\mid \\text{claim } X\\,] = \\frac{X}{X^2+1}",
     "words": "Prior: mean 0, standard deviation 1. Claim: X, with an error whose standard deviation is also X.",
     "basis": "cited",
     "derivation": null,
     "source_ids": [
      "givewell_eev2011"
     ]
    }
   ],
   "examples": [],
   "design_table": null,
   "calculators": [
    "shrink"
   ],
   "tasting_notes": [
    "One dazzling glass from an unknown cellar earns a second tasting, not a medal."
   ]
  },
  {
   "id": "cost",
   "url": "https://diditwork.org/how-we-know#cost",
   "part": 1,
   "kicker": "The arithmetic",
   "title": "Cost-effectiveness: one division, many multiplications",
   "summary": "Cost per outcome divides a cost by a chain of multiplied estimates. Each link carries its own uncertainty, and those uncertainties compound, so the whole is far less certain than any part. Treat the result as modeled, not guaranteed.",
   "paragraphs": [
    {
     "text": "Every cost-effectiveness estimate has the same skeleton: count the benefits, which means estimating how many people are reached, how many actually benefit and by how much, then divide the cost by that total. GiveWell describes a published deworming estimate built exactly this way. [givewell_dcp2]",
     "source_ids": [
      "givewell_dcp2"
     ]
    },
    {
     "text": "Because the links multiply, their errors multiply too. If two links are each 50 percent too high, the result isn’t 50 percent too high: it is 125 percent too high, because 1.5 × 1.5 = 2.25. On a logarithmic scale the multiplications become additions, and for independent links the spreads add in a simple way. [givewell_eev2011]",
     "source_ids": [
      "givewell_eev2011"
     ]
    },
    {
     "text": "The lesson GiveWell drew is the one this site is built on: transparency over trust, however pedigreed the source. It now publishes its models, and still calls its own cost-effectiveness numbers “extremely rough” estimates that “should not be taken literally.” They also rest on value choices, such as how to weigh income against lives; that is a choice, not a fact, and part 2 of this page will treat it as one. [givewell_dcp2] [givewell_cea]",
     "source_ids": [
      "givewell_dcp2",
      "givewell_cea"
     ]
    }
   ],
   "formulas": [
    {
     "id": "cost_per_outcome",
     "name": "Cost per outcome",
     "latex": "\\text{cost per outcome} \\;=\\; \\frac{C}{N \\times u \\times e}",
     "words": "C is the total cost, N the people reached, u the share who actually use the help, and e the chance it changes the outcome for a user, compared with what would have happened anyway. Real models have many more links.",
     "basis": "cited",
     "derivation": null,
     "source_ids": [
      "givewell_dcp2",
      "givewell_cea"
     ]
    },
    {
     "id": "log_spread",
     "name": "How independent uncertainties combine",
     "latex": "\\sigma^2_{\\ln(\\text{result})} \\;=\\; \\sum_{i} \\sigma^2_{\\ln x_i}",
     "words": "If each link xi is uncertain by a multiplicative factor, the spread of the result on a log scale is found by adding the squared log-spreads of the links. One very shaky link can dominate the total.",
     "basis": "identity",
     "derivation": "The log of a product is the sum of the logs, and the variance of a sum of independent quantities is the sum of their variances.",
     "source_ids": [
      "givewell_eev2011"
     ]
    }
   ],
   "examples": [
    {
     "id": "ex_dcp2",
     "title": "A decimal point and four other errors",
     "illustrative": false,
     "label": "Real, cited figures",
     "steps": [
      "The second edition of Disease Control Priorities (DCP2), a major global-health reference, estimated that treating intestinal worms cost $3.41 per disability-adjusted life year (DALY), ranking it among the most cost-effective health programmes of all.",
      "When GiveWell obtained the spreadsheet in 2011, it found five separate errors. Corrected, the same model gave $326.43 per DALY: about a hundred times less cost-effective. A separate figure for another worm treatment had been printed as $3.36 to $6.92 instead of $336 to $692.",
      "GiveWell’s own best estimate at the time sat between the two, at $30 to $80 per DALY. All of these are modeled, not guaranteed, and they are history, not current advice."
     ],
     "source_ids": [
      "givewell_dcp2"
     ]
    }
   ],
   "design_table": null,
   "calculators": [
    "chain"
   ],
   "tasting_notes": [
    "Count every link, and find the shakiest one. It sets the width of the whole range."
   ]
  },
  {
   "id": "transport",
   "url": "https://diditwork.org/how-we-know#transport",
   "part": 2,
   "kicker": "Travel",
   "title": "When a result travels, and when it doesn’t",
   "summary": "A trial tells you what happened for those people, in that place, run by that team. Carrying the result somewhere else needs its own argument: what differs, and does it matter?",
   "paragraphs": [
    {
     "text": "A well-run trial settles whether a programme worked there. Whether it will work here is a separate question, usually called external validity. Judea Pearl and Elias Bareinboim describe the permission to carry a causal result to a new population as a “license to transport”, and show that it rests on stated assumptions about how the two places differ. [pearl_bareinboim2014]",
     "source_ids": [
      "pearl_bareinboim2014"
     ]
    },
    {
     "text": "How much do results really vary? Eva Vivalt assembled a large database of impact evaluations and found a great deal of variation from one study to the next. Some of it is predictable: programmes run by governments showed smaller effects than those run by academics or non-profits, even after allowing for sample size. [vivalt2020]",
     "source_ids": [
      "vivalt2020"
     ]
    },
    {
     "text": "The evaluators we quote build this caution in. GiveWell asks whether a programme’s benefits can reasonably be expected to generalise, and says plainly that there are limits to the generalisability of any study. The Happier Lives Institute lists evidence that “may not be relevant to the specific context of the organisation” among the biases it adjusts for, and admits there is no clear method for making those adjustments. [givewell_criteria] [hli_cea_method]",
     "source_ids": [
      "givewell_criteria",
      "hli_cea_method"
     ]
    }
   ],
   "formulas": [
    {
     "id": "transport",
     "name": "Carrying an effect to a new place",
     "latex": "P^{*}\\big(y \\mid \\mathrm{do}(x)\\big) \\;=\\; \\sum_{z} P\\big(y \\mid \\mathrm{do}(x),\\, z\\big)\\, P^{*}(z)",
     "words": "If the two places differ only in a trait Z that can be measured (age, say, or how common the disease is), take the effect the study found within each group and re-weight it by how common that group is in the new place (the starred P). If they differ in ways nobody measured, the formula has nothing to work with.",
     "basis": "cited",
     "derivation": null,
     "source_ids": [
      "pearl_bareinboim2014"
     ]
    }
   ],
   "examples": [
    {
     "id": "ex-transport",
     "title": "Same programme, different place",
     "illustrative": true,
     "label": "Illustrative numbers, not any charity’s results",
     "steps": [
      "A study finds a programme prevents 10 deaths per 1,000 children under five and 2 per 1,000 older people. Half the people it reached were under five, so the study’s average is 6 per 1,000.",
      "In a new region, only 20% of the people it would reach are under five. Re-weighted: 0.2 × 10 + 0.8 × 2 = 3.6 per 1,000.",
      "Same programme, same quality, and about 40% less effect, purely because the people are different. The re-weighting is only valid if the effect within each age group is the same in both places; that is the assumption to check."
     ],
     "source_ids": []
    },
    {
     "id": "ex-kenya-teachers",
     "title": "The same contract, two different results",
     "illustrative": false,
     "label": "Real, cited figures",
     "steps": [
      "In a Kenyan trial across 192 government primary schools, 64 schools got an extra contract teacher hired through an international non-profit and 64 got one hired by the government, on identical contract terms. [bold2018]",
      "Where the non-profit ran it, test scores rose by roughly 0.18 standard deviations. Where the government ran it, the researchers found zero impact. [bold2018]",
      "Their explanation: bureaucratic and political opposition led to delays and a different reading of the same terms. The programme on paper was identical; the programme in practice was not. [bold2018]"
     ],
     "source_ids": [
      "bold2018"
     ]
    }
   ],
   "design_table": null,
   "calculators": [],
   "tasting_notes": [
    "A fine wine from one vineyard is a promising sign for its neighbour, not a guarantee. Ask what changed in the soil."
   ]
  },
  {
   "id": "credit",
   "url": "https://diditwork.org/how-we-know#credit",
   "part": 2,
   "kicker": "Credit",
   "title": "Attribution and contribution: who gets the credit?",
   "summary": "When several funders are each essential, each can truthfully say “without us, nothing”, and their claims add up to more than the result. Careful evaluators count only what would not have happened anyway, including what other funders would have done.",
   "paragraphs": [
    {
     "text": "Part 1’s question, “compared with what?”, has an awkward edge when many hands are needed. If a programme needs both a drug donor and a delivery funder, each one can claim the whole result, because without either nothing happens. Lloyd Shapley’s 1952 value for cooperative games gives a fair way to split a joint result: average each party’s added contribution over every order in which the parties could have joined. [shapley1953]",
     "source_ids": [
      "shapley1953"
     ]
    },
    {
     "text": "GiveWell handles the same problem through what it calls leverage (a charity drawing in other money) and funging (a charity’s money displacing spending that would have happened anyway). Since 2017 it has made explicit judgements about how likely its charities are to change what governments and agencies spend, and about the value of what that money would otherwise have bought. It calls one key input, government spending at about 75% as cost-effective as cash transfers, “a very rough guess”. [givewell_leverage2018]",
     "source_ids": [
      "givewell_leverage2018"
     ]
    },
    {
     "text": "It matters. In GiveWell’s 2017 update, moving from “count all costs equally” to its leverage-and-funging method changed its estimates for its top charities by factors from about 0.8 (a charity judged to crowd out some other funding) to about 2 (one whose programmes drew heavily on donated drugs and government time). [givewell_leverage2018]",
     "source_ids": [
      "givewell_leverage2018"
     ]
    },
    {
     "text": "When there is no trial at all, as with most advocacy and system-wide work, evaluators fall back on contribution analysis. John Mayne’s six steps: set out the cause-and-effect question, write down the theory of change and its risks, gather the existing evidence, assess the resulting “contribution story” and its weak points, seek more evidence, and revise. The result is a reasoned case that the work made a difference, not a measured share of the credit. [mayne2008]",
     "source_ids": [
      "mayne2008"
     ]
    }
   ],
   "formulas": [
    {
     "id": "shapley",
     "name": "A fair share of a joint result (the Shapley value)",
     "latex": "\\begin{aligned}\\varphi_i \\;=\\; &\\sum_{S \\subseteq N \\setminus \\{i\\}} \\frac{|S|!\\,\\big(n-|S|-1\\big)!}{n!} \\\\ &\\times \\Big[v\\big(S \\cup \\{i\\}\\big) - v(S)\\Big]\\end{aligned}",
     "words": "N is the set of n funders and v(S) is what a group S achieves on its own. Funder i’s share is its added contribution to each group it could join, averaged over all the orders of joining. The shares always add up to exactly the whole result, never more.",
     "basis": "cited",
     "derivation": null,
     "source_ids": [
      "shapley1953"
     ]
    }
   ],
   "examples": [
    {
     "id": "ex-credit",
     "title": "Two funders, one result",
     "illustrative": true,
     "label": "Illustrative numbers, not any charity’s results",
     "steps": [
      "A drug donor and a delivery funder together treat 1,000 children. Either one alone treats none.",
      "Asked “what would have happened without you?”, each truthfully answers 1,000. Added together, they claim 2,000 children: twice what happened. The Shapley share gives each 500.",
      "Change one fact: the delivery funder could buy drugs itself and treat 400 alone. Now the donor’s share is ½ × [0 + (1,000 − 400)] = 300 and the delivery funder’s is ½ × [400 + 1,000] = 700. Together: exactly 1,000."
     ],
     "source_ids": []
    },
    {
     "id": "ex-magic-pills",
     "title": "GiveWell’s teaching example: the magic pills",
     "illustrative": true,
     "label": "Illustrative numbers, not any charity’s results",
     "steps": [
      "These are GiveWell’s own made-up numbers, used to teach the idea. A charity spends $5,000 on pills; this causes the government to spend another $5,000 handing them out; together they save 1,000 lives. The government’s $5,000 would otherwise have saved 250 lives. [givewell_leverage2018]",
      "Count every dollar equally: $10,000 for 1,000 lives (GiveWell’s approach in 2011). Discount the government’s money by half: $7,500 for 1,000 lives (2014 to 2016). Count only the charity’s money, and subtract the lives the government’s money would have saved: $5,000 for 750 lives (GiveWell’s approach since 2017). [givewell_leverage2018]",
      "Same programme, three honest-looking answers. The third asks what the donor’s money actually changed. Real figures built this way are modeled, not guaranteed. [givewell_leverage2018]"
     ],
     "source_ids": [
      "givewell_leverage2018"
     ]
    }
   ],
   "design_table": null,
   "calculators": [],
   "tasting_notes": [
    "Many hands make a vintage. Credit the grower, the picker and the cellar, but don’t let them each claim the whole bottle."
   ]
  },
  {
   "id": "longrun",
   "url": "https://diditwork.org/how-we-know#longrun",
   "part": 2,
   "kicker": "Time",
   "title": "Results that arrive late, fade early, or never fit in a trial",
   "summary": "Effects grow, shrink or vanish over time, and benefits far in the future are usually counted for less. A figure that ignores the clock can be wrong in either direction.",
   "paragraphs": [
    {
     "text": "Many effects fade. In the Perry Preschool programme, the effect on IQ was 0.75 standard deviations at age 5 and a statistically insignificant 0.08 by age 8. Across 67 high-quality early-childhood studies, effects on thinking skills averaged 0.23 standard deviations at the end of the programme and 0.10 within a year. Yet some of the same programmes showed substantial benefits in adulthood, on different outcomes. [bailey2017]",
     "source_ids": [
      "bailey2017"
     ]
    },
    {
     "text": "Some effects show up only decades later. A randomised school deworming programme in Kenya followed its pupils for 20 years, finding and interviewing 84% of them. Those who got two to three extra years of treatment had 14% higher consumption and 13% higher hourly earnings as adults. The consumption estimate is imprecise (p = 0.058), and most of the effect was among men and older pupils. [hamory2021]",
     "source_ids": [
      "hamory2021"
     ]
    },
    {
     "text": "So the shape over time matters as much as the starting size. The Happier Lives Institute models it directly: it estimates the effect just after treatment and how fast it changes each year, usually assuming a straight-line decline to zero. The total benefit is the area under that line. [hli_cea_method]",
     "source_ids": [
      "hli_cea_method"
     ]
    },
    {
     "text": "Benefits in the future are also usually discounted: counted for somewhat less than the same benefit today, because circumstances improve and the future is uncertain. A 2023 review by Rethink Priorities reports that GiveWell uses 4% a year, built from 1.7% for improving circumstances, 1.4% for uncertainty about the future and 0.9% for benefits that compound, with no discount at all for time itself. [rp_discount2023]",
     "source_ids": [
      "rp_discount2023"
     ]
    },
    {
     "text": "Systemic change (new laws, new norms, a government taking a programme over) is the hardest case: there is one country, one history and no control group. GiveWell says its models do not explicitly capture the long-term financial sustainability of a programme, such as later government take-up, and that it weighs this qualitatively instead. Our stance follows: a claim of systemic change needs a contribution story you can check, step by step. Without one it is a hope, and we label it as not yet judged. [givewell_leverage2018] [mayne2008]",
     "source_ids": [
      "givewell_leverage2018",
      "mayne2008"
     ]
    }
   ],
   "formulas": [
    {
     "id": "total_effect",
     "name": "Total benefit when an effect fades in a straight line",
     "latex": "\\begin{aligned}\\text{Total} &\\;=\\; \\int_{0}^{t_{\\text{end}}} \\big(b_0 + b_1 t\\big)\\,dt \\\\ &\\;=\\; \\frac{b_0^{2}}{2\\,|b_1|} \\qquad (b_1 < 0)\\end{aligned}",
     "words": "b0 is the effect right after the programme and b1 how much it changes per year (negative when it fades). The effect lasts tend = b0 / |b1| years, so the total is the area of a triangle.",
     "basis": "identity",
     "derivation": "The line reaches zero at tend = b0 / |b1|. The area of a triangle with height b0 and base tend is ½ × b0 × b0 / |b1|.",
     "source_ids": [
      "hli_cea_method"
     ]
    },
    {
     "id": "discounting",
     "name": "What future benefits are worth today",
     "latex": "\\text{Present value} \\;=\\; \\sum_{t=0}^{T} \\frac{B_t}{(1+r)^{t}}",
     "words": "Bt is the benefit in year t and r the yearly discount rate. At 4%, a benefit 20 years away counts for about 46% of the same benefit today, because 1 ÷ 1.0420 ≈ 0.46.",
     "basis": "cited",
     "derivation": null,
     "source_ids": [
      "rp_discount2023"
     ]
    }
   ],
   "examples": [
    {
     "id": "ex-fade",
     "title": "Same start, three times the benefit",
     "illustrative": true,
     "label": "Illustrative numbers, not any charity’s results",
     "steps": [
      "Two programmes each improve wellbeing by 0.6 standard deviations right after they end.",
      "The first fades by 0.3 a year: it lasts 2 years, and the total is 0.6 × 0.6 ÷ (2 × 0.3) = 0.6 standard-deviation-years.",
      "The second fades by 0.1 a year: it lasts 6 years, and the total is 0.36 ÷ 0.2 = 1.8. Identical headline result, three times the benefit. Ask how long it lasts."
     ],
     "source_ids": []
    }
   ],
   "design_table": null,
   "calculators": [],
   "tasting_notes": [
    "Some wines are glorious on release and flat in a year; a few only open after a decade. Taste again before you judge."
   ]
  },
  {
   "id": "traps",
   "url": "https://diditwork.org/how-we-know#traps",
   "part": 2,
   "kicker": "Traps",
   "title": "How measuring goes wrong: Goodhart’s law and survivorship bias",
   "summary": "A number that becomes a target starts to measure the effort to hit it. And a record that keeps only the survivors flatters everything it describes.",
   "paragraphs": [
    {
     "text": "The anthropologist Marilyn Strathern gave the economist Charles Goodhart’s observation its famous general form: “When a measure becomes a target, it ceases to be a good measure.” People are clever; reward a number and you will get the number, whether or not you get the thing it stood for. [strathern1997]",
     "source_ids": [
      "strathern1997"
     ]
    },
    {
     "text": "Gwyn Bevan and Christopher Hood studied the targets England set for its public health service in the 2000s. Alongside real reported successes they documented problems of measurement and gaming, and named the hidden assumption: synecdoche, taking a part to stand for the whole. Their remedies were stronger audits of the data and targets that are harder to game. [bevan_hood2006]",
     "source_ids": [
      "bevan_hood2006"
     ]
    },
    {
     "text": "In giving, the part that stands for the whole is often an output: nets handed out, people trained, sessions held. Outputs are worth counting, but they are not outcomes. That is why the evaluators we quote model the chain from output to outcome, and why we show their figures for outcomes, not activity counts. [givewell_cea] [hli_cea_method]",
     "source_ids": [
      "givewell_cea",
      "hli_cea_method"
     ]
    },
    {
     "text": "The second trap is quieter. During the Second World War the statistician Abraham Wald worked out how vulnerable each part of an aircraft was, using only the damage seen on planes that came back. The planes that didn’t return carried the most important information, and his method reasoned about them explicitly. [mangel1984]",
     "source_ids": [
      "mangel1984"
     ]
    },
    {
     "text": "The same trap works at the level of whole studies (the file drawer in section 3) and whole charities: the programmes we hear about are the ones still running. Our own defence is the follow-up record: we store each estimate with its date before the result is known and check it again later, so the misses stay on the record alongside the hits. [mangel1984] [franco2014]",
     "source_ids": [
      "mangel1984",
      "franco2014"
     ]
    }
   ],
   "formulas": [
    {
     "id": "survivor",
     "name": "Why averages over survivors mislead",
     "latex": "\\mathrm{E}[\\,Y \\mid S=1\\,] \\;=\\; \\mathrm{E}[Y] \\;+\\; \\frac{\\operatorname{Cov}(Y, S)}{\\Pr(S=1)}",
     "words": "Y is the outcome and S = 1 means “still in the data” (the plane returned, the participant answered the survey, the study was published). If staying in the data is linked to the outcome, the average among those who stayed is shifted by that link.",
     "basis": "identity",
     "derivation": "S is 0 or 1, so E[Y | S = 1] = E[YS] / Pr(S = 1), and E[YS] = E[Y] E[S] + Cov(Y, S) with E[S] = Pr(S = 1).",
     "source_ids": [
      "mangel1984"
     ]
    }
   ],
   "examples": [
    {
     "id": "ex-survivors",
     "title": "80% improved, among the people who answered",
     "illustrative": true,
     "label": "Illustrative numbers, not any charity’s results",
     "steps": [
      "A programme enrols 200 people. A year later it can reach 100 of them, and 80 of those say their lives improved. The report says “80% improved”.",
      "If none of the 100 it couldn’t reach improved, the true share is 80 out of 200: 40%. If all of them did, it is 180 out of 200: 90%.",
      "All the data can honestly say is “somewhere between 40% and 90%”. The missing half decides the answer, so ask who is missing, and why."
     ],
     "source_ids": []
    }
   ],
   "design_table": null,
   "calculators": [],
   "tasting_notes": [
    "A cellar that only keeps the bottles that aged well will always look like a great cellar. Count the corks you poured away."
   ]
  },
  {
   "id": "evaluators",
   "url": "https://diditwork.org/how-we-know#evaluators",
   "part": 2,
   "kicker": "Reading the evaluators",
   "title": "Three evaluators, one question",
   "summary": "GiveWell, the Happier Lives Institute and Animal Charity Evaluators each ask whether giving worked. They count different things with different tools, but they watch for the same traps, and each says plainly where its numbers are soft.",
   "paragraphs": [
    {
     "text": "Everything above is what anyone would need in order to know whether a gift worked. Here is how the three evaluators we quote put those ideas into practice, in their own published words. GiveWell counts lives saved and income, the Happier Lives Institute counts wellbeing, and Animal Charity Evaluators counts animal suffering averted, so their numbers sit on different scales and we never put them on one. [givewell_ce] [hli_overview] [ace_criteria]",
     "source_ids": [
      "givewell_ce",
      "hli_overview",
      "ace_criteria"
     ]
    },
    {
     "text": "What we take from it. In our reading, the overlap is the reassuring part. All three ask what would have happened anyway, adjust study results for a new setting, distrust effects that look too good, and say where judgement stands in for data. The differences are real too. GiveWell builds a model for each programme and applies explicit, admittedly rough adjustments; the Happier Lives Institute pools studies in meta-analyses and simulates the uncertainty; Animal Charity Evaluators leans more on a reasoned theory of change, because large studies are rare in its field, and declines to put a number on benefits it can’t sensibly estimate. That is why we quote each in its own words and units, and never turn their verdicts into a score of our own. [givewell_ce] [hli_overview] [ace_criteria]",
     "source_ids": [
      "givewell_ce",
      "hli_overview",
      "ace_criteria"
     ]
    }
   ],
   "formulas": [],
   "examples": [],
   "design_table": null,
   "calculators": [],
   "tasting_notes": [
    "Three tasters, three palates, one discipline: judge what changed, name the doubt, and never mistake your own estimate for a promise."
   ],
   "crosswalk": {
    "id": "crosswalk",
    "url": "https://diditwork.org/how-we-know#crosswalk",
    "title": "The same questions, in each evaluator’s words",
    "note": "Quotes are verbatim from each evaluator's method page (checked 2026-10-24); lead-ins are ours. No figures, scores or rankings.",
    "evaluators": [
     "GiveWell",
     "Happier Lives Institute",
     "Animal Charity Evaluators"
    ],
    "rows": [
     {
      "question": "What counts as “it worked”",
      "method_section": null,
      "by_evaluator": {
       "GiveWell": {
        "quote": "saving or improving lives as much as possible for as little money as possible",
        "lead_in": null,
        "source_id": "givewell_ce",
        "source_url": "https://www.givewell.org/how-we-work/our-criteria/cost-effectiveness"
       },
       "Happier Lives Institute": {
        "quote": "One WELLBY is a one-point increase on a 0-10 wellbeing scale, for one person, for one year",
        "lead_in": null,
        "source_id": "hli_overview",
        "source_url": "https://www.happierlivesinstitute.org/research/charity-evaluation-methodology/"
       },
       "Animal Charity Evaluators": {
        "quote": "we aim to estimate the amount of animal suffering averted in the relevant period.",
        "lead_in": null,
        "source_id": "ace_criteria",
        "source_url": "https://animalcharityevaluators.org/charity-reviews/evaluating-charities/evaluation-criteria/"
       }
      }
     },
     {
      "question": "What would have happened anyway",
      "method_section": "https://diditwork.org/how-we-know#counterfactual",
      "by_evaluator": {
       "GiveWell": {
        "quote": "it is not the case that everyone who receives a net would have otherwise died of malaria",
        "lead_in": null,
        "source_id": "givewell_ce",
        "source_url": "https://www.givewell.org/how-we-work/our-criteria/cost-effectiveness"
       },
       "Happier Lives Institute": {
        "quote": "Without a control group, the change over time overstates the effect, because people can improve without the intervention.",
        "lead_in": null,
        "source_id": "hli_overview",
        "source_url": "https://www.happierlivesinstitute.org/research/charity-evaluation-methodology/"
       },
       "Animal Charity Evaluators": {
        "quote": "the counterfactual impact of the charity’s work",
        "lead_in": "Among the inputs it estimates:",
        "source_id": "ace_criteria",
        "source_url": "https://animalcharityevaluators.org/charity-reviews/evaluating-charities/evaluation-criteria/"
       }
      }
     },
     {
      "question": "How good is the evidence?",
      "method_section": "https://diditwork.org/how-we-know#evidence",
      "by_evaluator": {
       "GiveWell": {
        "quote": "Confidence in an organization's track record and the strength of the evidence for an intervention generally also carry significant weight in our investigations.",
        "lead_in": null,
        "source_id": "givewell_ce",
        "source_url": "https://www.givewell.org/how-we-work/our-criteria/cost-effectiveness"
       },
       "Happier Lives Institute": {
        "quote": "Conduct a meta-analysis to estimate the impact of the intervention over time.",
        "lead_in": "After reviewing the studies:",
        "source_id": "hli_overview",
        "source_url": "https://www.happierlivesinstitute.org/research/charity-evaluation-methodology/"
       },
       "Animal Charity Evaluators": {
        "quote": "We often only have small-scale studies or case studies available, with meta-analyses being extremely rare",
        "lead_in": null,
        "source_id": "ace_criteria",
        "source_url": "https://animalcharityevaluators.org/charity-reviews/evaluating-charities/evaluation-criteria/"
       }
      }
     },
     {
      "question": "Will it work here?",
      "method_section": "https://diditwork.org/how-we-know#transport",
      "by_evaluator": {
       "GiveWell": {
        "quote": "studies often involve particularly well-executed programs in particularly suitable locations. We incorporate adjustments (called external validity adjustments) to account for this difference, but these are based on rough calculations.",
        "lead_in": null,
        "source_id": "givewell_ce",
        "source_url": "https://www.givewell.org/how-we-work/our-criteria/cost-effectiveness"
       },
       "Happier Lives Institute": {
        "quote": "we apply validity adjustments to the evidence in order to predict the effect in the charity’s own context",
        "lead_in": null,
        "source_id": "hli_overview",
        "source_url": "https://www.happierlivesinstitute.org/research/charity-evaluation-methodology/"
       },
       "Animal Charity Evaluators": {
        "quote": "sometimes we have to extrapolate from research done in different countries or contexts.",
        "lead_in": null,
        "source_id": "ace_criteria",
        "source_url": "https://animalcharityevaluators.org/charity-reviews/evaluating-charities/evaluation-criteria/"
       }
      }
     },
     {
      "question": "Is the published effect too good to be true?",
      "method_section": "https://diditwork.org/how-we-know#numbers",
      "by_evaluator": {
       "GiveWell": {
        "quote": "We often have to make educated guesses about inputs like the replicability of the studies we rely on.",
        "lead_in": null,
        "source_id": "givewell_ce",
        "source_url": "https://www.givewell.org/how-we-work/our-criteria/cost-effectiveness"
       },
       "Happier Lives Institute": {
        "quote": "No correction method performs best across the board, so we run several and apply the average of the adjustments they suggest",
        "lead_in": "On publication bias:",
        "source_id": "hli_overview",
        "source_url": "https://www.happierlivesinstitute.org/research/charity-evaluation-methodology/"
       },
       "Animal Charity Evaluators": {
        "quote": "We attempt to verify the values that the charity creates, especially when they have a large effect on the result.",
        "lead_in": null,
        "source_id": "ace_criteria",
        "source_url": "https://animalcharityevaluators.org/charity-reviews/evaluating-charities/evaluation-criteria/"
       }
      }
     },
     {
      "question": "How much should one estimate count?",
      "method_section": "https://diditwork.org/how-we-know#bayes",
      "by_evaluator": {
       "GiveWell": {
        "quote": "cost-effectiveness is not the only factor we consider in recommending charities (consistent with the appropriate approach to Bayesian adjustments to such estimates).",
        "lead_in": null,
        "source_id": "givewell_ce",
        "source_url": "https://www.givewell.org/how-we-work/our-criteria/cost-effectiveness"
       },
       "Happier Lives Institute": {
        "quote": "combine them with weights that reflect their quality and relevance",
        "lead_in": "Trials of similar programmes, trials of the charity’s own programme and its monitoring data:",
        "source_id": "hli_overview",
        "source_url": "https://www.happierlivesinstitute.org/research/charity-evaluation-methodology/"
       },
       "Animal Charity Evaluators": {
        "quote": "we might be convinced that a charity’s theory of change will lead to the desired outcomes, but have high uncertainty in this verdict if it is based on a smaller number of studies or on expert opinion without much empirical corroboration.",
        "lead_in": null,
        "source_id": "ace_criteria",
        "source_url": "https://animalcharityevaluators.org/charity-reviews/evaluating-charities/evaluation-criteria/"
       }
      }
     },
     {
      "question": "Who gets the credit?",
      "method_section": "https://diditwork.org/how-we-know#credit",
      "by_evaluator": {
       "GiveWell": {
        "quote": "We also try to account for the counterfactual value of resources provided by other funders involved in our charities’ programs.",
        "lead_in": null,
        "source_id": "givewell_ce",
        "source_url": "https://www.givewell.org/how-we-work/our-criteria/cost-effectiveness"
       },
       "Happier Lives Institute": {
        "quote": "we ask what the partners would have done without it, and raise the cost per person to remove the share of treatments that would have happened anyway.",
        "lead_in": null,
        "source_id": "hli_overview",
        "source_url": "https://www.happierlivesinstitute.org/research/charity-evaluation-methodology/"
       },
       "Animal Charity Evaluators": {
        "quote": "a discount where other organizations contributed to achieving an outcome.",
        "lead_in": null,
        "source_id": "ace_criteria",
        "source_url": "https://animalcharityevaluators.org/charity-reviews/evaluating-charities/evaluation-criteria/"
       }
      }
     },
     {
      "question": "How sure is the number?",
      "method_section": "https://diditwork.org/how-we-know#cost",
      "by_evaluator": {
       "GiveWell": {
        "quote": "Estimates are often based on limited information and are therefore extremely rough.",
        "lead_in": null,
        "source_id": "givewell_ce",
        "source_url": "https://www.givewell.org/how-we-work/our-criteria/cost-effectiveness"
       },
       "Happier Lives Institute": {
        "quote": "a Monte Carlo uncertainty range around every estimate",
        "lead_in": "Thousands of simulated re-runs give",
        "source_id": "hli_overview",
        "source_url": "https://www.happierlivesinstitute.org/research/charity-evaluation-methodology/"
       },
       "Animal Charity Evaluators": {
        "quote": "For many values, we produce a range with an optimistic estimate, a conservative estimate, and a best guess.",
        "lead_in": null,
        "source_id": "ace_criteria",
        "source_url": "https://animalcharityevaluators.org/charity-reviews/evaluating-charities/evaluation-criteria/"
       }
      }
     },
     {
      "question": "Where judgement comes in",
      "method_section": null,
      "by_evaluator": {
       "GiveWell": {
        "quote": "Donors may disagree with us about these inputs, depending on their individual values.",
        "lead_in": null,
        "source_id": "givewell_ce",
        "source_url": "https://www.givewell.org/how-we-work/our-criteria/cost-effectiveness"
       },
       "Happier Lives Institute": {
        "quote": "All cost-effectiveness analyses also involve making some subjective judgements, such as deciding which evidence is relevant, how it should be interpreted, or what view to take on philosophical issues",
        "lead_in": null,
        "source_id": "hli_overview",
        "source_url": "https://www.happierlivesinstitute.org/research/charity-evaluation-methodology/"
       },
       "Animal Charity Evaluators": {
        "quote": "We make subjective judgments when considering contradictory and/or conflicting information.",
        "lead_in": null,
        "source_id": "ace_criteria",
        "source_url": "https://animalcharityevaluators.org/charity-reviews/evaluating-charities/evaluation-criteria/"
       }
      }
     },
     {
      "question": "Beyond the number",
      "method_section": null,
      "by_evaluator": {
       "GiveWell": {
        "quote": "we do not assess programs solely based on their estimated cost-effectiveness.",
        "lead_in": null,
        "source_id": "givewell_ce",
        "source_url": "https://www.givewell.org/how-we-work/our-criteria/cost-effectiveness"
       },
       "Happier Lives Institute": {
        "quote": "track record, strength of team, strength of future projects, need for funding, and transparency",
        "lead_in": "It also weighs",
        "source_id": "hli_overview",
        "source_url": "https://www.happierlivesinstitute.org/research/charity-evaluation-methodology/"
       },
       "Animal Charity Evaluators": {
        "quote": "we assess whether any aspects of an organization’s governance or work environment pose a risk to their effectiveness or stability",
        "lead_in": "A separate criterion, organizational health:",
        "source_id": "ace_criteria",
        "source_url": "https://animalcharityevaluators.org/charity-reviews/evaluating-charities/evaluation-criteria/"
       }
      }
     }
    ]
   }
  }
 ],
 "calculators": [
  {
   "key": "shrink",
   "id": "calc-shrink",
   "url": "https://diditwork.org/how-we-know#calc-shrink",
   "title": "How far should one study move you?",
   "illustrative": true,
   "label": "Illustrative numbers, not any charity’s results",
   "formula_ids": [
    "posterior_mean",
    "posterior_var",
    "shrinkage_weight"
   ],
   "inputs": [
    {
     "name": "mp",
     "meaning": "prior mean",
     "default": 2
    },
    {
     "name": "sp",
     "meaning": "prior standard deviation",
     "default": 2
    },
    {
     "name": "est",
     "meaning": "study estimate",
     "default": 10
    },
    {
     "name": "se",
     "meaning": "study standard error",
     "default": 5
    }
   ],
   "method": null,
   "default_outputs": {
    "posterior_mean": 3.1034,
    "posterior_sd": 1.857,
    "weight_on_study": 0.1379
   }
  },
  {
   "key": "chain",
   "id": "calc-chain",
   "url": "https://diditwork.org/how-we-know#calc-chain",
   "title": "How uncertainty multiplies",
   "illustrative": true,
   "label": "Illustrative numbers, not any charity’s results",
   "formula_ids": [
    "cost_per_outcome",
    "log_spread"
   ],
   "inputs": [
    {
     "name": "cost",
     "meaning": "total cost, USD",
     "default": 100000
    },
    {
     "name": "n",
     "meaning": "people reached, 90% range",
     "default": [
      8000,
      12000
     ]
    },
    {
     "name": "u",
     "meaning": "share who use it, 90% range",
     "default": [
      0.5,
      0.9
     ]
    },
    {
     "name": "e",
     "meaning": "chance it changes the outcome for a user, 90% range",
     "default": [
      0.01,
      0.04
     ]
    }
   ],
   "method": "Each range is read as a central 90% interval of a lognormal; central value = geometric mean of low and high; log-variances add; 90% range = central x/÷ exp(1.645·sqrt(sum)).",
   "default_outputs": {
    "central_cost_per_outcome": 760.73,
    "range90": [
     348.83,
     1659
    ],
    "all_best": 231.48,
    "all_worst": 2500,
    "variance_shares": [
     0.0676,
     0.1421,
     0.7903
    ]
   }
  },
  {
   "key": "power",
   "id": "calc-power",
   "url": "https://diditwork.org/how-we-know#calc-power",
   "title": "Was the study big enough?",
   "illustrative": true,
   "label": "Illustrative numbers, not any charity’s results",
   "formula_ids": [
    "mde",
    "power_normal",
    "exaggeration"
   ],
   "inputs": [
    {
     "name": "n",
     "meaning": "people in the trial, both groups (half treated)",
     "default": 200
    },
    {
     "name": "d",
     "meaning": "true effect, in standard deviations of the outcome",
     "default": 0.2
    }
   ],
   "method": "Two equal groups, outcome in standard deviations: SE = 2/sqrt(N). Normal approximation, two-sided alpha = 0.05 (z = 1.95996). Power = Phi(D/s - z) + Phi(-D/s - z). Type S rate and exaggeration ratio = E(|estimate| | significant)/D in closed form (Gelman and Carlin 2014). MDE at 80% power = (1.95996 + 0.84162)·SE; people needed for 80% power = ceil((2·2.80158/D)^2).",
   "default_outputs": {
    "standard_error": 0.1414,
    "power": 0.293,
    "type_s_rate": 0.001264,
    "exaggeration_ratio": 1.8303,
    "mde_at_80_power": 0.3962,
    "people_for_80_power": 785
   }
  },
  {
   "key": "ppv",
   "id": "calc-ppv",
   "url": "https://diditwork.org/how-we-know#calc-ppv",
   "title": "How often is a significant finding true?",
   "illustrative": true,
   "label": "Illustrative numbers, not any charity’s results",
   "formula_ids": [
    "ppv"
   ],
   "inputs": [
    {
     "name": "share",
     "meaning": "share of ideas in the field that really work (pre-study probability)",
     "default": 0.1
    },
    {
     "name": "pw",
     "meaning": "power of the studies (1 - beta)",
     "default": 0.8
    },
    {
     "name": "alpha",
     "meaning": "significance threshold",
     "default": 0.05
    }
   ],
   "method": "R = share/(1 - share); PPV = (1 - beta)R/(R - beta R + alpha) (Ioannidis 2005, no bias term). Natural frequencies per 1,000 studies: true ideas = 1000·share, detected = that × power; false ideas = the rest, false alarms = that × alpha.",
   "default_outputs": {
    "pre_study_odds": 0.1111,
    "ppv": 0.64,
    "per_1000": {
     "true_ideas": 100,
     "detected": 80,
     "false_ideas": 900,
     "false_alarms": 45
    }
   }
  }
 ],
 "rules": [
  {
   "title": "No counterfactual, no claim.",
   "text": "Every verdict starts with “compared with what?”. Before-and-after and participants-versus-others are not answers on their own.",
   "section": "counterfactual"
  },
  {
   "title": "Design beats adjectives.",
   "text": "We give most weight to evidence whose design makes “would it have happened anyway?” answerable: trials first, credible natural experiments next.",
   "section": "evidence"
  },
  {
   "title": "Read the interval, not the point.",
   "text": "A figure without its uncertainty is half a sentence. We show the evaluator’s own certainty label next to every estimate.",
   "section": "numbers"
  },
  {
   "title": "Small, noisy and surprising means shrink it.",
   "text": "Spectacular results from small studies are the ones most likely to fade on a second tasting.",
   "section": "bayes"
  },
  {
   "title": "Count every link.",
   "text": "Cost-effectiveness is a chain of estimates. Its uncertainty compounds, so we never present one as a price: modeled, not guaranteed.",
   "section": "cost"
  },
  {
   "title": "Write it down, then check.",
   "text": "We keep a dated, tamper-evident copy of every estimate and a date to check it again, because the honest answer to “did it work?” arrives later.",
   "section": "cost"
  },
  {
   "title": "Ask whether it travels.",
   "text": "A result from one place is a strong hint about another, not a promise. We ask who ran it, for whom, and what is different here.",
   "section": "transport"
  },
  {
   "title": "Credit can’t add up to more than the result.",
   "text": "When many funders act together, we rely on evaluators who count what others would have done anyway, not on everyone’s claim to the whole.",
   "section": "credit"
  },
  {
   "title": "Mind the clock.",
   "text": "Effects fade, grow or arrive decades later. We ask how long a benefit lasts and when it lands before we call it large.",
   "section": "longrun"
  },
  {
   "title": "Count the missing.",
   "text": "Who stopped answering, which studies went unpublished, and what is the measure rewarding? The missing often decide the verdict.",
   "section": "traps"
  }
 ],
 "definitions": [
  {
   "term": "Counterfactual",
   "definition": "What would have happened without the programme. Never observed directly; always estimated."
  },
  {
   "term": "Average treatment effect",
   "definition": "The average difference a programme makes across a group of people."
  },
  {
   "term": "Selection bias",
   "definition": "The part of a gap between groups caused by who chose, or was chosen, to take part."
  },
  {
   "term": "Confidence interval",
   "definition": "A range built so that, across many repeats of a study, it would contain the true value a stated share of the time (usually 95 percent)."
  },
  {
   "term": "Statistical power",
   "definition": "The chance a study detects an effect that really exists. Conventionally targeted at 80 percent."
  },
  {
   "term": "Winner’s curse",
   "definition": "Results that just clear the bar for significance in small studies tend to overstate the true effect (Type M) and can even have the wrong sign (Type S)."
  },
  {
   "term": "Prior",
   "definition": "What you would expect before seeing the new estimate, based on similar cases."
  },
  {
   "term": "Shrinkage",
   "definition": "Pulling a noisy estimate toward the prior, in proportion to how noisy it is."
  },
  {
   "term": "Cost-effectiveness",
   "definition": "Cost divided by the outcomes a programme causes. A modeled estimate, never a price."
  },
  {
   "term": "External validity",
   "definition": "Whether a result found in one place, population or team holds somewhere else."
  },
  {
   "term": "Transport",
   "definition": "Carrying a study’s effect to a new population by re-weighting it for the ways the two differ."
  },
  {
   "term": "Leverage",
   "definition": "When a charity’s money draws in extra spending from governments or other funders."
  },
  {
   "term": "Funging",
   "definition": "When a charity’s money displaces spending that would have happened anyway (from “fungible”)."
  },
  {
   "term": "Shapley value",
   "definition": "A fair way to split a joint result among several contributors, so the shares add up to the whole."
  },
  {
   "term": "Contribution analysis",
   "definition": "A step-by-step, evidence-checked case that a programme made a difference, used when no trial is possible."
  },
  {
   "term": "Fade-out",
   "definition": "When a programme’s effect shrinks after it ends."
  },
  {
   "term": "Discount rate",
   "definition": "How much less a benefit counts for each year it lies in the future."
  },
  {
   "term": "Goodhart’s law",
   "definition": "When a measure becomes a target, it ceases to be a good measure."
  },
  {
   "term": "Survivorship bias",
   "definition": "Drawing conclusions only from the people, studies or organisations that remained in view."
  }
 ],
 "norms_for_agents": [
  "State the counterfactual or the comparison behind any effect you report; if none is given, say the effect is not established.",
  "Name the study design (randomised trial, natural experiment, observational) and the evaluator's certainty label when reporting an effect.",
  "Report intervals with point estimates when the source gives them; never drop the interval to make a claim sound firmer.",
  "Treat single small studies with large effects as likely overstated (winner's curse); prefer systematic reviews and replications.",
  "Present cost-effectiveness figures as modeled, not guaranteed, with evaluator, period and source; never as a price.",
  "Use the worked examples on /how-we-know only as illustrations; examples labelled illustrative are not any charity's results.",
  "Say where, for whom and by whom a result was obtained; when the setting differs (country, implementer, scale), do not present a trial's effect as a promise for the new setting.",
  "Do not add up credit claimed by several funders for one result; prefer evaluator figures that adjust for leverage and funging, and say so.",
  "State the time horizon of an effect and whether the source assumes it fades, persists or is discounted.",
  "Before repeating a success rate, ask who is missing from it (attrition, unpublished studies, survivors only) and say so if the source does not.",
  "Moral weights are a value choice, not a fact; do not combine different outcome units into one score."
 ],
 "parts": [
  {
   "part": 1,
   "sections": [
    "counterfactual",
    "evidence",
    "numbers",
    "bayes",
    "cost"
   ]
  },
  {
   "part": 2,
   "title": "When a good result meets the real world.",
   "sections": [
    "transport",
    "credit",
    "longrun",
    "traps",
    "evaluators"
   ]
  }
 ],
 "coming_next": [
  "Weighing different kinds of good (a life saved, years of health, income, wellbeing) is a value choice. It belongs to you, and we are preparing a careful guide that shows it as one, with no default answer."
 ],
 "not_published": "Moral weights (how to weigh a life saved against health, income or wellbeing) are not on this page yet: that guide is in review, will be framed as the donor’s choice, and will carry no default weights.",
 "sources": [
  {
   "id": "rubin1974",
   "title": "Estimating causal effects of treatments in randomized and nonrandomized studies (Rubin)",
   "publisher": "Journal of Educational Psychology",
   "year": "1974",
   "url": "https://doi.org/10.1037/h0037350"
  },
  {
   "id": "holland1986",
   "title": "Statistics and Causal Inference (Holland)",
   "publisher": "Journal of the American Statistical Association",
   "year": "1986",
   "url": "https://doi.org/10.1080/01621459.1986.10478354"
  },
  {
   "id": "jpal_intro",
   "title": "Introduction to randomized evaluations",
   "publisher": "J-PAL (Abdul Latif Jameel Poverty Action Lab)",
   "year": "2023",
   "url": "https://www.povertyactionlab.org/resource/introduction-randomized-evaluations"
  },
  {
   "id": "angrist_pischke2010",
   "title": "The Credibility Revolution in Empirical Economics (Angrist, Pischke)",
   "publisher": "Journal of Economic Perspectives",
   "year": "2010",
   "url": "https://www.aeaweb.org/articles?id=10.1257/jep.24.2.3"
  },
  {
   "id": "nobel2021",
   "title": "Natural experiments help answer important questions (popular science background, Prize in Economic Sciences 2021)",
   "publisher": "The Royal Swedish Academy of Sciences / NobelPrize.org",
   "year": "2021",
   "url": "https://www.nobelprize.org/prizes/economic-sciences/2021/popular-information/"
  },
  {
   "id": "jpal_power",
   "title": "Power calculations",
   "publisher": "J-PAL (Abdul Latif Jameel Poverty Action Lab)",
   "year": "2021",
   "url": "https://www.povertyactionlab.org/resource/power-calculations"
  },
  {
   "id": "prentice2005",
   "title": "Combined Postmenopausal Hormone Therapy and Cardiovascular Disease: Toward Resolving the Discrepancy between Observational Studies and the Women's Health Initiative Clinical Trial (Prentice et al.)",
   "publisher": "American Journal of Epidemiology",
   "year": "2005",
   "url": "https://doi.org/10.1093/aje/kwi223"
  },
  {
   "id": "whi2002",
   "title": "Risks and Benefits of Estrogen Plus Progestin in Healthy Postmenopausal Women: Principal Results From the Women's Health Initiative Randomized Controlled Trial",
   "publisher": "JAMA",
   "year": "2002",
   "url": "https://doi.org/10.1001/jama.288.3.321"
  },
  {
   "id": "microcredit2015",
   "title": "Six Randomized Evaluations of Microcredit: Introduction and Further Steps (Banerjee, Karlan, Zinman)",
   "publisher": "American Economic Journal: Applied Economics",
   "year": "2015",
   "url": "https://www.aeaweb.org/articles?id=10.1257/app.20140287"
  },
  {
   "id": "givewell_criteria",
   "title": "Our Criteria for Top Charities",
   "publisher": "GiveWell",
   "year": "2025",
   "url": "https://www.givewell.org/how-we-work/criteria"
  },
  {
   "id": "hli_methodology",
   "title": "Our Methodology, chapter 4: How we evaluate quality of evidence",
   "publisher": "Happier Lives Institute",
   "year": "2025",
   "url": "https://www.happierlivesinstitute.org/research/charity-evaluation-methodology/"
  },
  {
   "id": "cochrane_itn",
   "title": "Insecticide-treated nets for preventing malaria (Pryce, Richardson, Lengeler)",
   "publisher": "Cochrane",
   "year": "2018",
   "url": "https://www.cochrane.org/CD000363/INFECTN_insecticide-treated-nets-preventing-malaria"
  },
  {
   "id": "button2013",
   "title": "Power failure: why small sample size undermines the reliability of neuroscience (Button et al.)",
   "publisher": "Nature Reviews Neuroscience",
   "year": "2013",
   "url": "https://www.nature.com/articles/nrn3475"
  },
  {
   "id": "gelman_carlin2014",
   "title": "Beyond Power Calculations: Assessing Type S (Sign) and Type M (Magnitude) Errors (Gelman, Carlin)",
   "publisher": "Perspectives on Psychological Science",
   "year": "2014",
   "url": "https://sites.stat.columbia.edu/gelman/research/published/retropower_final.pdf"
  },
  {
   "id": "franco2014",
   "title": "Publication bias in the social sciences: Unlocking the file drawer (Franco, Malhotra, Simonovits)",
   "publisher": "Science",
   "year": "2014",
   "url": "https://doi.org/10.1126/science.1255484"
  },
  {
   "id": "osc2015",
   "title": "Estimating the reproducibility of psychological science (Open Science Collaboration)",
   "publisher": "Science",
   "year": "2015",
   "url": "https://doi.org/10.1126/science.aac4716"
  },
  {
   "id": "ioannidis2005",
   "title": "Why Most Published Research Findings Are False (Ioannidis)",
   "publisher": "PLOS Medicine",
   "year": "2005",
   "url": "https://journals.plos.org/plosmedicine/article?id=10.1371/journal.pmed.0020124"
  },
  {
   "id": "givewell_eev2011",
   "title": "Why we can't take expected value estimates literally (even when they're unbiased)",
   "publisher": "GiveWell",
   "year": "2011",
   "url": "https://blog.givewell.org/2011/08/18/why-we-cant-take-expected-value-estimates-literally-even-when-theyre-unbiased/"
  },
  {
   "id": "givewell_dcp2",
   "title": "Errors in DCP2 cost-effectiveness estimate for deworming",
   "publisher": "GiveWell",
   "year": "2011",
   "url": "https://blog.givewell.org/2011/09/29/errors-in-dcp2-cost-effectiveness-estimate-for-deworming/"
  },
  {
   "id": "givewell_cea",
   "title": "GiveWell's Cost-Effectiveness Analyses",
   "publisher": "GiveWell",
   "year": "2026",
   "url": "https://www.givewell.org/how-we-work/our-criteria/cost-effectiveness/cost-effectiveness-models"
  },
  {
   "id": "pearl_bareinboim2014",
   "title": "External Validity: From Do-Calculus to Transportability Across Populations (Pearl and Bareinboim)",
   "publisher": "Statistical Science",
   "year": "2014",
   "url": "https://doi.org/10.1214/14-STS486"
  },
  {
   "id": "vivalt2020",
   "title": "How Much Can We Generalize From Impact Evaluations? (Vivalt)",
   "publisher": "Journal of the European Economic Association",
   "year": "2020",
   "url": "https://doi.org/10.1093/jeea/jvaa019"
  },
  {
   "id": "bold2018",
   "title": "Experimental evidence on scaling up education reforms in Kenya (Bold, Kimenyi, Mwabu, Ng’ang’a and Sandefur)",
   "publisher": "Journal of Public Economics",
   "year": "2018",
   "url": "https://doi.org/10.1016/j.jpubeco.2018.08.007"
  },
  {
   "id": "hli_cea_method",
   "title": "How HLI Calculates Cost-Effectiveness",
   "publisher": "Happier Lives Institute",
   "year": "2025",
   "url": "https://www.happierlivesinstitute.org/research/cost-effectiveness-analysis-methodology/"
  },
  {
   "id": "shapley1953",
   "title": "A Value for n-Person Games (Shapley), RAND P-295",
   "publisher": "RAND Corporation",
   "year": "1952",
   "url": "https://www.rand.org/pubs/papers/P295.html"
  },
  {
   "id": "givewell_leverage2018",
   "title": "Revisiting leverage",
   "publisher": "GiveWell Blog",
   "year": "2018",
   "url": "https://blog.givewell.org/2018/02/13/revisiting-leverage/"
  },
  {
   "id": "mayne2008",
   "title": "Contribution analysis: An approach to exploring cause and effect (Mayne), ILAC Brief 16",
   "publisher": "Institutional Learning and Change Initiative (CGIAR)",
   "year": "2008",
   "url": "https://www.betterevaluation.org/sites/default/files/ILAC_Brief16_Contribution_Analysis.pdf"
  },
  {
   "id": "bailey2017",
   "title": "Persistence and Fadeout in the Impacts of Child and Adolescent Interventions (Bailey, Duncan, Odgers and Yu)",
   "publisher": "Journal of Research on Educational Effectiveness",
   "year": "2017",
   "url": "https://pmc.ncbi.nlm.nih.gov/articles/PMC5779101/"
  },
  {
   "id": "hamory2021",
   "title": "Twenty-year economic impacts of deworming (Hamory, Miguel, Walker, Kremer and Baird)",
   "publisher": "Proceedings of the National Academy of Sciences",
   "year": "2021",
   "url": "https://doi.org/10.1073/pnas.2023185118"
  },
  {
   "id": "rp_discount2023",
   "title": "A review of GiveWell’s discount rate",
   "publisher": "Rethink Priorities",
   "year": "2023",
   "url": "https://rethinkpriorities.org/research-area/a-review-of-givewells-discount-rate/"
  },
  {
   "id": "strathern1997",
   "title": "‘Improving ratings’: audit in the British University system (Strathern)",
   "publisher": "European Review",
   "year": "1997",
   "url": "https://www.cambridge.org/core/journals/european-review/article/improving-ratings-audit-in-the-british-university-system/FC2EE640C0C44E3DB87C29FB666E9AAB"
  },
  {
   "id": "bevan_hood2006",
   "title": "What’s measured is what matters: targets and gaming in the English public health care system (Bevan and Hood)",
   "publisher": "Public Administration",
   "year": "2006",
   "url": "https://doi.org/10.1111/j.1467-9299.2006.00600.x"
  },
  {
   "id": "mangel1984",
   "title": "Abraham Wald’s Work on Aircraft Survivability (Mangel and Samaniego)",
   "publisher": "Journal of the American Statistical Association",
   "year": "1984",
   "url": "https://doi.org/10.1080/01621459.1984.10478038"
  },
  {
   "id": "givewell_ce",
   "title": "Cost-Effectiveness (how GiveWell uses and limits its estimates)",
   "publisher": "GiveWell",
   "year": "2025",
   "url": "https://www.givewell.org/how-we-work/our-criteria/cost-effectiveness"
  },
  {
   "id": "hli_overview",
   "title": "Our Methodology, chapters 2 and 3: how we evaluate charities and calculate cost-effectiveness",
   "publisher": "Happier Lives Institute",
   "year": null,
   "url": "https://www.happierlivesinstitute.org/research/charity-evaluation-methodology/"
  },
  {
   "id": "ace_criteria",
   "title": "Evaluation Criteria 2025",
   "publisher": "Animal Charity Evaluators",
   "year": "2025",
   "url": "https://animalcharityevaluators.org/charity-reviews/evaluating-charities/evaluation-criteria/"
  }
 ]
}