Did it work? Here’s how anyone could know.

Statisticians write the problem down in one line. For any person, there are two possible outcomes: the one they would have with the programme and the one they would have without it. The effect on that person is the difference.12

$$\tau_i \;=\; Y_i(1) \;-\; Y_i(0)$$

In words The effect on person i is their outcome with the programme, Y_i(1), minus their outcome without it, Y_i(0). 12

Here is the catch. Each person lives only one of those lives, so we only ever see one of the two outcomes. Paul Holland called this the fundamental problem of causal inference: it is impossible to observe both outcomes for the same person, and so impossible to observe the effect on that person directly.2

Because individual effects are invisible, careful researchers estimate an average effect for a group instead. That is the honest unit of evidence, and it is what the figures on this site describe.23

$$\text{ATE} \;=\; \mathrm{E}\big[\,Y(1) - Y(0)\,\big]$$

In words The average treatment effect is the average, across the people concerned, of the difference between the two outcomes. 2

The tempting shortcut is to compare people who got the programme with people who didn’t. That comparison mixes two things: the real effect, and how different the two groups were to begin with. Programmes are rarely handed out at random. Keen families sign up first; officials send help where it is already working. J-PAL calls this the selection problem.34

$$\begin{gathered}\underbrace{\mathrm{E}[Y\mid D{=}1]-\mathrm{E}[Y\mid D{=}0]}_{\text{the gap you see}}\\=\;\underbrace{\mathrm{E}[Y(1)-Y(0)\mid D{=}1]}_{\text{the effect on those who took part}}\\+\;\underbrace{\mathrm{E}[Y(0)\mid D{=}1]-\mathrm{E}[Y(0)\mid D{=}0]}_{\text{selection bias}}\end{gathered}$$

In words The gap between participants (D = 1) and everyone else (D = 0) equals the true effect on participants plus how differently the two groups would have done anyway. Only the first part is the programme’s doing. 34

Before-and-after comparisons have the same flaw in time rather than across people: things change anyway, through harvests, prices, other programmes and plain luck. A gift that “worked” has to beat the world that would have happened without it, not the world as it was the year before.35

Tasting noteNo counterfactual, no claim. Every verdict here starts with one question: compared with what?

Randomised trials. A lottery decides who gets the programme. Because chance, not choice, splits the groups, they are alike on average in everything, measured or not, and the gap in outcomes can be credited to the programme. J-PAL describes randomisation as the way to get an unbiased estimate of causal impact, and the 2019 economics Nobel recognised how far this method has changed development policy.3

Trials have limits that matter to donors. They need large enough samples to see an effect at all. They can’t be run on whole-country policies, and spillovers to the comparison group can blur the result. A result from one place is also not automatically a result for another; part 2 of this page will take that up.36

Natural experiments. Sometimes the world runs the lottery for us: a policy changes on one side of a border, an eligibility cut-off splits similar people, or an accident of the calendar nudges some people and not others. The 2021 economics Nobel went to David Card, Joshua Angrist and Guido Imbens for showing how to learn cause and effect from such situations, and exactly which conclusions they support.54

$$\begin{gathered}\hat\tau_{\text{DiD}} = \big(\bar Y_{T,\,\text{after}} - \bar Y_{T,\,\text{before}}\big)\\-\;\big(\bar Y_{C,\,\text{after}} - \bar Y_{C,\,\text{before}}\big)\end{gathered}$$

In words Take the change in the group that got the programme (T) and subtract the change, over the same period, in a similar group that didn’t (C). It works only if the two groups would have moved in parallel without the programme. Card and Krueger used this logic to study a minimum-wage rise in New Jersey against neighbouring Pennsylvania. 5

$$\hat\tau_{\text{RD}} = \lim_{x\,\downarrow\,c}\mathrm{E}[Y\mid X{=}x] \;-\; \lim_{x\,\uparrow\,c}\mathrm{E}[Y\mid X{=}x]$$

In words Compare people just above an eligibility cut-off c with people just below it. They are nearly identical except for the programme. The answer holds near the cut-off, and only if nothing else changes at exactly that line. 35

$$\text{LATE} = \frac{\mathrm{E}[Y\mid Z{=}1]-\mathrm{E}[Y\mid Z{=}0]}{\mathrm{E}[D\mid Z{=}1]-\mathrm{E}[D\mid Z{=}0]}$$

In words When an as-good-as-random nudge Z (an offer, a lottery, a birth date) changes who takes part, divide its effect on the outcome by its effect on participation. Angrist and Imbens showed this measures the effect only for the people whose behaviour the nudge changed. 5

Observational studies. These compare people who happened to receive something with people who didn’t, and adjust for the differences researchers can measure. They are cheap, fast and often the only option, but they can’t adjust for what nobody measured. That is where they fail, sometimes badly.34

Tasting noteDesign beats adjectives. A modest result from a trial outranks a dazzling one from a comparison that can’t say who would have done well anyway.

Relative and absolute effects. The same result can be told two ways. Cochrane’s review of insecticide-treated bed nets reports both, and both matter: one says how strong the protection is, the other how many lives that adds up to.12

$$\hat\theta \;\pm\; 1.96 \times \mathrm{SE}(\hat\theta)$$

In words The estimate plus or minus about two standard errors. If a study like this were repeated many times, intervals built this way would contain the true value about 95 times in 100. When the interval for a ratio includes 1, or for a difference includes 0, the data are consistent with no effect at all. 126

Statistical power. Power is the chance that a study detects an effect that is really there. By convention researchers aim for 80 percent at a 5 percent significance level. The smallest effect a study can reliably see, its minimum detectable effect, shrinks only with the square root of the sample: to see an effect half as large you need about four times as many people.6

$$\text{MDE} = \big(t_{1-\kappa} + t_{\alpha/2}\big)\sqrt{\frac{1}{P(1-P)}\cdot\frac{\sigma^2}{N}}$$

In words The smallest true effect a study can expect to detect depends on the power you want (1 − κ), the significance level (α), the share of people treated (P), how much the outcome varies (σ²) and the sample size (N). 6

Underpowered studies are common. One review estimated the median statistical power of neuroscience studies at between about 8 and 31 percent, and showed that low power doesn’t only miss real effects: it makes the effects that are reported less likely to be true, and larger than reality.13

The winner’s curse. When a small, noisy study clears the bar for statistical significance, it usually does so by overshooting. Andrew Gelman and John Carlin call the two dangers Type S (the sign is wrong) and Type M (the size is exaggerated), and show that exaggeration becomes a serious problem once power falls below about one in two.14

$$1-\beta \;=\; \Phi\big(\tfrac{D}{s} - 1.96\big) + \Phi\big(-\tfrac{D}{s} - 1.96\big)$$

In words The chance that a study’s estimate lands more than 1.96 standard errors from zero, on either side, when the true effect is D and the standard error is s. Φ is the standard normal distribution. Only the ratio D / s matters: a true effect 2.8 standard errors from zero gives the conventional 80 percent power. 146

$$\text{Exaggeration} \;=\; \frac{\mathrm{E}\big[\,|\hat\theta| \mid |\hat\theta| > 1.96\,s\,\big]}{D}$$

In words Among the estimates that clear the bar for significance, how much larger they are than the true effect D, on average. Gelman and Carlin also track the Type S rate: the share of significant estimates whose sign is wrong. 14

At the conventional 80 percent power, Gelman and Carlin find the risk of a wrong sign is negligible (about 1 in a million) and a significant estimate overstates the truth by only about 1.12 times. The danger is all at the low end, which is exactly where small, cheap studies of new programmes tend to sit.14

Try it · Illustrative numbers, not any charity’s results

Was the study big enough?

A trial puts half its people in a programme and half in a comparison group. Say the programme truly improves an outcome by 0.2 standard deviations, and the trial enrols 200 people. Change either number to see what the study can and can’t tell you.

Chance the trial detects the effect (its power): 29%.

Below one in two, a significant result is likely to overstate the effect badly. The smallest effect this trial can reliably see is 0.4 standard deviations.

• Power29%

The line marks 80%, the power trial designers usually aim for.

If it does come out significant

The reported effect will overstate the truth about 1.8 times on average, and the chance it points the wrong way is 0.1%. To reach 80% power for this effect, the trial would need about 785 people.

Assumes two equal groups, a roughly normal estimate and the usual 5% two-sided threshold. Exaggeration and wrong-sign chances follow Gelman and Carlin. 146

These are the example’s numbers. With JavaScript on, you can change them.

The file drawer. What gets published is not a fair sample of what was found. In a known set of 221 social-science experiments, strong results were 40 percentage points more likely to be published than results showing no effect, mostly because those were never written up.15 When 100 published psychology studies were repeated, 97 percent of the originals had been statistically significant but only 36 percent of the repeats were, and the repeated effects were on average half as large.16

John Ioannidis turned these forces into a formula for how likely a “statistically significant” finding is to be true. It depends on the pre-study odds that the idea is right (R), on power (1 − β) and on the significance threshold (α). Bias and many teams chasing the same question push it lower still.17

$$\text{PPV} = \frac{(1-\beta)\,R}{R - \beta R + \alpha}$$

In words The positive predictive value of a significant result, before any bias. A finding is more likely true than false only when (1 − β)R is larger than α. 17

Try it · Illustrative numbers, not any charity’s results

How often is a significant finding true?

Picture a field where 1 idea in 10 really works, studied with 80% power at the usual 5% threshold. Change the numbers to see how often a “significant” result is real.

A significant result is real 64% of the time.

Of every 1,000 studies, 100 test ideas that work and 80 of those come out significant. The other 900 test ideas that don’t, and 45 of those come out significant anyway. So there are 125 significant results, and only the first group is real.

• Real, among significant results64%

Before any bias. Ioannidis shows that bias, and many teams chasing the same question, push the share lower still. 17

These are the example’s numbers. With JavaScript on, you can change them.

Tasting noteRead the interval, not the point. And when a small study reports a spectacular effect, expect the second tasting to be quieter.

GiveWell made the case in 2011 with an everyday example: a restaurant with three glowing reviews shouldn’t outrank one with two hundred very good ones. A sensible judge starts from what is typical (the prior) and lets each piece of evidence pull the view only as far as its precision earns.18

When both the prior and the estimate’s error are roughly bell-shaped, the arithmetic is exact. The updated view is an average of the two, each weighted by its precision (one over its variance), and the updated uncertainty is smaller than either.18

$$\mu_{\text{post}} \;=\; \frac{\mu_p/\sigma_p^2 \;+\; \hat\theta/\sigma_e^2}{1/\sigma_p^2 \;+\; 1/\sigma_e^2}$$

In words μ_p and σ_p² are the prior’s centre and variance; θ̂ and σ_e² are the new estimate and the variance of its error. The more precise side gets more weight. 18

$$\sigma_{\text{post}}^2 \;=\; \left(\frac{1}{\sigma_p^2} + \frac{1}{\sigma_e^2}\right)^{-1}$$

In words Precisions add. The combined view is always at least as precise as the more precise of the two inputs. 18

$$\mu_{\text{post}} = \mu_p + w\,(\hat\theta - \mu_p), \qquad w = \frac{\sigma_p^2}{\sigma_p^2 + \sigma_e^2}$$

In words Start at the prior and move a fraction w of the way toward the new estimate. When the estimate is noisy (σ_e large), w is small and you barely move. 18

Where it comes from Multiply the numerator and denominator of the updated-estimate formula by σ_p²σ_e² and rearrange.

Try it · Illustrative numbers, not any charity’s results

How far should one study move you?

Say similar programmes usually improve an outcome by about 2 points, give or take 2. A new study reports 10 points, with a standard error of 5. Change any number to see how much the new study earns.

Updated estimate: 3.1 points, give or take 1.86.

The study gets 14% of the weight, because its uncertainty is large next to what we already know.

Prior 2 Updated 3.1 Study 10

These are the example’s numbers. With JavaScript on, you can change them.

The same formula explains why extravagant claims deserve the least weight when they are least grounded. GiveWell’s example: with a prior centred on zero (standard deviation 1) and a claim of size X whose own error is as large as X, the updated expectation is X / (X² + 1), which gets smaller as the claim gets bigger.18

$$\mathrm{E}[\,\text{value}\mid \text{claim } X\,] = \frac{X}{X^2+1}$$

In words Prior: mean 0, standard deviation 1. Claim: X, with an error whose standard deviation is also X. 18

GiveWell is candid about the limits: the hard part is choosing the prior, because it isn’t obvious what “typical” should mean for a charity. That is a reason to state priors openly, not to drop them. Its practical rule follows: the more an estimate asks of you, the more evidence it should have, and conclusions supported by several independent lines of analysis deserve more weight.18

Tasting noteOne dazzling glass from an unknown cellar earns a second tasting, not a medal.

Every cost-effectiveness estimate has the same skeleton: count the benefits, which means estimating how many people are reached, how many actually benefit and by how much, then divide the cost by that total. GiveWell describes a published deworming estimate built exactly this way.19

$$\text{cost per outcome} \;=\; \frac{C}{N \times u \times e}$$

In words C is the total cost, N the people reached, u the share who actually use the help, and e the chance it changes the outcome for a user, compared with what would have happened anyway. Real models have many more links. 1920

Because the links multiply, their errors multiply too. If two links are each 50 percent too high, the result isn’t 50 percent too high: it is 125 percent too high, because 1.5 × 1.5 = 2.25. On a logarithmic scale the multiplications become additions, and for independent links the spreads add in a simple way.18

$$\sigma^2_{\ln(\text{result})} \;=\; \sum_{i} \sigma^2_{\ln x_i}$$

In words If each link x_i is uncertain by a multiplicative factor, the spread of the result on a log scale is found by adding the squared log-spreads of the links. One very shaky link can dominate the total. 18

Where it comes from The log of a product is the sum of the logs, and the variance of a sum of independent quantities is the sum of their variances.

Try it · Illustrative numbers, not any charity’s results

How uncertainty multiplies

A $100,000 programme. Each link in the chain is a range you think is about 90% likely to contain the truth. Narrow or widen any range and watch which link sets the width of the answer.

Central estimate: $761 per outcome.

A reasonable 90% range is $349 to $1,659, a factor of 2.2 either way, if the links are independent. Every link at its best gives $231; every link at its worst gives $2,500.

Where the uncertainty comes from

• People reached7%
• Share who actually use it14%
• Chance it changes the outcome for a user79%

These are the example’s numbers. With JavaScript on, you can change them.

The lesson GiveWell drew is the one this site is built on: transparency over trust, however pedigreed the source. It now publishes its models, and still calls its own cost-effectiveness numbers “extremely rough” estimates that “should not be taken literally.” They also rest on value choices, such as how to weigh income against lives; that is a choice, not a fact, and part 2 of this page will treat it as one.1920

Tasting noteCount every link, and find the shakiest one. It sets the width of the whole range.

Part 1’s question, “compared with what?”, has an awkward edge when many hands are needed. If a programme needs both a drug donor and a delivery funder, each one can claim the whole result, because without either nothing happens. Lloyd Shapley’s 1952 value for cooperative games gives a fair way to split a joint result: average each party’s added contribution over every order in which the parties could have joined.25

$$\begin{aligned}\varphi_i \;=\; &\sum_{S \subseteq N \setminus \{i\}} \frac{|S|!\,\big(n-|S|-1\big)!}{n!} \\ &\times \Big[v\big(S \cup \{i\}\big) - v(S)\Big]\end{aligned}$$

In words N is the set of n funders and v(S) is what a group S achieves on its own. Funder i’s share is its added contribution to each group it could join, averaged over all the orders of joining. The shares always add up to exactly the whole result, never more. 25

GiveWell handles the same problem through what it calls leverage (a charity drawing in other money) and funging (a charity’s money displacing spending that would have happened anyway). Since 2017 it has made explicit judgements about how likely its charities are to change what governments and agencies spend, and about the value of what that money would otherwise have bought. It calls one key input, government spending at about 75% as cost-effective as cash transfers, “a very rough guess”.26

It matters. In GiveWell’s 2017 update, moving from “count all costs equally” to its leverage-and-funging method changed its estimates for its top charities by factors from about 0.8 (a charity judged to crowd out some other funding) to about 2 (one whose programmes drew heavily on donated drugs and government time).26

When there is no trial at all, as with most advocacy and system-wide work, evaluators fall back on contribution analysis. John Mayne’s six steps: set out the cause-and-effect question, write down the theory of change and its risks, gather the existing evidence, assess the resulting “contribution story” and its weak points, seek more evidence, and revise. The result is a reasoned case that the work made a difference, not a measured share of the credit.27

Tasting noteMany hands make a vintage. Credit the grower, the picker and the cellar, but don’t let them each claim the whole bottle.

Many effects fade. In the Perry Preschool programme, the effect on IQ was 0.75 standard deviations at age 5 and a statistically insignificant 0.08 by age 8. Across 67 high-quality early-childhood studies, effects on thinking skills averaged 0.23 standard deviations at the end of the programme and 0.10 within a year. Yet some of the same programmes showed substantial benefits in adulthood, on different outcomes.28

Some effects show up only decades later. A randomised school deworming programme in Kenya followed its pupils for 20 years, finding and interviewing 84% of them. Those who got two to three extra years of treatment had 14% higher consumption and 13% higher hourly earnings as adults. The consumption estimate is imprecise (p = 0.058), and most of the effect was among men and older pupils.29

So the shape over time matters as much as the starting size. The Happier Lives Institute models it directly: it estimates the effect just after treatment and how fast it changes each year, usually assuming a straight-line decline to zero. The total benefit is the area under that line.24

$$\begin{aligned}\text{Total} &\;=\; \int_{0}^{t_{\text{end}}} \big(b_0 + b_1 t\big)\,dt \\ &\;=\; \frac{b_0^{2}}{2\,|b_1|} \qquad (b_1 < 0)\end{aligned}$$

In words b₀ is the effect right after the programme and b₁ how much it changes per year (negative when it fades). The effect lasts t_end = b₀ / |b₁| years, so the total is the area of a triangle. 24

Where it comes from The line reaches zero at t_end = b₀ / |b₁|. The area of a triangle with height b₀ and base t_end is ½ × b₀ × b₀ / |b₁|.

Benefits in the future are also usually discounted: counted for somewhat less than the same benefit today, because circumstances improve and the future is uncertain. A 2023 review by Rethink Priorities reports that GiveWell uses 4% a year, built from 1.7% for improving circumstances, 1.4% for uncertainty about the future and 0.9% for benefits that compound, with no discount at all for time itself.30

$$\text{Present value} \;=\; \sum_{t=0}^{T} \frac{B_t}{(1+r)^{t}}$$

In words B_t is the benefit in year t and r the yearly discount rate. At 4%, a benefit 20 years away counts for about 46% of the same benefit today, because 1 ÷ 1.04²⁰ ≈ 0.46. 30

Systemic change (new laws, new norms, a government taking a programme over) is the hardest case: there is one country, one history and no control group. GiveWell says its models do not explicitly capture the long-term financial sustainability of a programme, such as later government take-up, and that it weighs this qualitatively instead. Our stance follows: a claim of systemic change needs a contribution story you can check, step by step. Without one it is a hope, and we label it as not yet judged.2627

Tasting noteSome wines are glorious on release and flat in a year; a few only open after a decade. Taste again before you judge.

The anthropologist Marilyn Strathern gave the economist Charles Goodhart’s observation its famous general form: “When a measure becomes a target, it ceases to be a good measure.” People are clever; reward a number and you will get the number, whether or not you get the thing it stood for.31

Gwyn Bevan and Christopher Hood studied the targets England set for its public health service in the 2000s. Alongside real reported successes they documented problems of measurement and gaming, and named the hidden assumption: synecdoche, taking a part to stand for the whole. Their remedies were stronger audits of the data and targets that are harder to game.32

In giving, the part that stands for the whole is often an output: nets handed out, people trained, sessions held. Outputs are worth counting, but they are not outcomes. That is why the evaluators we quote model the chain from output to outcome, and why we show their figures for outcomes, not activity counts.2024

The second trap is quieter. During the Second World War the statistician Abraham Wald worked out how vulnerable each part of an aircraft was, using only the damage seen on planes that came back. The planes that didn’t return carried the most important information, and his method reasoned about them explicitly.33

$$\mathrm{E}[\,Y \mid S=1\,] \;=\; \mathrm{E}[Y] \;+\; \frac{\operatorname{Cov}(Y, S)}{\Pr(S=1)}$$

In words Y is the outcome and S = 1 means “still in the data” (the plane returned, the participant answered the survey, the study was published). If staying in the data is linked to the outcome, the average among those who stayed is shifted by that link. 33

Where it comes from S is 0 or 1, so E[Y | S = 1] = E[YS] / Pr(S = 1), and E[YS] = E[Y] E[S] + Cov(Y, S) with E[S] = Pr(S = 1).

The same trap works at the level of whole studies (the file drawer in section 3) and whole charities: the programmes we hear about are the ones still running. Our own defence is the follow-up record: we store each estimate with its date before the result is known and check it again later, so the misses stay on the record alongside the hits.3315

Cellar noteA cellar that only keeps the bottles that aged well will always look like a great cellar. Count the corks you poured away.

Everything above is what anyone would need in order to know whether a gift worked. Here is how the three evaluators we quote put those ideas into practice, in their own published words. GiveWell counts lives saved and income, the Happier Lives Institute counts wellbeing, and Animal Charity Evaluators counts animal suffering averted, so their numbers sit on different scales and we never put them on one.343536

What we take from it. In our reading, the overlap is the reassuring part. All three ask what would have happened anyway, adjust study results for a new setting, distrust effects that look too good, and say where judgement stands in for data. The differences are real too. GiveWell builds a model for each programme and applies explicit, admittedly rough adjustments; the Happier Lives Institute pools studies in meta-analyses and simulates the uncertainty; Animal Charity Evaluators leans more on a reasoned theory of change, because large studies are rare in its field, and declines to put a number on benefits it can’t sensibly estimate. That is why we quote each in its own words and units, and never turn their verdicts into a score of our own.343536

Tasting noteThree tasters, three palates, one discipline: judge what changed, name the doubt, and never mistake your own estimate for a promise.