Did it work? The donor's tasting card
Taste the result, not the label.
The rules we use to judge whether giving worked, turned into questions you can put to any charity, annual report or “studies show” claim. Print it, keep it by your desk, take it to the next gala.
If you ask only one
Compared with what?
Every honest answer to “did it work?” is a comparison with what would have happened anyway. In our view, a charity that can answer this one clearly is already worth a closer look.
Fourteen questions in four groups. Each has a good sign to listen for and a warning sign to watch for.
First Did it work at all?
-
Compared with what?
Good sign A group that didn’t get the help, chosen the same way, ideally by lottery.
Watch for Only a before-and-after, or people who joined compared with people who chose not to.
Our rule: No counterfactual, no claim.
-
How was it tested?
Good sign A randomised trial, or a natural experiment that comes close, named and linked.
Watch for Adjectives where a design should be: “proven”, “evidence-based”, “transformative”.
Our rule: Design beats adjectives.
-
Who was chosen, and when?
Good sign People or places picked by a rule, not because things looked worst that month.
Watch for Help aimed at the worst cases, then a big improvement and no comparison group.
Our rule: Ask who was chosen, and when.
-
Is it too good to be true?
Good sign A large effect that has been repeated, or a modest one from a big study.
Watch for One small study with a spectacular result. Expect it to shrink on a second tasting.
Second How sure can anyone be?
-
What is the range, not just the headline?
Good sign An estimate shown with its uncertainty, or the evaluator’s own certainty label.
Watch for One precise-looking number with no range, no source and no date.
Our rule: Read the interval, not the point.
-
How many studies were run, not just published?
Good sign Trials registered before they began, and an evaluator that says how much it discounts for what is missing.
Watch for Only the winners on show.
Our rule: Ask how many studies were run.
-
Did the studies agree?
Good sign Results pointing the same way in most places, and a stated range for a new one.
Watch for One tidy average that hides places where it did nothing.
Our rule: Ask for the range, not just the average.
-
Was the group it “works best for” chosen in advance?
Good sign Groups named in a plan written before the data came in, and compared directly.
Watch for “It works especially well for…”, discovered after the fact.
Our rule: Decide the slices before you look.
Third Will it work here, and for how long?
-
Will it travel to where my money goes?
Good sign Tested somewhere like the place you are funding, with a clear reason why it works.
Watch for A result from another country, decade or delivery team, assumed to carry over.
Our rule: Ask whether it travels.
-
How long does the benefit last?
Good sign Follow-up after the programme ended, saying whether the effect faded, held or grew.
Watch for Results measured on the last day, then treated as permanent.
Our rule: Mind the clock.
-
Who and what is missing?
Good sign A count of the people who dropped out, and measures that are hard to game.
Watch for A target that improved while the thing it stood for didn’t; success stories only.
Our rule: Count the missing.
Fourth What does my gift add?
-
What does one good outcome cost, and how sure is that?
Good sign An evaluator’s estimate with its period and assumptions, called an estimate.
Watch for A price for a life, “your gift saves a child”, said as a promise.
Our rule: Count every link.
-
Would it have happened without my gift?
Good sign An honest account of what other funders would have paid for anyway.
Watch for Every funder claiming the whole result.
-
Will we know in a year?
Good sign A promise to report back, bad news included, and your own note of what you expected.
Watch for No plan to look again.
Our rule: Write it down, then check.
How to use it
- Before any gift
- Ask the first four questions. Any charity that measures its results should be able to answer them.
- For a large gift
- Work through the whole card, and read the independent evaluator’s own report alongside the charity’s.
- A year later
- Come back to the last question. That is when the honest answer arrives.