Did it work? The chain card
Eight questions between a gift and “it worked”
Take any claim that a gift worked and walk it down these eight links, in order. Each link gets its own answer and its own level of certainty. Where nobody can answer, that is the weakest link, and it sets the headline.
-
L0 Defined
Worked at what, compared with what, for whom, by when?
Ask Compared with what?
Rule No counterfactual, no claim.
-
L1 Happened
Was it delivered, and did the measured outcome move, counting the missing and checking the measure was not gamed?
Ask Who and what is missing?
Rule Count the missing.
-
L2 Caused
Would it have happened anyway?
Ask How was it tested? Who was chosen, and when?
Rule Design beats adjectives. Ask who was chosen, and when.
-
L3 Sized
How big is the effect, honestly?
Ask What is the range, not just the headline? Is it too good to be true? How many studies were run, not just published? Did the studies agree? Was the group it “works best for” chosen in advance?
Rule Read the interval, not the point. Small, noisy and surprising means shrink it. Ask how many studies were run. Ask for the range, not just the average. Decide the slices before you look.
-
L4 Travels
Will it work here, run by this organisation, at this scale, now?
Ask Will it travel to where my money goes?
Rule Ask whether it travels.
-
L5 Worth it
What good per dollar, in a unit someone values, for how long?
Ask What does one good outcome cost, and how sure is that? How long does the benefit last?
Rule Count every link. Mind the clock.
-
L6 Adds
Will my marginal dollar change what happens?
Ask Would it have happened without my gift?
Rule Credit can’t add up to more than the result.
-
L7 Checked
Was a prediction written down, and has anyone looked again?
Ask Will we know in a year?
Rule Write it down, then check.
How to read a claim with it
- Report every link on its own. Never add them up into a score or count how many look strong.
- The weakest link sets the headline: the first one, in order, whose evidence is unknown, contested or only the organisation’s own estimate.
- A strong link never lifts a weak one.
- Never fill a blank with a guess of your own. “Nobody knows yet” is an answer.