Article: How to Translate Research Into Practice
Reviewed: November 2026 · Method version 2 · How these reviews work
Each factual claim in the article was checked against the source it cites, then challenged by reviewers whose job was to find the strongest evidence against it: replication attempts, meta-analyses, later papers that cite the original work, and research from other fields that study the same mechanism. A separate reviewer weighed both sides, and a script re-checked every quoted source. Where a claim was overstated or misattributed, the article was corrected. The research below supports the article as it now reads.
Christopher Bryan, Elizabeth Tipton and David Yeager argue that most treatment effects are heterogeneous, so results should be expected to vary across studies whenever researchers don’t sample and test for moderators systematically. The point is still debated: in close replications of lab findings, effects varied little, but larger effects varied more.
Multi-lab replications in psychology found little heterogeneity when the population and setting changed only slightly, and more as effects grew larger (Olsson-Collentine et al., 2020). Real product contexts differ far more than those replications, so the article’s practical advice still holds.
That is still a real, highly significant effect, at about a sixth of the size in percentage points, or about a quarter in relative terms.
The ratio between the two samples is about one sixth in percentage points (1.4 vs 8.7) and about one quarter in relative increase (8.0% vs 33.4%).
Experts are not much better at guessing which ideas will work. In a megastudy, 30 scientists designed 54 four-week digital programs to get 61,293 members of an American gym chain exercising more. Of the interventions, 45% significantly increased weekly gym visits, by 9% to 27%. When the researchers later asked public health professors, behavioral science practitioners and members of the public to predict the results, the average prediction was 9.1 times too optimistic, and none of the three groups reliably picked the winners.
The megastudy’s forecasters were third parties (301 online participants, 156 public health professors and 90 practitioners), not the scientists who designed the programs. Work in experimental economics finds experts forecast average effects well but rank treatments no better than non-experts (DellaVigna and Pope, 2018).
Participants received a bonus of 125 points, worth about 9 cents, for returning to the gym after a missed workout. It produced an estimated 0.40 more weekly visits per person, a 27% increase over the placebo condition, though the authors note that estimates for the top performers are probably inflated because they were picked for being the largest.
Milkman and colleagues caution that the top-ranked arms’ effects are probably inflated by selection (the winner’s curse), which adds to the case for discounting the forecast.
For the forecast, apply the discount the evidence suggests. In DellaVigna and Linos’s data, nudge-unit effects were about a sixth of the journal average in percentage points and about a quarter in relative terms, mostly because the journals published the bigger results. Plan for an effect well below the published one and size the test to detect it.
The size of the discount varies by field: scaled-up physical activity programs kept a median of about 59% of their pilot effect (Lane et al., 2021), and published education trials showed about twice the effect of unpublished or large ones (Cheung and Slavin, 2016).
Pilot on a representative slice. Early adopters and heavy users can respond differently from everyone else, sometimes more strongly and sometimes less.
In Opower the early, high-usage households responded more strongly; in other settings the direction can reverse, so a pilot on heavy users is a poor guide in either direction.
These edits were made on November 12, 2026 after the review.
Claim reviewed: Bryan, Tipton and Yeager argue that most treatment effects are heterogeneous, so results should be expected to vary across studies whenever researchers don’t sample and test for moderators systematically.
Now reads: Christopher Bryan, Elizabeth Tipton and David Yeager argue that most treatment effects are heterogeneous, so results should be expected to vary across studies whenever researchers don’t sample and test for moderators systematically. The point is still debated: in close replications of lab findings, effects varied little, but larger effects varied more.
Why: The article reports Bryan, Tipton and Yeager’s argument accurately and frames it as their view. The premise is debated, though: a review of 68 multi-lab direct replications found little heterogeneity, while heterogeneity became more likely as effects got larger. A short note would keep readers from taking ‘most effects vary’ as settled. Now cites Olsson-Collentine, Wicherts & van Assen, Psychological Bulletin (2020).
Claim reviewed: The nudge-unit effect is still real and highly significant, at about a sixth of the size of the journal effect.
Now reads: That is still a real, highly significant effect, at about a sixth of the size in percentage points, or about a quarter in relative terms.
Why: “Highly significant” is the paper’s own phrase. “About a sixth” is the article’s arithmetic and holds only in percentage points (1.4 vs 8.7). In relative terms the nudge-unit effect is about a quarter of the journal effect (8.0% vs 33.4%), so the sentence should say which measure it means.
Claim reviewed: Experts are not much better at guessing which ideas will travel: on average, predictions about the interventions’ benefits were 9.1 times too optimistic, and forecasts by impartial judges failed to predict which interventions would be most effective.
Now reads: Experts are not much better at guessing which ideas will work. In a megastudy, 30 scientists designed 54 four-week digital programs to get 61,293 members of an American gym chain exercising more. Of the interventions, 45% significantly increased weekly gym visits, by 9% to 27%. When the researchers later asked public health professors, behavioral science practitioners and members of the public to predict the results, the average prediction was 9.1 times too optimistic, and none of the three groups reliably picked the winners.
Why: The 9.1x figure and the ‘impartial judges’ quote are accurate. But the article places them right after the 30 scientists who designed the programs, which suggests those scientists made the forecasts. They did not: the predictions came from lay online participants, public health professors and behavioral science practitioners, asked 14 months later, and the 9.1x pools all three. Other studies are mixed on experts: they predict average effects reasonably well but do no better than non-experts at ranking which treatment wins.
Claim reviewed: The best performer gave a bonus of 125 points, worth about 9 cents, for returning to the gym after a missed workout, producing an estimated 0.40 more weekly visits per person, a 27% increase over placebo.
Now reads: Participants received a bonus of 125 points, worth about 9 cents, for returning to the gym after a missed workout. It produced an estimated 0.40 more weekly visits per person, a 27% increase over the placebo condition, though the authors note that estimates for the top performers are probably inflated because they were picked for being the largest.
Why: The numbers are exact. But the source warns, in the sentence just before, that the top performers’ estimates are probably inflated because they were chosen for being the largest. The worked example builds on this arm, so readers should see that warning.
Claim reviewed: People who volunteer for a habit program are likely to be motivated already.
Now reads: People who volunteer for a habit program are often already engaged: StepUp’s participants were visiting the gym about 1.3 times a week before the study began.
Why: The claim is uncited and hedged. The megastudy did not measure motivation, but its participants were already going to the gym about 1.3 times a week before joining, which fits the idea without proving it. No source found contradicts it. Now cites Milkman et al., Megastudies improve the impact of applied behavioural science, Nature (2021).
Claim reviewed: The average gap in DellaVigna and Linos’s data was roughly sixfold, so teams should plan for an effect well below the published one and size the test to detect it; a test powered only for the journal’s number will probably come back flat.
Now reads: For the forecast, apply the discount the evidence suggests. In DellaVigna and Linos’s data, nudge-unit effects were about a sixth of the journal average in percentage points and about a quarter in relative terms, mostly because the journals published the bigger results. Plan for an effect well below the published one and size the test to detect it.
Why: The advice to expect a smaller effect and power the test for it is well supported: scaled-up physical activity programs keep less than 60% of their effect, and published education trials show about twice the effect of large ones. The specific ‘sixfold’ discount is shakier. It holds only in percentage points (about fourfold in relative terms), and DellaVigna and Linos attribute most of the gap to selective publication, so it measures inflation in the literature rather than loss in a new setting.
Claim reviewed: Early adopters and heavy users respond differently from everyone else, so piloting on your most engaged users repeats the early-Opower problem.
Now reads: Pilot on a representative slice. Early adopters and heavy users can respond differently from everyone else, sometimes more strongly and sometimes less.
Why: Allcott’s data back the core warning: a pilot on a non-representative group can mislead about everyone else. But in Opower the heavy users responded more because they had more to save, and in other settings the most engaged users may respond less, since past behavior changes how an intervention works. ‘Differently’ is right; readers should not assume the direction.