attck-research

Evidence review: Reading Research Without Getting Lost

Article: Reading Research Without Getting Lost
Reviewed: November 2026 · Method version 2 · How these reviews work

How this review works

Each factual claim in the article was checked against the source it cites, then challenged by reviewers whose job was to find the strongest evidence against it: replication attempts, meta-analyses, later papers that cite the original work, and research from other fields that study the same mechanism. A separate reviewer weighed both sides, and a script re-checked every quoted source. Where a claim was overstated or misattributed, the article was corrected. The research below supports the article as it now reads.

Background research

Two Large Attempts to Check the Record

In 2015, the Open Science Collaboration published replications of 100 studies from three psychology journals, using high-powered designs and the original materials where available.

Confirmed against the abstract of the Open Science Collaboration paper.

97% of the original studies had statistically significant results, 36% of the replications did, and replication effects were on average half the size of the originals.

The figures match the Open Science Collaboration abstract. A published critique (Gilbert et al., 2016) disputes what the figures mean, not the figures themselves; later large replication projects found similar or larger declines.

Camerer and colleagues replicated 21 social science experiments published in Nature and Science between 2010 and 2015, with samples on average about five times larger than the originals.

Confirmed against the Camerer et al. (2018) abstract.

Thirteen of the 21 replications (62%) found a significant effect in the same direction, and effect sizes again came out at about half.

Confirmed against the Camerer et al. (2018) abstract.

Prediction markets put the replication rate at 63.4% and surveys at 60.6%, against an observed 61.9%; researchers’ beliefs were strongly related to which studies succeeded, and the authors concluded the failures were not the result of chance alone.

The prediction-market and survey figures were confirmed in the authors’ preprint (SocArXiv), since the published full text is paywalled. Later forecasting projects find markets predict replication with roughly 70% accuracy, useful but not perfect.

The 2015 team’s correlational tests suggested that the strength of the original evidence predicted success better than who ran either study.

The Open Science Collaboration reported this as what correlational tests ‘suggest’; the sentence now carries that hedge.

A multi-lab project led by Richard Klein tested 13 effects and found that replicability seemed to depend more on the effect itself than on whether the sample was online or in a lab, in the US or elsewhere.

Many Labs 1 drew this conclusion from 13 effects and called it a small sample; the larger Many Labs 2 (28 effects) found limited variation across samples and settings, consistent with it.

Two Large Attempts to Check the Record / Where the Extra Size Comes From

So the two projects disagree on the rate and agree on what matters for a reader. Published effects tend to shrink when someone checks, by about half in these two projects and by more in some larger efforts since, and failures are not random.

Later replication projects found larger declines than half: Many Labs 2 (median d from 0.60 to 0.15) and SCORE across the social and behavioral sciences (median r from 0.25 to 0.10). Some decline is expected because only significant originals were re-tested.

Where the Extra Size Comes From

Simmons, Nelson and Simonsohn (2011) showed that ordinary flexibility in collecting, analysing and reporting data dramatically raises the real rate of false positives; in many cases a researcher is more likely to find false evidence of an effect than to correctly find none.

Confirmed against the Simmons, Nelson and Simonsohn abstract; later simulation work agrees.

Preregistration (stating the question and analysis plan before seeing the outcome) keeps a prediction distinct from a story fitted afterwards, a mistake hindsight makes easy.

Preregistration’s value depends on adherence: audits find most preregistered studies deviate from their plans, often without saying so. The article’s advice to compare plan and paper addresses this.

Scheel, Schijen and Lakens compared 71 registered reports with 152 standard psychology papers; the first hypothesis was supported in 96% of standard papers and 44% of registered reports.

Checking takes a little care. A preregistered study usually says so in its methods section and links to the plan on a public registry, and services for this now exist across all disciplines. Compare the plan with the paper, because departures are common and often go unmentioned. If the headline result comes from an analysis the plan never mentioned, read it as a promising lead rather than a confirmed finding.

Audits of preregistered studies found that 25 of 27 early ones in Psychological Science deviated from their plans, and 72% of a sample in organizational psychology did, 68% with at least one undisclosed change. Comparing plan and paper is still the right habit; it just takes more than a glance.

The classic that averaged out

Scheibehenne and colleagues (2010) pooled 63 conditions from 50 experiments, about 5,000 people, and found a mean effect of virtually zero with wide variation between studies.

Chernev and colleagues’ 2015 meta-analysis of 99 observations found choice overload is real under specific conditions: complex choice sets, difficult tasks, uncertain preferences and a goal of minimising effort.

Confirmed against the source abstract. The paper appeared online in 2014 and in print in 2015.

The number with no denominator

At Microsoft, a ‘too good to be true’ alert on a surprising lift usually signals a bug.

The statistic that doesn’t add up

Nuijten and colleagues checked more than 250,000 p-values from eight psychology journals (1985-2013); half of papers reporting significance tests had at least one inconsistent p-value, and one in eight had an inconsistency that may have changed the statistical conclusion.

Tool: statcheck

The tool behind that study, described by its authors as a spellchecker for statistics. It finds significance tests reported in APA style, recalculates each p-value and flags the ones that don’t match. It usually can’t read statistics in tables, so a clean result covers only the tests it found. Run a paper through the web version before your team builds on it.

The ‘spellchecker’ description comes from statcheck.io, the authors’ web version. Its FAQ notes it usually misses statistics reported in tables.

What changed in the article

These edits were made on November 12, 2026 after the review.

Two Large Attempts to Check the Record: supported with caveats

Claim reviewed: The 2015 team found that the strength of the original evidence predicted replication success better than who ran either study.

Now reads: The 2015 team’s correlational tests suggested that the strength of the original evidence predicted success better than who ran either study.

Why: The cited paper says this, but hedged: its correlational tests ‘suggest’ original evidence strength predicted success better than team characteristics. The article states it as a firm finding. A one-word change restores the paper’s caution.

Two Large Attempts to Check the Record: supported with caveats

Claim reviewed: A multi-lab project led by Richard Klein found that replicability depended more on the effect itself than on whether the sample was online or in a lab, in the US or elsewhere.

Now reads: A multi-lab project led by Richard Klein tested 13 effects and found that replicability seemed to depend more on the effect itself than on whether the sample was online or in a lab, in the US or elsewhere.

Why: The cited paper reaches this conclusion but calls its basis ‘this small sample of effects’ (13 effects). Many Labs 2, a larger follow-up, found broadly the same pattern, so the claim holds; the article should just signal the small base.

Two Large Attempts to Check the Record / Where the Extra Size Comes From: supported with caveats

Claim reviewed: Published effects tend to shrink by about half when someone checks, and failures are not random.

Now reads: So the two projects disagree on the rate and agree on what matters for a reader. Published effects tend to shrink when someone checks, by about half in these two projects and by more in some larger efforts since, and failures are not random.

Why: Both cited projects found effects roughly halved, and Camerer et al. support ‘not random’. But the article turns two projects into a general rule, and larger later efforts found bigger drops: Many Labs 2’s median effect fell from d = 0.60 to 0.15 and SCORE’s from r = 0.25 to 0.10. ‘About half’ is closer to a floor than a typical figure, which makes the article’s advice to plan for smaller effects stronger, not weaker. Now cites SCORE collaboration, Nature (2026), via Europe PMC.

Where the Extra Size Comes From: supported with caveats

Claim reviewed: If the headline result comes from an analysis the preregistered plan never mentioned, read it as a promising lead rather than a confirmed finding; checking takes a few minutes.

Now reads: Checking takes a little care. A preregistered study usually says so in its methods section and links to the plan on a public registry, and services for this now exist across all disciplines. Compare the plan with the paper, because departures are common and often go unmentioned. If the headline result comes from an analysis the plan never mentioned, read it as a promising lead rather than a confirmed finding.

Why: Treating unplanned analyses as leads, not confirmed findings, follows directly from the cited paper. But ‘checking takes a few minutes’ undersells the job: audits find most preregistered studies depart from their plans and many don’t say so, so a reader has to compare the two documents with some care rather than trust the label. Now cites Preregistration adherence in industrial and organizational psychology, Journal of Occupational and Organizational Psychology (2025).

The classic that averaged out: misattributed

Claim reviewed: The jam study is the field’s favourite evidence that more choice can mean fewer purchases.

Now reads: The jam study from the fourth article, in which 30% of shoppers who stopped at a display of 6 jams bought a jar against 3% of those who stopped at a display of 24, is the field’s favorite evidence that more choice can mean fewer purchases.

Why: The jam study is Iyengar and Lepper (2000), which the fourth article lists as a source but this article does not: of shoppers who stopped at the tasting table, 30% bought a jar from 6 jams and 3% from 24. Calling it the field’s favorite evidence is fair; it has about four times the citations of either meta-analysis cited in the paragraph, and a 2024 review calls it the classic and seminal example of choice overload. Replication attempts and field data from retail, pensions, health plans, search engines and a mental health app show the effect is conditional, which the next sentences already say, so the only fix is to cite the paper here. Now cites Iyengar & Lepper, When Choice Is Demotivating: Can One Desire Too Much of a Good Thing?, Journal of Personality and Social Psychology (2000).

Tool: statcheck: supported with caveats

Claim reviewed: statcheck, described by its authors as a spellchecker for statistics, finds APA-style significance tests, recalculates each p-value and flags the ones that don’t match.

Now reads: The tool behind that study, described by its authors as a spellchecker for statistics. It finds significance tests reported in APA style, recalculates each p-value and flags the ones that don’t match. It usually can’t read statistics in tables, so a clean result covers only the tests it found. Run a paper through the web version before your team builds on it.

Why: statcheck does what the article says, and its authors do call it a spellchecker for statistics (on statcheck.io rather than in the cited paper). Its accuracy is high, but the tool’s own FAQ says it usually can’t read statistics in tables, so a clean report is not a full check. Readers told to run a paper through it should know that. Now cites statcheck.io (web version and FAQ).