Article: Behavioral Signals in Adaptive Interfaces
Reviewed: November 2026 · Method version 2 · How these reviews work
Each factual claim in the article was checked against the source it cites, then challenged by reviewers whose job was to find the strongest evidence against it: replication attempts, meta-analyses, later papers that cite the original work, and research from other fields that study the same mechanism. A separate reviewer weighed both sides, and a script re-checked every quoted source. Where a claim was overstated or misattributed, the article was corrected. The research below supports the article as it now reads.
In Office 2000, menus promoted commands you used to a short menu; the rest waited behind a chevron at the bottom or appeared after a few seconds. The idea, as Jensen Harris described it, was that you would eventually have a fully tuned interface that showed only what you needed.
Harris’s own account of Office 2000’s adaptive menus was confirmed in the archived post.
Writing in 2005, as his team built the Ribbon, Harris said Adaptive Menus ‘were not successful’: people use a wide range of features so no default short menu suited everyone, a wrong short menu forced two scans, and the full menu reset your place when it opened. The menus also felt random: a command was in one spot, and two days later it was gone.
Independent lab and field studies (Mitchell and Shneiderman 1989; McGrenere, Baecker and Booth 2002) found users slower with or less fond of menus that reorder themselves, matching Harris’s account.
Harris: ‘Auto-customization, unless it does a perfect job, is usually worse than no customization at all.’
As a general rule Harris’s line is too strong: spatially stable adaptation such as ephemeral menus (Findlater et al., 2009) was faster than static menus at high accuracy and not significantly slower at low accuracy. The article’s ‘half right’ framing already allows for this.
An adaptation changes the place people come back to, so when it moves things, people have to relearn where they are.
Mitchell and Shneiderman (1989) found that people who started with self-reordering menus were significantly slower, and 81% preferred static menus.
Implicit signals (clicks, dwell time, purchases, skips) are plentiful and cheap, which is why most adaptive systems lean on them.
Kelly and Teevan (2003) describe implicit feedback as attractive because large quantities can be gathered at no extra cost to the user.
Explicit signals are scarce and biased in their own way, since the people who bother to rate are not everyone.
Ratings in a real recommender system were found to be missing not at random (Marlin and Zemel, 2009).
At 70% accuracy people used the adaptive toolbar 86% of the times it held the right button, against 71% at 50%, and finished faster; predictability had no significant effect on speed.
The no-effect-on-speed result for predictability comes from one lab study; no replication was found among the citing papers checked.
Findlater and McGrenere compared static, adaptive and user-adaptable menus with 27 people: the static menu was significantly faster than the adaptive one, and most participants preferred the version they could arrange themselves.
Findlater and McGrenere’s own 2008 follow-up found high-accuracy adaptive menus improved speed and satisfaction, especially on small screens.
Harris’s line fits adaptation that moves things. Later studies found that adaptation which leaves items in place, adding a shortcut or briefly highlighting likely choices, cost little even when its guesses were often wrong, though it can make people less aware of the features it doesn’t surface.
Ephemeral adaptation (Findlater et al., 2009) kept menu items in place and was not significantly slower than a static menu even at low accuracy; Findlater and McGrenere (2008) note that accurate adaptation can reduce awareness of other features.
YouTube added watch time in 2012 and saw an immediate 20% drop in views, a cost it chose to accept; Goodrow describes staying up late watching random videos he would rather not have spent the evening on.
Kelly and Belkin (2004) found no general relationship between reading time and usefulness, supporting the article’s point that time spent is not value.
‘When a measure becomes a target, it ceases to be a good measure.’ (Marilyn Strathern, ‘Improving Ratings’: Audit in the British University System)
Strathern’s 1997 wording is the popular form of what she and Keith Hoskin call Goodhart’s law.
These edits were made on November 12, 2026 after the review.
Claim reviewed: Research published two years later suggests Harris was half right.
Now reads: Research published in 2008 suggests he was half right.
Why: Harris’s post is dated October 10, 2005 and the Gajos study appeared at CHI in April 2008, so the gap is closer to two and a half years than two. The ‘half right’ reading is fair: in that study accuracy drove speed and use, while predictability still mattered for satisfaction.
Claim reviewed: Because adaptation persists, its errors accumulate, and every time the layout shifts, people have to relearn where things are.
Now reads: An adaptation changes the place people come back to, so when it moves things, people have to relearn where they are.
Why: Lab studies show that menus which move items slow people down while they relearn positions, so the core point holds. The claim that errors accumulate is plausible but was not tested directly, and adaptation that leaves items where they are avoids the relearning cost. The sentence is uncited. Now cites Mitchell & Shneiderman, Dynamic versus static menus: an exploratory comparison, SIGCHI Bulletin (1989).
Claim reviewed: Gajos et al. (2008) had 23 experienced Office users work in a modified Word interface with an extra toolbar of eight buttons that held the needed button 50% or 70% of the time and updated either predictably (most recently used) or at random.
Now reads: Twenty-three intermediate to expert users of Office-style software worked in a modified Word interface with an extra toolbar of eight buttons that changed as they went.
Why: The design details are right: 23 participants, a modified Word interface, eight adaptive buttons, 50% or 70% accuracy, and most-recently-used versus random updates. The paper describes participants as intermediate to expert users of Office-style applications, which is a little looser than ‘experienced Office users’.
Claim reviewed: Harris’s line holds for adaptation that moves things; adaptation that adds a shortcut and stays legible has more room for error.
Now reads: Harris’s line fits adaptation that moves things. Later studies found that adaptation which leaves items in place, adding a shortcut or briefly highlighting likely choices, cost little even when its guesses were often wrong, though it can make people less aware of the features it doesn’t surface.
Why: The idea is sound, but neither cited study compared copying with moving, so the article leans on them for something they did not test. Later work on menus that keep items in place and only highlight likely choices supports it directly. That work also found a cost: accurate shortcuts make people less aware of the rest of the features. Now cites Findlater, Moffatt, McGrenere & Dawson, Ephemeral adaptation: the use of gradual onset to improve menu selection performance, CHI (2009).
Claim reviewed: Most product teams skip re-checking signals; a signal is usually chosen once, when a feature is built, and then quietly becomes the definition of success.
Now reads: Teams often skip that step. A signal is usually chosen once, when a feature is built, and can quietly become the definition of success.
Why: No study was found measuring how many product teams re-check their signals after launch, so ‘most product teams’ is an unsupported generalization. The risk it describes is well documented, but its prevalence is not.