attck-research

Evidence review: The Future of Behavioral Design in AI-driven Environments

Article: The Future of Behavioral Design in AI-driven Environments
Reviewed: November 2026 · Method version 2 · How these reviews work

How this review works

Each factual claim in the article was checked against the source it cites, then challenged by reviewers whose job was to find the strongest evidence against it: replication attempts, meta-analyses, later papers that cite the original work, and research from other fields that study the same mechanism. A separate reviewer weighed both sides, and a script re-checked every quoted source. Where a claim was overstated or misattributed, the article was corrected. The research below supports the article as it now reads.

Background research

Introduction

In 1998, B.J. Fogg set out an agenda for captology, the study of computers as persuasive technologies; one of his perspectives was a ‘functional triad’ in which computers persuade as tools, as media, and as social actors.

For the next quarter century, the third role stayed limited. People already responded to computers socially, and chatbots and voice assistants could answer back, but mostly from scripts. Teams could frame a price, choose a default or rewrite a button, and this series has spent most of its fifty articles on exactly those moves. A screen still couldn’t hold a real conversation. That has changed.

Pre-LLM research (Computers Are Social Actors) shows people treated computers socially long before 2025; the article now says the change is in conversational ability.

In September 2025, ChatGPT began letting US users buy some products inside the chat, tapping Buy and confirming the order without leaving the conversation. OpenAI later let merchants send buyers back to their own checkouts, but the step had been taken: the social actor could close a sale.

The September 2025 launch is accurate; OpenAI changed the checkout model in March 2026, which the article now notes.

Is Personalized Persuasion the Real Risk?

Salvi et al. (2025) put 900 people into short multi-round online debates against a human or GPT-4, some opponents given basic sociodemographic data; where AI and humans were not equally persuasive, GPT-4 with that data was more persuasive 64.4% of the time, an 81.2% relative increase in the odds of moving agreement.

Confirmed against the abstract of Salvi et al. (2025). Later work suggests attitude shifts in such studies may not carry over to behavior.

Hackenburg and Margetts (2024) fed 8,587 participants’ self-reported demographic and political data into GPT-4 prompts to write a message for each person; the messages persuaded, in some cases shifting support by up to 12 percentage points.

Microtargeted messages did no better than generic ones (4.83 vs 6.20 points, not statistically significant); the authors concluded the power may lie in persuasive generic messages rather than tailoring.

Confirmed against the abstract and corroborated by Argyle et al. (2025) and Hackenburg et al. (2025).

The studies used different formats, a live debate against a single written message, so they can both be right. Later work split on the format question. The largest experiment so far, with about 77,000 participants, found a conversation with the AI 41 to 52 percent more persuasive than a static message from the same model, while two smaller preregistered studies found that the back-and-forth added little over one well-made message. These later studies also found that personal data added little and that relevant information did most of the work. Our reading: the conversation may matter more than the dossier, and the content matters more than either. Limiting what a system knows about someone is still worth doing, but it does not make a fluent system harmless.

Pass 5 read the full text of the largest study (Hackenburg et al., Science 2025): it compared conversation with a static message directly and found conversation 41 to 52 percent more persuasive, with personalization adding little. Argyle et al. 2025, Costello et al. ‘Just the facts’ (a single tailored message kept 79 percent of the dialogue’s effect) and Altay et al. 2022 found little added effect from dialogue. The article keeps its hedged reading and now names the split and the role of content.

Costello, Pennycook and Rand had 2,044 conspiracy believers discuss their belief with GPT-4 Turbo, which answered with tailored evidence; belief fell by about 20% and stayed lower two months later.

Confirmed against the Science abstract. Durability may not hold in every domain; a 2026 study of prejudice found gains faded within a week.

The one-answer shelf

OpenAI said Instant Checkout’s product results were organic and unsponsored, ranked on relevance, though when several merchants sold the same item, whether they offered Instant Checkout was one of the factors. The user confirmed order, shipping and payment before anything was bought.

OpenAI’s own page lists Instant Checkout availability as a ranking factor among merchants selling the same item; the article now says so.

Our reading, not a finding: a shelf narrowed to one item may act like a strong default, since defaults work best when they read as a recommendation, and that puts much of the weight of “Choice Architecture for Ethical Persuasion” on the confirmation step.

Defaults are most effective when seen as an endorsement (Jachimowicz et al. 2019), which supports the analogy; the comparison to ‘the strongest default’ is softened.

The loop that learned to flatter

On April 25, 2025, OpenAI shipped a GPT-4o update that made ChatGPT noticeably more sycophantic and began rolling it back three days later.

In 2023, Sharma and colleagues found that both people and the preference models trained on their judgments chose convincingly written sycophantic answers over correct ones a non-negligible fraction of the time.

The lesson for any product that learns from reactions is the one from “Measuring Behavioral Change Over Time”: a signal that pays off within seconds may not predict, and can even reverse, one that runs for months. Google’s long-running experiments found the same with ordinary A/B tests.

Supported by online-experimentation research on short-term versus long-term effects (Hohnhold et al. 2015), now cited.

The customer who is software

Ersoy and colleagues tested six web agents built on three language models in mock shopping, streaming and news sites; with a single dark pattern on the page, agents fell for it 41% of the time on average.

Confirmed against the abstract of Ersoy et al.; published at IEEE S&P 2026.

A second team, in a study not yet peer reviewed, compared agents, people and human–agent pairs across 16 dark-pattern types.

Confirmed against the arXiv abstract; the article now notes the study is a preprint.

Where the Rules Already Apply

In the European Union, the AI Act’s Article 5 has, since February 2, 2025, banned AI systems that use subliminal, purposefully manipulative or deceptive techniques to materially distort behavior by appreciably impairing people’s ability to make an informed decision, where that causes or is reasonably likely to cause significant harm.

The summary now keeps the Act’s qualifiers (‘appreciably’, ‘reasonably likely’).

A July 2026 amendment left both duties in place and gave existing generative systems until December 2, 2026 to start marking their output as AI-generated.

What Fifty Articles Add Up To

The person on the other side of an assistant still has limited attention, still leans on defaults, still trusts what reads easily and, on average, still tends to weigh a loss more heavily than an equal gain, though how much more is still debated.

Defaults and fluency are well established; the size of loss aversion is debated (Brown et al. 2024 versus Walasek et al. 2018), and the article now hedges it.

What changed in the article

These edits were made on November 12, 2026 after the review.

Introduction: supported with caveats

Claim reviewed: For the next quarter century the social-actor role was the weak one: a screen still couldn’t talk back.

Now reads: For the next quarter century, the third role stayed limited. People already responded to computers socially, and chatbots and voice assistants could answer back, but mostly from scripts. Teams could frame a price, choose a default or rewrite a button, and this series has spent most of its fifty articles on exactly those moves. A screen still couldn’t hold a real conversation. That has changed.

Why: It is fair to say conversational persuasion was limited before large language models. But ‘a screen still couldn’t talk back’ goes too far: research since the 1990s shows people respond to computers as social actors, and chatbots, voice assistants and even notifications addressed people in person-like ways. What changed is the quality of the conversation, not whether screens could talk. Now cites Lombard and Xu (2021), Social Responses to Media Technologies in the 21st Century: The Media are Social Actors Paradigm, Human-Machine Communication 2.

Introduction: supported with caveats

Claim reviewed: In September 2025, ChatGPT began letting US users buy some products inside the chat, tapping Buy and confirming the order without leaving the conversation; ‘the social actor has a checkout.’

Now reads: In September 2025, ChatGPT began letting US users buy some products inside the chat, tapping Buy and confirming the order without leaving the conversation. OpenAI later let merchants send buyers back to their own checkouts, but the step had been taken: the social actor could close a sale.

Why: OpenAI’s launch post confirms that in September 2025 US users could buy from Etsy sellers inside ChatGPT, tapping Buy and confirming without leaving the chat. The article is dated February 2027, though, and in March 2026 OpenAI said it was letting merchants use their own checkout and refocusing on product discovery. The present-tense ‘The social actor has a checkout’ may read as current when it no longer fully is. Now cites Retail Gazette, OpenAI pivots ChatGPT shopping strategy after Instant Checkout underwhelms (25 March 2026).

Is Personalized Persuasion the Real Risk?: supported with caveats

Claim reviewed: Because the two studies used different formats (live debate vs a single written message), both can be right; ‘the conversation may matter more than the dossier.’

Now reads: The studies used different formats, a live debate against a single written message, so they can both be right. Later work split on the format question. The largest experiment so far, with about 77,000 participants, found a conversation with the AI 41 to 52 percent more persuasive than a static message from the same model, while two smaller preregistered studies found that the back-and-forth added little over one well-made message. These later studies also found that personal data added little and that relevant information did most of the work. Our reading: the conversation may matter more than the dossier, and the content matters more than either. Limiting what a system knows about someone is still worth doing, but it does not make a fluent system harmless.

Why: The largest experiment so far, with about 77,000 participants, tested the format directly and found that talking with the AI was 41 to 52 percent more persuasive than reading a static message from the same model, while personal data added little. Three smaller studies found the opposite: a single well-made message did about as well as a dialogue, and in the conspiracy work a single tailored message kept 79 percent of the dialogue’s effect. The article’s hedged ‘may matter more than the dossier’ is a fair reading of that split, but it should say the question is open and that content did most of the work in every study. Now cites Hackenburg et al. (2025), The levers of political persuasion with conversational artificial intelligence, Science; and Argyle et al. (2025), Testing theories of political persuasion using AI, PNAS.

The one-answer shelf: supported with caveats

Claim reviewed: OpenAI says Instant Checkout’s product results are organic and unsponsored, ranked purely on relevance, and that the user confirms order, shipping and payment before anything is bought.

Now reads: OpenAI said Instant Checkout’s product results were organic and unsponsored, ranked on relevance, though when several merchants sold the same item, whether they offered Instant Checkout was one of the factors. The user confirmed order, shipping and payment before anything was bought.

Why: The attribution is accurate: OpenAI’s page says results are organic, unsponsored and ranked purely on relevance, and that users confirm order, shipping and payment. But the same page says that when several merchants sell the same product, whether Instant Checkout is enabled is a ranking factor, which sits awkwardly beside ‘purely on relevance’. The flow described also changed in March 2026.

The one-answer shelf: contested

Claim reviewed: A results page shows ten options; a conversational answer usually leads with one.

Now reads: A results page shows ten links. A conversational answer shows a handful of picks, and people look past them far less often: in one Pew study, Google users clicked a regular search result in 8 percent of visits when an AI summary appeared, against 15 percent when none did.

Why: This is uncited, and the article’s own example contradicts the wording. OpenAI’s post says ChatGPT shows ‘the most relevant products’, and its help page describes a carousel of product options, not one item. Pew found only 1 percent of Google AI summaries cite a single source. The narrowing is real, though: users clicked a regular result in 8 percent of visits with an AI summary against 15 percent without, a review puts the drop in organic clicks at 35 to 61 percent, and a causal study found AI Overviews cut Wikipedia traffic by about 15 percent. ‘Usually leads with one’ overstates it; ‘a handful of picks’ is what the evidence shows. Now cites Pew Research Center, Google users are less likely to click on links when an AI summary appears in the results (22 July 2025).

The one-answer shelf: supported with caveats

Claim reviewed: A shelf narrowed to one item behaves like the strongest default this series has covered, so the confirmation step carries the whole weight of ethical choice architecture.

Now reads: Our reading, not a finding: a shelf narrowed to one item may act like a strong default, since defaults work best when they read as a recommendation, and that puts much of the weight of “Choice Architecture for Ethical Persuasion” on the confirmation step.

Why: This is openly flagged as the authors’ reading, and the mechanism is sound: defaults work best when they read as the choice architect’s recommendation, which is what a single assistant answer looks like. But no study ranks this against other defaults, so ‘the strongest default this series has covered’ is more than the evidence shows, and default effects vary widely. Now cites Jachimowicz et al. (2019), When and why defaults influence decisions: a meta-analysis of default effects, Behavioural Public Policy.

The loop that learned to flatter: supported with caveats

Claim reviewed: For any product that learns from reactions, a signal that pays off within seconds can point the opposite way from one that runs for months.

Now reads: The lesson for any product that learns from reactions is the one from “Measuring Behavioral Change Over Time”: a signal that pays off within seconds may not predict, and can even reverse, one that runs for months. Google’s long-running experiments found the same with ordinary A/B tests.

Why: The lesson is well supported, but uncited here. Google’s long-term experiments showed short-term A/B effects do not reliably predict long-term ones, and marketing panel data show promotions that lift sales now can make customers more price-sensitive later. ‘Can point the opposite way’ is the strong case; the usual finding is that short-term results fail to predict, sometimes in the opposite direction. Now cites Hohnhold, O’Brien and Tang (2015), Focusing on the Long-term: It’s Good for Users and Business, KDD.

The customer who is software: supported

Claim reviewed: A second team compared agents, people and human-agent pairs across 16 dark-pattern types: agents often failed to recognize the patterns and, even when they did, put finishing the task ahead of protecting the user; people fell for shortcuts and habit; human oversight helped at the cost of tunnel vision and added cognitive load.

Now reads: A second team, in a study not yet peer reviewed, compared agents, people and human–agent pairs across 16 dark-pattern types.

Why: The article matches the abstract: 16 dark-pattern types, agents failing to recognize them or putting task completion first, people falling for shortcuts and habit, and oversight costing attention and cognitive load. Confidence is medium because it is a preprint with no independent follow-up yet, which the body text does not say.

Where the Rules Already Apply: supported with caveats

Claim reviewed: Since February 2, 2025, the EU AI Act’s Article 5 has banned AI systems that use subliminal, purposefully manipulative or deceptive techniques to materially distort behavior by impairing people’s ability to make an informed decision, where that causes or is likely to cause significant harm.

Now reads: In the European Union, the AI Act’s Article 5 has, since February 2, 2025, banned AI systems that use subliminal, purposefully manipulative or deceptive techniques to materially distort behavior by appreciably impairing people’s ability to make an informed decision, where that causes or is reasonably likely to cause significant harm.

Why: The date and the substance are right: Chapter II, which holds Article 5, has applied since February 2, 2025. The summary drops two legal qualifiers, ‘appreciably’ impairing and ‘reasonably’ likely to cause significant harm, which narrow the ban. A small wording fix makes it exact.

What Fifty Articles Add Up To: supported with caveats

Claim reviewed: People did not change: the person on the other side of an assistant still has limited attention, still leans on defaults, still trusts what reads easily and still weighs a loss more heavily than an equal gain.

Now reads: The person on the other side of an assistant still has limited attention, still leans on defaults, still trusts what reads easily and, on average, still tends to weigh a loss more heavily than an equal gain, though how much more is still debated.

Why: Defaults and the pull of easy-to-process information both have strong meta-analytic support. Loss aversion is the shaky item: one large meta-analysis puts losses at about twice gains, another finds a much smaller ratio of about 1.3, and a 2025 re-analysis argues the effect is not robust. Saying people ‘still weigh a loss more heavily’ as settled fact overstates that last point. Now cites Brown, Imai, Vieider and Camerer (2024), Meta-analysis of Empirical Estimates of Loss Aversion, Journal of Economic Literature.