When Clinical Chatbots Lose Attention on Long Records
Large language models dropped from 90 percent accuracy to near-total failure on extended cognitive tests, exposing risks when patients submit lengthy symptom histories.
Psychologists have used the Stroop test since 1935 to measure selective attention and processing speed by asking subjects to name ink colours that clash with printed words. When researchers recently adapted this classic cognitive battery to evaluate commercial large language models across sequences of varying lengths, the systems hit an unexpected wall. While the models matched human baselines at short prompt lengths, accuracy collapsed from above 90 percent to below 10 percent as the sequence length increased.
The mechanism driving this drop involves contextual dilution across transformer attention heads. As a prompt accumulates tokens, the mathematical weighting that keeps the initial diagnostic instruction salient begins to disperse. For an individual using a conversational assistant to sort through months of blood tests, subjective journal entries, and discharge notes, this architectural limitation introduces silent diagnostic drift.
Context degradation in long-tail health histories
Most health tracking produces messy, longitudinal text. A runner documenting gastrointestinal issues across six months of training generates thousands of tokens of non-linear descriptions. When you feed that history into a general-purpose model, the system maintains fluent grammar even as its ability to track chronological conflicts degrades.
You can protect your own self-management routines by breaking long health timelines into short, dated summaries before querying any tool. When you ask software to evaluate a trend, restrict each prompt to one specific parameter, such as fasting glucose over four weeks, rather than uploading an entire multi-year medical dossier.
The longer view
One headline rarely tells the story. See how today’s news fits the bigger shifts on AI Trends, or learn to read your own data on How it works.