Stroop Tests Expose Severe Attention Decay in LLMs
Language models lose processing accuracy on sustained cognitive tests, casting doubt on their reliability for parsing long-term personal health records.
Researchers evaluating large language models using the Stroop test, a psychological assessment designed to measure selective attention and cognitive control, identified a sharp decline in processing reliability over extended sequences. While top systems accurately resolved conflicting color-word prompts in brief evaluations, accuracy collapsed from above 90% to near-zero as prompt length and task complexity increased.
In human psychology, the Stroop effect measures how easily your executive function suppresses automatic semantic reading to name conflicting font colors. In language models, however, the collapse reflects architectural limitations in transformer attention mechanisms: token distraction compounds over long input windows, causing the system to abandon system instructions in favor of raw statistical token frequency.
This degradation pattern exposes a critical vulnerability for anyone delegating complex health management to an automated interface. A model that correctly identifies an allergen or drug interaction in a five-line prompt often drops identical details when processing a dense, thirty-page laboratory history.
Do not paste raw, multi-year medical histories into a single consumer AI prompt expecting an accurate clinical synthesis. Break your records into discrete, chronologically constrained questions, and independently verify every medication dosage against the original laboratory sheet.
The longer view
One headline rarely tells the story. See how today’s news fits the bigger shifts on AI Trends, or learn to read your own data on How it works.