conv.

All stories
AIQuiet 5d · day 5

Study finds 44% of published LLM findings expire within 15 months

Replication of 50 widely cited claims across 20 models shows most research about AI behavior becomes outdated as capabilities improve.

newest coverage 5d agosummary written 5d ago3 pieces141 likes & upvotes13 comments Hacker NewsBlueskyMastodon

What to know

  • Researchers retested 50 widely cited LLM claims across 20 models released 2024–2026, finding 44% expired entirely and 34% weakened significantly by September 2026.
  • Models have improved dramatically on fabricated citations (from 55% in GPT-3.5 to rare occurrences), hallucination, and stereotyping—but sycophancy and political lean persist.
  • Published findings about LLM behavior become outdated within roughly 15 months as model capabilities advance, suggesting the field's knowledge base has a short half-life.

Zou Group Research teamEthan Mollick (emollick) AI researcherWalters and Wilder Researchers (2023)

How it unfolded 1 development · click the chart to see its coverage posts

Peak 2 pieces in two hours at Nov 2, 8 PM; 3 pieces over 6 days (3 posts) Nov 2, 8 PM — 2 pieces · 2 posts — Bluesky 1, Hacker News 1Nov 2, 10 PM — quietNov 3, 12 AM — 1 piece · 1 post — Mastodon 1Nov 3, 2 AM — quietNov 3, 4 AM — quietNov 3, 6 AM — quietNov 3, 8 AM — quietNov 3, 10 AM — quietNov 3, 12 PM — quietNov 3, 2 PM — quietNov 3, 4 PM — quietNov 3, 6 PM — quietNov 3, 8 PM — quietNov 3, 10 PM — quietNov 4, 12 AM — quietNov 4, 2 AM — quietNov 4, 4 AM — quietNov 4, 6 AM — quietNov 4, 8 AM — quietNov 4, 10 AM — quietNov 4, 12 PM — quietNov 4, 2 PM — quietNov 4, 4 PM — quietNov 4, 6 PM — quietNov 4, 8 PM — quietNov 4, 10 PM — quietNov 5, 12 AM — quietNov 5, 2 AM — quietNov 5, 4 AM — quietNov 5, 6 AM — quietNov 5, 8 AM — quietNov 5, 10 AM — quietNov 5, 12 PM — quietNov 5, 2 PM — quietNov 5, 4 PM — quietNov 5, 6 PM — quietNov 5, 8 PM — quietNov 5, 10 PM — quietNov 6, 12 AM — quietNov 6, 2 AM — quietNov 6, 4 AM — quietNov 6, 6 AM — quietNov 6, 8 AM — quietNov 6, 10 AM — quietNov 6, 12 PM — quietNov 6, 2 PM — quietNov 6, 4 PM — quietNov 6, 6 PM — quietNov 6, 8 PM — quietNov 6, 10 PM — quietYesterday, 12 AM — quietYesterday, 2 AM — quietYesterday, 4 AM — quietYesterday, 6 AM — quietYesterday, 8 AM — quietYesterday, 10 AM — quietYesterday, 12 PM — quietYesterday, 2 PM — quietYesterday, 4 PM — quietYesterday, 6 PM — quietYesterday, 8 PM — quietYesterday, 10 PM — quietToday, 12 AM — quietToday, 2 AM — quietToday, 4 AM — quietToday, 6 AM — quiet 1
Nov 3Nov 4Nov 5Nov 6yesterdaynow · 8:43 AM ET
  1. 1

    Emollick highlights that conventional wisdom about LLMs is outdated

    AI researcher Emollick (Ethan Mollick) shared the study on Bluesky, noting that recent models no longer invent citations and both hallucination and stereotyping have dropped, but models remain sycophantic and lean left.

    “Recent models no longer invent citations and both hallucination rates & stereotyping has dropped. But they are still sycophantic & lean left.”
    — emollick.bsky.social
    • emollick.bsky.social

      Here is a mass attempted replication of old findings of LLMs with new models. Much conventional wisdom is outdated. Recent models no longer invent citations and both hallucination rates & stereotyping has dropped. But they are still sycophantic & lean left. zou-group.

      emollick.bsky.socialBluesky5d ago111▲view on Bluesky ↗
  2. background

    Zou Group replicates 50 LLM claims across 20 modern models — Researchers tested 50 widely cited findings about LLMs across 20 models released from July 2024 to September 2026. They classified findings as expired (44%), weakened (34%), persisting (16%), or strengthened (6%), and found that within 15 months of publication, most claims either no longer held or showed substantially diminished effects.

  3. background

    Walters and Wilder find 55% of ChatGPT citations are fabricated — A 2023 study by Walters and Wilder tested ChatGPT (using GPT-3.5) on writing literature reviews and found that 55% of the citations it generated were fake.