conv.

All stories
AIQuiet 10d · day 13

OpenAI to honor a shared robots.txt signal for AI-training opt-outs

OpenAI says it will support a cross-provider robots.txt token, built on an IETF draft, letting sites set one AI-training preference for multiple crawlers.

newest coverage 10d agosummary written 11d ago5 pieces5 likes & upvotes1 comments NewswiresHacker News

What to know

  • OpenAI will honor a new shared robots.txt token (__usage_ai-training) letting sites express one AI-training preference across multiple participating providers instead of separate per-crawler rules.
  • The signal comes from an individual IETF Internet-Draft authored by Anthropic's Kevin Kelley; the IETF notes such drafts are not endorsed and have no formal standing in the standards process.
  • The mechanism adds no new robots.txt syntax, keeps existing crawler-control directives unchanged, and allows private agreements to override the expressed preference.
  • OpenAI has not yet implemented the signal, saying details will follow in the coming months, and is encouraging other AI providers to adopt it as well.

“User-Agent: __usage_ai-training”

draft-kelley-aipref-training-preference-signal-00, IETF Internet-Draft example robots.txt group · Unite.AI ↗ · Oct 29

OpenAI AI companyKevin Kelley Anthropic engineer, draft authorAnthropic AI company, IETF working group participantIETF AI Preferences working group Standards body

OpenAI to honor a shared robots.txt signal for AI-training opt-outs
unite.ai

How it unfolded 2 developments, newest first · click a bar or a number to jump articlesposts

Peak 2 pieces in 3h at Oct 30, 4 PM; 5 pieces over 13 days (3 articles · 2 posts) Oct 30, 4 AM — 1 piece · 1 article — Newswires 1Oct 30, 7 AM — quietOct 30, 10 AM — quietOct 30, 1 PM — quietOct 30, 4 PM — 2 pieces · 2 articles — Newswires 2Oct 30, 7 PM — 1 piece · 1 post — Hacker News 1Oct 30, 10 PM — quietOct 31, 1 AM — quietOct 31, 4 AM — quietOct 31, 7 AM — quietOct 31, 10 AM — quietOct 31, 1 PM — quietOct 31, 4 PM — quietOct 31, 7 PM — quietOct 31, 10 PM — quietNov 1, 1 AM — quietNov 1, 3 AM — quietNov 1, 6 AM — quietNov 1, 9 AM — quietNov 1, 12 PM — 1 piece · 1 post — Hacker News 1Nov 1, 3 PM — quietNov 1, 6 PM — quietNov 1, 9 PM — quietNov 2, 12 AM — quietNov 2, 3 AM — quietNov 2, 6 AM — quietNov 2, 9 AM — quietNov 2, 12 PM — quietNov 2, 3 PM — quietNov 2, 6 PM — quietNov 2, 9 PM — quietNov 3, 12 AM — quietNov 3, 3 AM — quietNov 3, 6 AM — quietNov 3, 9 AM — quietNov 3, 12 PM — quietNov 3, 3 PM — quietNov 3, 6 PM — quietNov 3, 9 PM — quietNov 4, 12 AM — quietNov 4, 3 AM — quietNov 4, 6 AM — quietNov 4, 9 AM — quietNov 4, 12 PM — quietNov 4, 3 PM — quietNov 4, 6 PM — quietNov 4, 9 PM — quietNov 5, 12 AM — quietNov 5, 3 AM — quietNov 5, 6 AM — quietNov 5, 9 AM — quietNov 5, 12 PM — quietNov 5, 3 PM — quietNov 5, 6 PM — quietNov 5, 9 PM — quietNov 6, 12 AM — quietNov 6, 3 AM — quietNov 6, 6 AM — quietNov 6, 9 AM — quietNov 6, 12 PM — quietNov 6, 3 PM — quietNov 6, 6 PM — quietNov 6, 9 PM — quietNov 7, 12 AM — quietNov 7, 3 AM — quietNov 7, 6 AM — quietNov 7, 9 AM — quietNov 7, 12 PM — quietNov 7, 3 PM — quietNov 7, 6 PM — quietNov 7, 9 PM — quietNov 8, 12 AM — quietNov 8, 3 AM — quietNov 8, 6 AM — quietNov 8, 9 AM — quietNov 8, 12 PM — quietNov 8, 3 PM — quietNov 8, 6 PM — quietNov 8, 9 PM — quietNov 9, 12 AM — quietNov 9, 3 AM — quietNov 9, 6 AM — quietNov 9, 9 AM — quietNov 9, 12 PM — quietNov 9, 3 PM — quietNov 9, 6 PM — quietNov 9, 9 PM — quietYesterday, 12 AM — quietYesterday, 3 AM — quietYesterday, 6 AM — quietYesterday, 9 AM — quietYesterday, 12 PM — quietYesterday, 3 PM — quietYesterday, 6 PM — quietYesterday, 9 PM — quietToday, 12 AM — quietToday, 3 AM — quietToday, 6 AM — quietToday, 9 AM — quietToday, 12 PM — quietToday, 3 PM — quietToday, 6 PM — quiet 12
Oct 31Nov 1Nov 2Nov 3Nov 4Nov 5Nov 6Nov 7Nov 8Nov 9yesterdaynow · 8:41 PM ET
  1. 2

    IETF draft by Anthropic's Kevin Kelley defines the new signal

    Reporting detailed the underlying mechanism: an individual, not-yet-endorsed IETF Internet-Draft titled "Machine-Readable Training Preference Signal," authored by Anthropic's Kevin Kelley, defines a cross-vendor policy token (__usage_ai-training) that binds "no AI training" preferences to paths using existing robots.txt syntax, with no new grammar and existing parsers unaffected.

    “no AI training…”
    — Machine-Readable Training Preference Signal draft
    1. 1 outlet OpenAI to Honor Shared Robots.txt Signal for AI Training Preferences

      first by Unite.AI, 12d ago

  2. 1

    OpenAI commits to honoring a shared AI-training preference signal

    OpenAI announced it will honor a shared robots.txt signal so site operators can communicate one AI-training preference to multiple participating providers instead of per-crawler rules, calling the step voluntary and saying it will implement the signal in the coming months.

Also covered reported alongside — the timeline has no entry for these yet

  1. 2 outlets Making it easier for websites to express AI training preferences

    first by OpenAI, 12d ago · also OpenAI News