conv.

All stories
AIQuiet 10d · day 13

Google, Microsoft, Anthropic, OpenAI back unified AI-training opt-out in robots.txt

Four major AI developers plan to support a single robots.txt signal letting websites block AI training scraping across providers at once.

newest coverage 9d agosummary written 12d ago23 pieces21 likes & upvotes Hacker NewsMastodonNewswiresRedditBluesky

What to know

  • A single 'policy token' in robots.txt would let website owners opt out of AI training scraping across multiple providers instead of listing each crawler separately.
  • Google, Microsoft, Anthropic and OpenAI have submitted a draft proposal to the IETF to fold the mechanism into the Robots Exclusion Protocol.
  • The opt-out would not apply to models performing 'exclusively non-generative tasks'; Microsoft and OpenAI plan rollout in coming months, Google in 2027.
  • Meta has expressed support for the concept but has not committed to implementing it.

“The setting wouldn't apply to models that "perform exclusively non-generative tasks."”

SiliconANGLE, Reporting outlet · SiliconANGLE ↗ · Oct 29

Google Search/AI companyMicrosoft Technology companyAnthropic AI developerOpenAI AI developerMeta Platforms Inc. Technology companyAIPREF working group (IETF) Standards body

Google, Microsoft, Anthropic, OpenAI back unified AI-training opt-out in robots.txt
engadget.com

How it unfolded 2 developments, newest first · click a bar or a number to jump articlesposts

Peak 7 pieces in 3h at Oct 30, 3 PM; 23 pieces over 13 days (6 articles · 17 posts) Oct 30, 3 PM — 7 pieces · 5 articles · 2 posts — Newswires 5, Hacker News 1, Mastodon 1Oct 30, 6 PM — quietOct 30, 9 PM — quietOct 31, 12 AM — quietOct 31, 3 AM — quietOct 31, 6 AM — 6 pieces · 1 article · 5 posts — Mastodon 4, Newswires 1, Reddit 1Oct 31, 9 AM — quietOct 31, 12 PM — quietOct 31, 3 PM — quietOct 31, 6 PM — quietOct 31, 9 PM — quietNov 1, 12 AM — quietNov 1, 2 AM — quietNov 1, 5 AM — quietNov 1, 8 AM — quietNov 1, 11 AM — 1 piece · 1 post — Hacker News 1Nov 1, 2 PM — quietNov 1, 5 PM — 1 piece · 1 post — Hacker News 1Nov 1, 8 PM — 1 piece · 1 post — Mastodon 1Nov 1, 11 PM — quietNov 2, 2 AM — quietNov 2, 5 AM — quietNov 2, 8 AM — 1 piece · 1 post — Bluesky 1Nov 2, 11 AM — 2 pieces · 2 posts — Hacker News 2Nov 2, 2 PM — quietNov 2, 5 PM — quietNov 2, 8 PM — quietNov 2, 11 PM — 1 piece · 1 post — Mastodon 1Nov 3, 2 AM — quietNov 3, 5 AM — 1 piece · 1 post — Reddit 1Nov 3, 8 AM — quietNov 3, 11 AM — quietNov 3, 2 PM — quietNov 3, 5 PM — quietNov 3, 8 PM — quietNov 3, 11 PM — quietNov 4, 2 AM — quietNov 4, 5 AM — quietNov 4, 8 AM — 1 piece · 1 post — Bluesky 1Nov 4, 11 AM — quietNov 4, 2 PM — quietNov 4, 5 PM — quietNov 4, 8 PM — quietNov 4, 11 PM — quietNov 5, 2 AM — quietNov 5, 5 AM — quietNov 5, 8 AM — quietNov 5, 11 AM — quietNov 5, 2 PM — quietNov 5, 5 PM — quietNov 5, 8 PM — quietNov 5, 11 PM — quietNov 6, 2 AM — quietNov 6, 5 AM — quietNov 6, 8 AM — quietNov 6, 11 AM — 1 piece · 1 post — Hacker News 1Nov 6, 2 PM — quietNov 6, 5 PM — quietNov 6, 8 PM — quietNov 6, 11 PM — quietNov 7, 2 AM — quietNov 7, 5 AM — quietNov 7, 8 AM — quietNov 7, 11 AM — quietNov 7, 2 PM — quietNov 7, 5 PM — quietNov 7, 8 PM — quietNov 7, 11 PM — quietNov 8, 2 AM — quietNov 8, 5 AM — quietNov 8, 8 AM — quietNov 8, 11 AM — quietNov 8, 2 PM — quietNov 8, 5 PM — quietNov 8, 8 PM — quietNov 8, 11 PM — quietNov 9, 2 AM — quietNov 9, 5 AM — quietNov 9, 8 AM — quietNov 9, 11 AM — quietNov 9, 2 PM — quietNov 9, 5 PM — quietNov 9, 8 PM — quietNov 9, 11 PM — quietNov 10, 2 AM — quietNov 10, 5 AM — quietNov 10, 8 AM — quietNov 10, 11 AM — quietNov 10, 2 PM — quietNov 10, 5 PM — quietNov 10, 8 PM — quietNov 10, 11 PM — quietYesterday, 2 AM — quietYesterday, 5 AM — quietYesterday, 8 AM — quietYesterday, 11 AM — quietYesterday, 2 PM — quietYesterday, 5 PM — quietYesterday, 8 PM — quietYesterday, 11 PM — quietToday, 2 AM — quietToday, 5 AM — quietToday, 8 AM — quietToday, 11 AM — quietToday, 2 PM — quietToday, 5 PM — quietToday, 8 PM — quiet 1–2
Oct 31Nov 1Nov 2Nov 3Nov 4Nov 5Nov 6Nov 7Nov 8Nov 9Nov 10yesterdaynow · 9:20 PM ET
  1. 2

    Microsoft details separate 'AI Grounding' crawling control

    Microsoft said in a blog post it will offer an additional setting called AI Grounding, letting site owners block AI models from accessing pages to answer user queries rather than for training, extending to third-party tools using Microsoft technology.

    • coywolf@coywolf.social

      A new policy token in robots.txt lets site owners set one AI training preference across major AI providers, though rollout timing varies and the signal isn't enforceable. https:// coywolf.com/news/seo/google-op enai-microsoft-and-anthropic-back-a-shared-robotstxt-ai-training-signal/ # AI # SEO

      coywolf@coywolf.socialMastodon10d ago2▲view on Mastodon ↗
    2 more of the top 3 · 4 posts in this stretch
    • phipi.me

      OpenAI, Google, Microsoft and Anthropic say they'll honor one shared robots.txt token that opts a site out of AI training. Anthropic supports it now, Google and Microsoft plan to next year, OpenAI in "coming months." Via @theverge.com #AI ✍️ Gloria

      phipi.meBluesky10d agoview on Bluesky ↗
    • mascottus@veganism.social

      Google, OpenAI, Microsoft et Anthropic adoptent une règle robots.txt commune contre l’entraînement IA https://www. blogdumoderateur.com/google-op en-ai-microsoft-anthropic-regle-robots-txt-commune-contre-entrainement-ia/

      mascottus@veganism.socialMastodon9d agoview on Mastodon ↗
    all of them →
  2. background

    Microsoft, Anthropic and OpenAI join Google in backing the standard — SiliconANGLE reported that Microsoft, Anthropic and OpenAI also plan to support the new robots.txt crawling standard, which has already been submitted as a draft to the IETF, while Meta backs the idea but hasn't committed to implementing it.

  3. 1

    Google announces shared AI-training opt-out plan for robots.txt

    Google published a blog post announcing plans to support a cross-provider 'policy token' in robots.txt, built on IETF AIPREF working group standards, extending its existing Google-Extended control.

    “Today, we're announcing our plans to support a shared AI training control in robots.txt.”
    — Google
    1. 2 outlets Google, Microsoft, OpenAI and Anthropic back one AI training opt-out

      first by PPC Land, 13d ago · also Thurrott

      2 more headlines
    2. 1 outlet Google, Anthropic and others to support new AI crawling opt-out standard

      first by SiliconANGLE, 13d ago

What people are saying 1 voices from 1 site · best of 4 · verbatim