HN commenters question methodology and compare it to prior AI-StarCraft work
2 Sep 19 4:30 PM · 6d ago · 9 comments · 1 source · development 2 of 3
Commenters asked whether agents played in real time via screenshots/clicks or through some other symbolic interface, and drew comparisons to a 2010 UC Santa Cruz Brood War AI tournament and to DeepMind's SC2 agents, as well as a separate LLM benchmark on 9x9 Go (GoBench).
“Did it play by looking at screenshots and sending clicks, or was there other mediation/symbolization?”
gadtflybenswerd Creator of Brood War BenchOpenAI Codex AI agent tested in the benchmarkGrok 4.6 (xAI) AI agent tested in the benchmarkClaude Fable (Anthropic) AI agent tested in the benchmark
The whole story articlespostscomments the bright band is this development · numbered dots are the others · click one to jump
What was reported 1 claim about this development
-
2 outlets Brood War Bench
first by HN Best, 7d ago · also HN Frontpage
What people said 9 voices · verbatim
-
Back in 2010, during the early days of bwapi, there was a Brood War AI tournament held by the Expressive Intelligence Studio at UC Santa Cruz. It's interesting to see how different the approaches were back then, vs this or Deepmind's SC2 work.https://web.archive.org/web/20091124210529/http://eis.ucsc.e...There's a great contemporary Ars Technica…
-
Unrelated to the benchmark...I love StarCraft. I started playing it right from the beginning, most of my friends right now are from that era. I literally met people that have spread to almost every continent when I was in my early teens. We played at internet cafes and did not have access to the internet, that was priced differently...I miss those…
-
A friend of mine created GoBench[1][2] that evaluates LLMs on 9×9 Go using KataGo opponents as Elo anchors, you see real capability differences there, like Astra Max substantially leading all other models. I think strategy is a generally interesting area to evaluate LLMs on[1]
-
Did it play by looking at screenshots and sending clicks, or was there other mediation/symbolization?It sounds like it might have been actually played in real time, which would be very important to distinguish.I have recently seen other harnesses letting agents play real-time games in what seems like discrete time slices, turning eg Portal into…
-
Oh boy, I've spent more hours playing it than I dare to admit. I won over 10000 battle.net games ... on just one of my several accounts :) When the SC2 beta came out, I played about 50-100 games and never bought the full game, because I knew it would be like heroin to me, and I was already an adult that had to take care of himself.
-
I agree.My first time playing StarCraft was at summer camp around a decade after it came out.All the smartest people played it so I wanted to too. Great decision, I have been continually impressed with the people who StarCraft introduced me to.
-
If you want to see human written bots in action or compete in the bot ladder yourself, try
-
Even at Burning Man, in the middle of the desert, there is a camp that hosts a StarCraft tournament every year (on the dustiest setups you've ever seen!) :)
-
There is currently a bot beating everyone on the ladder. Just watched it today on Artosiscasts yt channel.
All 3 developments of Brood War Bench pits AI coding agents against each other in… →
Hacker NewsNewswiresMastodon