Skeptics question benchmark size and contamination risk
4 Oct 31 8:52 PM · 12d ago · 1 post · 3 comments · 2 sources · development 4 of 4
Other commenters cautioned that the test used only ten problems from an older, possibly saturated benchmark and that the underlying result needs more data before drawing conclusions.
“I'd wait for more benchmarks before making any conclusions since this benchmark has only "The first ten problems of their LiveCodeBench Hard list".”
zihotkiguregodevo Author/submitter of the Memdoor reproductionLee Author of the original GVS5H paper (2026)Fable 5 Proprietary model used as the comparison baselineQwen3.8-27B Open-weight model used in the five-agent team
The whole story articlespostscomments the bright band is this development · numbered dots are the others · click one to jump
What people said 24 voices · best of 28 · verbatim
-
> Five small models solve a programming problem together: a planner, two ideators, a solver and a manager, plus a repair worker. Each starts from a fresh context and talks only through files on disk. This seems to tie back to the paper Adaptive Auto-Harness: Sustained Self-Improvement for Agentic System Deployment on Open-Ended Task Streams[0].In…
-
I'm not surprised.I've been running something similar for few weeks, but with Qwen3.8-Flash-Next. I started with opus 4.8 limited to 200k context as the supervisor and 5 Qwen3.8-Flash-Next agents(262k context each, autocompacted). A very simple way of calling agents via a command line using pi.Telling agents to maintain task status in files.This…
-
So far, the comments have focused on hopes for the effectiveness of open-weight models versus proprietary frontier models. That's fine, but I'm thinking about the approach and the results, as opposed to the model.Taking the results at face value, what is it about the "team" that makes the system so effective? That's the key here: presumably, the…
-
This is good timing. My setup at home involves a two servers with 5080s each and an Rtx spark laptop with 128gb. I plan I have qwen flash next on the latter and the 27b variant on the former. I would’ve preferred to have purchased a dgx spark but everything is just ridiculously expensive now. My own setup would be 3x the cost to buy again.I’m…
-
Local models cannot win. Local GPUs for gaming win over nvidia geforce now because of network bandwidth and latency and that cost is within the same order of magnitude as owning a GPU (though still substantially cheaper)With LLMs, network latency hardly matters compared to overall response generation time, and GPUs processing multiple requests…
-
That's the point I am really convinced today that few small size agents actually better than one remote begemoth. I switched fully to local models (from claude, sol, and deepseek). A 27B model can be served on any modern PC or laptop with 64 GB or RAM (or VRAM on non-Macs). A bunch of Qwen coding agents is fantastic with opencode, works great also…
-
There is something to say about the power of brute force approaches. Right now scaling multi agent is seemingly what is becoming the next important axis, though very strongly constrained by quality of communication protocol and compute. Even if one model is not strong enough individually, more models and more attempts as well as differing opinions…
-
I’m skeptical of the result as claimed BUT I really do think model diversity really helps.Depending on the feature, I often these days have opus plan first and work with it on business specs, then have it reviewed by 1 or all of the cheaper models: haiku, sonnet, terra, sol, grok, muse spark,Then take the updated plan in opus and forward to Astra…
-
I use a similar approach for some types of work. I use a planner (read only), researcher (read only), implementer, and reviewer (read only). I allow the implementer to spawn researchers and reviewers to help it with its work. I interact with a controller to manage the subagent trees. The agents use a file-based communication system. I get really…
-
> With anthropic now restricting game reverse engineering I think we're about to cross a Rubicon of censorship.This might be the undoing of the frontier models: if they're impinging on more or less legitimate use cases with their restrictions people will turn to open models that get out of the way and let them get on with what they need to.
-
Maybe slightly off-topic from the main article but is there a good resource or site that clearly helps understand which of the "open" models are good for programming tasks, like C, C++, Python, Java?I've looked and there are tons of sites that aggregate models vs. benchmarks but I'm not sure how much of that is trust worthy.
-
Open Weight models have already reached level at which you can do a lot of productive work with them. They really don’t have to keep beating newer and newer models, of course it’s great for them to do things cheaper and faster, but we’re already past point where open models have to “catch up”
-
If Qwen is so great, why did they use Claude to do the write-up? This text is pure, unfiltered Claude. (e.g. "Ten problems, five passes, not the authors' 100. On a sample this small, 9.8 and 9.6 are the same number. The claim is parity with their harness, not a gain on it.")
-
I spent several weeks working on trying to get swarms of cheap models to match the coding ability of a single expensive one. My take: it's remarkably easy to build a harness that improves a single eval, and remarkably difficult to build one that generalizes.
-
I'd wait for more benchmarks before making any conclusions since this benchmark has only "The first ten problems of their LiveCodeBench Hard list". And LiveCodeBench Hard is a several years old benchmark. Could be very saturated
-
This is probably not a novel idea but I wonder how it would work if you just stacked a bunch of smaller models on top of eachother and used that as the base to train a much much larger model. Maybe more efficient than starting from scratch?
-
Yes it is extremely aggravating that Claude will often raise copyright objections even when doing something like putting images in a report for personal use. The AI companies are the most blatant copyright abusers in history.
-
So someone used a small model with many requests & a massive amount of tokens, to eventually beat a large model with a single request & tiny amount of tokens. .....Cool? Were they not expecting that to work?
-
What I find legitimately insane about all that is that Anthropic obviously does not have these censorship restrictions internally. So it's like.. They're just the whole world's adversary now?
-
This working journal is so much slop it’s basically impossible to read. Perhaps the author could write a short human written and readable summary so we can evaluate if this is interesting or not.
-
if and when the AI bubble bursts, we'll know if local models will hold up. Assuming the cyclical pullback in investment is strong enough to see local machines with 512gb or more of RAM
-
>Anthropic restricting game reverse engineeringWhich is literally the coolest thing to come out of this...Did they get a letter from Nintendo's legal team or something?
-
I agree. I am thinking this can only happen if we make training dramatically cheaper and efficient. This is why we are building a new kind of hardware.
-
People want the team on their own GPU not on OpenRouter. If you ran this where? a Mac with 24 GB card at home? a rented GPU by the hour?
All 4 developments of Open reproduction confirms 5-agent Qwen3.8-27B swarm can… →
Hacker NewsMastodonNewswires