Open reproduction confirms 5-agent Qwen3.8-27B swarm can beat Fable 5 on coding
An independently rebuilt Memdoor workflow reruns the GVS5H multi-agent method and publishes all 50 graded runs, matching its original claim.
What to know
- An independent, byte-for-byte reproduction of the GVS5H multi-agent method was rebuilt as an open Memdoor workflow and run on the same open model, Qwen3.8-27B, rather than taken on faith.
- The rebuilt five-agent team scored 9.8/10 versus 9.6 for the original harness and 8.0 for Fable 5 alone on ten LiveCodeBench Hard problems, with all 50 runs, costs (~$35.79–$38.77) and infrastructure logs published for scrutiny.
- The author flags real caveats: only 10 of the paper's 100 problems were tested, the contest problems may have been in the model's training data, and host hardware/precision differed from the original paper's own FP8 vLLM setup.
- HN discussion centers on whether consumer hardware can now run Fable-5-level coding agents locally, alongside skepticism about benchmark size/saturation and worries about censorship of open-weight models.
The dispute Whether beating Fable 5 on just ten possibly-saturated benchmark problems is meaningful evidence of real capability parity, or an overstated result from an underpowered test. · positions read across 35 posts and comments
This shows Fable-5-level coding performance is achievable on local/consumer hardware with good harnessing.
-
“Which in turn suggests that Fable 5 coding performance can be achieved locally with a bit of clever harnessing.”
roenxi · Hacker News ↗
The benchmark is too small and possibly contaminated/saturated to support strong conclusions.
-
“I'd wait for more benchmarks before making any conclusions since this benchmark has only "The first ten problems of their LiveCodeBench Hard list". And LiveCodeBench Hard is a several years old benchmark. Could be very saturated”
zihotki · Hacker News ↗
Keeping local/open models competitive matters because proprietary providers are increasingly restrictive and uncensored weights may disappear.
-
“We desperately need to keep local models competitive with the hyper scalers. I won't want to be beholden to them and their whims forever. With anthropic now restricting game reverse engineering I think we're about to cross a Rubicon of…”
fedpost · Hacker News ↗
guregodevo Author/submitter of the Memdoor reproductionLee Author of the original GVS5H paper (2026)Fable 5 Proprietary model used as the comparison baselineQwen3.8-27B Open-weight model used in the five-agent team
How it unfolded 4 developments, newest first · click a bar or a number to jump articlespostscomments
-
background
HN commenters debate local hardware feasibility — Several commenters described running or planning to run Qwen3.8-27B-class models on consumer GPU rigs, arguing the result shows Fable-5-level coding performance may be achievable locally with better harnessing.
-
4
Skeptics question benchmark size and contamination risk
Other commenters cautioned that the test used only ten problems from an older, possibly saturated benchmark and that the underlying result needs more data before drawing conclusions.
“I'd wait for more benchmarks before making any conclusions since this benchmark has only "The first ten problems of their LiveCodeBench Hard list".”
— zihotki -
> Five small models solve a programming problem together: a planner, two ideators, a solver and a manager, plus a repair worker. Each starts from a fresh context and talks only through files on disk. This seems to tie back to the paper Adaptive Auto-Harness: Sustained Self-Improvement for Agentic System Deployment on Open-Ended Task Streams[0].In…
2 more of the top 3 · 28 posts in this stretch
-
I'm not surprised.I've been running something similar for few weeks, but with Qwen3.8-Flash-Next. I started with opus 4.8 limited to 200k context as the supervisor and 5 Qwen3.8-Flash-Next agents(262k context each, autocompacted). A very simple way of calling agents via a command line using pi.Telling agents to maintain task status in files.This…
-
So far, the comments have focused on hopes for the effectiveness of open-weight models versus proprietary frontier models. That's fine, but I'm thinking about the approach and the results, as opposed to the model.Taking the results at face value, what is it about the "team" that makes the system so effective? That's the key here: presumably, the…
-
-
3
Commenters link result to open-model censorship concerns
Some HN commenters tied the result to broader worries about proprietary model providers restricting usage and about uncensored open weights being removed from hosting platforms.
“We desperately need to keep local models competitive with the hyper scalers. I won't want to be beholden to them and their whims forever.”
— fedpost, HN commenter · source -
Remember those gpus into chips https://inference-docs.cerebras.ai/models/qwen-3.8-27b , where you have this chip that implements a model that run at those extremely high speeds... the ones that people ask "why do we need that"... if this is true... one of those chips with the correct harness is all that's needed to get fable like intelligence in…
2 more of the top 3 · 4 posts in this stretch
-
The nice thing is that current local modes don't go away. The weights are out. Qwen 27b and deepseek are already very useful. Hardware will keep getting stronger so after the current memory crisis you can run them more cheaply.Of course the proprietary models will continue to improve and we have no guarantee that there will be new open models. But…
-
Wait and see, I give it fairly high odds everything tagged as 'uncensored' or 'abliterated' will get wiped from huggingface at some point in the future. The main barrier to wider spread adoption of this right now is that very few people have the legit on-prem hardware resources/budget to run something like an abliterated Q8 of deepseek v4.1 flash…
-
-
2
Project spreads to Mastodon and HN Frontpage RSS feeds
The same Memdoor writeup was mirrored on Mastodon and syndicated via HN's frontpage RSS feed within hours of the original post.
“Which in turn suggests that Fable 5 coding performance can be achieved locally with a bit of clever harnessing.”
— roenxi, HN commenter · source -
That's pretty interesting, because with the Bonsai 2 quantisations it is quite achievable to get Qwen 27B running at 50 t/s on reasonably priced gaming machine [0]. Which in turn suggests that Fable 5 coding performance can be achieved locally with a bit of clever harnessing.Although the same harnessing should also improve Fable's performance. The…
1 more of the top 2 · 3 posts in this stretch
-
This also appeared on HN last week: https://news.ycombinator.com/item?id=50228167, it's the other method they mention, GVS5H
-
-
1
Memdoor publishes full parity benchmark beating Fable 5
The rebuilt harness, run on OpenRouter with the authors' exact prompts, scored a mean 9.8/10 over five passes on ten LiveCodeBench Hard problems versus 9.6 for the original harness and 8.0 for Fable 5 alone; all 50 runs, costs and hosts were published, with 49 of 50 problems solved.
“49 of 50 solved; the 50 runs cost $35.79 (15.3M output tokens), $38.77 with the 16 attempts that were cut by a clock, stopped for a fix or paused.”
— guregodevo (Memdoor README) -
background
Coder generates Memdoor's rebuild of the GVS5H harness — Starting only from a description of the GVS5H method, an AI coding agent wrote the Memdoor workflow files itself, implementing the five-role planner/ideator/solver/manager/repair-worker loop.
-
background
Lee publishes GVS5H paper claiming Qwen swarm matches Fable 5 — The GVS5H paper (Lee, 2026) claimed a five-model team of Qwen3.8-27B agents could match the proprietary Fable 5 model on LiveCodeBench Hard; it was discussed on HN about a week before the Memdoor reproduction.
Also covered reported alongside — the timeline has no entry for these yet
-
first by HN Best, 11d ago · also HN Frontpage
What people are saying 14 voices from 1 site · best of 35 · verbatim
- Does the Memdoor multi-agent harness generalize to agentic tasks beyond coding (e.g., monitoring or long-running tasks)?
- Is LiveCodeBench Hard saturated or contaminated by training data, undermining the comparison?
- Nov 1
-
> With anthropic now restricting game reverse engineering I think we're about to cross a Rubicon of censorship.This might be the undoing of the frontier models: if they're impinging on more or less legitimate use cases with their restrictions people will turn to open models that get out of the way and let them get on with what they need to.
-
Local models cannot win. Local GPUs for gaming win over nvidia geforce now because of network bandwidth and latency and that cost is within the same order of magnitude as owning a GPU (though still substantially cheaper)With LLMs, network latency hardly matters compared to overall response generation time, and GPUs processing multiple requests…
-
That's the point I am really convinced today that few small size agents actually better than one remote begemoth. I switched fully to local models (from claude, sol, and deepseek). A 27B model can be served on any modern PC or laptop with 64 GB or RAM (or VRAM on non-Macs). A bunch of Qwen coding agents is fantastic with opencode, works great also…
-
I use a similar approach for some types of work. I use a planner (read only), researcher (read only), implementer, and reviewer (read only). I allow the implementer to spawn researchers and reviewers to help it with its work. I interact with a controller to manage the subagent trees. The agents use a file-based communication system. I get really…
-
Open Weight models have already reached level at which you can do a lot of productive work with them. They really don’t have to keep beating newer and newer models, of course it’s great for them to do things cheaper and faster, but we’re already past point where open models have to “catch up”
-
I’m skeptical of the result as claimed BUT I really do think model diversity really helps.Depending on the feature, I often these days have opus plan first and work with it on business specs, then have it reviewed by 1 or all of the cheaper models: haiku, sonnet, terra, sol, grok, muse spark,Then take the updated plan in opus and forward to Astra…
-
I spent several weeks working on trying to get swarms of cheap models to match the coding ability of a single expensive one. My take: it's remarkably easy to build a harness that improves a single eval, and remarkably difficult to build one that generalizes.
-
If Qwen is so great, why did they use Claude to do the write-up? This text is pure, unfiltered Claude. (e.g. "Ten problems, five passes, not the authors' 100. On a sample this small, 9.8 and 9.6 are the same number. The claim is parity with their harness, not a gain on it.")
-
Yes it is extremely aggravating that Claude will often raise copyright objections even when doing something like putting images in a report for personal use. The AI companies are the most blatant copyright abusers in history.
-
Maybe slightly off-topic from the main article but is there a good resource or site that clearly helps understand which of the "open" models are good for programming tasks, like C, C++, Python, Java?I've looked and there are tons of sites that aggregate models vs. benchmarks but I'm not sure how much of that is trust worthy.
-
This is probably not a novel idea but I wonder how it would work if you just stacked a bunch of smaller models on top of eachother and used that as the base to train a much much larger model. Maybe more efficient than starting from scratch?
-
There is something to say about the power of brute force approaches. Right now scaling multi agent is seemingly what is becoming the next important axis, though very strongly constrained by quality of communication protocol and compute. Even if one model is not strong enough individually, more models and more attempts as well as differing opinions…
- Oct 31
-
So someone used a small model with many requests & a massive amount of tokens, to eventually beat a large model with a single request & tiny amount of tokens. .....Cool? Were they not expecting that to work?
-
This is good timing. My setup at home involves a two servers with 5080s each and an Rtx spark laptop with 128gb. I plan I have qwen flash next on the latter and the 27b variant on the former. I would’ve preferred to have purchased a dgx spark but everything is just ridiculously expensive now. My own setup would be 3x the cost to buy again.I’m…