Sean Goedecke debunks "superhuman" LLM weight-editing claims
A researcher argues that GPT-6 Astra's ability to directly modify neural network weights, while impressive, is not genuinely superhuman or a breakthrough.
What to know
- A paper claimed GPT-6 Astra can read and edit neural network weights at a superhuman level, raising concerns about potential recursive self-improvement.
- Sean Goedecke argues the feats—identifying game models and fixing a Qwen LLM's mathematical looping—use standard ML techniques that competent human researchers routinely apply.
- The core disagreement hinges on whether direct weight editing represents a genuine capability leap or is merely an impressive but not revolutionary tool.
- Even if weight-editing becomes practical, Goedecke argues normal training remains the better approach for model improvement.
The dispute Whether LLM weight-editing represents a breakthrough capability that signals recursive self-improvement or a clever but non-revolutionary application of standard ML techniques. · positions read across 11 posts and comments
Weight-editing by LLMs is a significant breakthrough that could enable recursive self-improvement and change AI research fundamentally.
-
“Superhuman ability to improve language models is precisely the kind of thing that could cause recursive self-improvement.”
Paper authors (as reported) · seangoedecke.com ↗
Weight-editing is an impressive capability but relies on standard ML techniques and should not be framed as genuinely superhuman or transformative.
-
“I think a smart human ML researcher could perform all of these tasks.”
Sean Goedecke · seangoedecke.com ↗
“It's neat that Astra was able to calculate the norms in its head without executing Python, but not that impressive, and certainly not something that should shock anyone who's been paying attention.”
Sean Goedecke, ML researcher and analyst · seangoedecke.com ↗ · Nov 1
Sean Goedecke ML researcher and analystGPT-6 Astra Frontier LLM model
How it unfolded 2 developments, newest first · click a bar or a number to jump articlesposts
-
2
Goedecke's post gains traction on Hacker News
The analysis is shared on Hacker News where it reaches 51 points and sparks discussion. The post gains visibility as the technical community engages with the argument that weight-editing, while a neat capability, does not represent a paradigm shift in AI research.
-
One could say they might be "superhuman" here in the same way that LLMs might be said to be superhuman at math. Perhaps they do this work faster and more effectively than any person could manually. But they are certainly still doing it the "human" way ie reasoning with legible and coherent thoughts, rather than communing through some Matrix-like…
2 more of the top 3 · 11 posts in this stretch
-
False, in a way that genuinely matters.True: they are superhuman at every one of these tasks, in the boring way they are better—according to a wall clock and the coupled standard of actual results—at other things, just not in some mystical way.
-
> the best way to work with binaries is still to write source code and compile it. Likewise, the best way to update the weights of a model is to train it normally.This is why the distinction between open weights and open source is important.
-
-
1
Goedecke argues the results are not genuinely superhuman
Sean Goedecke publishes a technical analysis rejecting the paper's framing. He argues that identifying games from activation heatmaps is straightforward visual pattern-matching, and that Astra's weight-editing approach—using steering and ablation—matches the standard methodology any competent ML researcher would deploy.
“A smart human ML researcher could perform all of these tasks.”
— Sean Goedecke -
first by AIM, 5d ago
1 more headline
- LLMs are not superhuman at reading and writing neural network weights seangoedecke.com · 5d ago
-
-
background
Paper claims GPT-6 Astra can superhuman-level manipulate neural network weights — A paper argues that frontier LLM GPT-6 Astra can interact directly with neural network weights "well past what we'd expect from humans or conventional algorithms." Key demonstrations include identifying games from network weights and editing a 0.8B Qwen model to fix its mathematical reasoning beyond 32k tokens.
What people are saying 6 voices from 1 site · best of 11 · verbatim
- Nov 4
-
I disagree with the last paragraph. Training is costly because data is costly. Reduced reliance on training is a significant development.
- Nov 3
-
I would have gotten none of them right. Now if you said "here are the four possible games, now match them up" then I probably could do decently. But if you gave me nothing and said tell me the game -- I'd get 0 out of the 4.
-
Really? Pac-Man seemed easiest to me.(One possible disconnect: each row in the grid is one game, so Pac-Man is the third row, not the third image.)
-
We keep moving the goalposts, I love it. A few years ago LLMs writing simple functions was downplayed and now this. I lovehate the future this will bring.
-
"If someone showed you that heatmap without the game labels, I think you’d be able to immediately identify at least the first three games" -- absolutely not. Pac-Man in particular seems like a huge stretch to me.
-
"I think a smart human ML researcher could perform all of these tasks."And do that for how long?