SoL-Pi: Researchers Cut Coding Agent Costs by a Third
MIT, NVIDIA, and NTU researchers report token-efficient harness design cuts API costs by ~33% while matching performance.
What to know
- SoL-Pi identifies four reusable harness mechanisms that cut token traffic by ~45% and API costs by ~33% while maintaining performance parity with baseline agents.
- The finding that harness design—not just model capability—significantly impacts coding agent efficiency has sparked community interest in component-level benchmarking.
- Researchers and practitioners are debating terminology ('harness' as enabler vs. guardrail), which models belong in benchmarks, and whether lab results transfer to production use cases.
The dispute Whether the research applies meaningfully to production systems and current frontier models, and what 'harness' actually means in different contexts. · positions read across 12 posts and comments
Harness design legitimately matters and needs more rigorous, component-level study.
-
“we definitely need more principled studies on the role of harnesses. I'd also say that there aren't too many benchmarks where the more complicated harnesses consistently outperform extremely simple agents.”
lieret · Hacker News ↗
The research is incomplete without testing on frontier open-source and Chinese models.
-
“This is done on Nemotron models + mistral, so its not very relevant to the current frontier of cheap chinese models + big models from Claude/GPT. Big miss not having qwen or deepseek in this research.”
vblanco · Hacker News ↗
Lab benchmarks may not translate to real-world performance differences.
-
“I ran mouse on the Frontier Harness benchmark and scored the highest pass rate, however I'm not convinced the results there actually translate to meaning the 'best' harness in practice.”
Aeroi · Hacker News ↗
“Token efficiency therefore becomes important for scaling recursive self-improvement.”
SoL-Pi authors (Haozhe Liu et al.), Researchers · arXiv ↗ · Sep 17
Haozhe Liu Lead researcher, SoL-Pi studyRun-Ze Fan Lead researcher, harness components studyNVIDIA, NTU, MIT Collaborating institutions
How it unfolded 3 developments, newest first · click a bar or a number to jump articlespostscomments
-
3
SoL-Pi findings show 44–49% token reduction and ~$5–$13 hourly savings
Detailed results from the SoL-Pi study show token traffic falls 44.7–49.0%, API costs drop by roughly one third, with estimated hourly savings of $8.75–$13.50 against native Codex and Claude Code harnesses and $4.36–$5.71 against the Pi baseline. Performance matches across GPT-5.6 Sol and Opus 5 on the 51-task EdgeBench evaluation.
“estimated hourly savings are $8.75-$13.50 relative to native Codex and Claude Code harnesses, and $4.36-$5.71 relative to Pi.”
— SoL-Pi researchers, Authors · source -
Something that totally confuses me is the use of the term “harness.”Is it the harness that enables the LLM to use tools and implement plans, or is it the short leash that, through many guardrails, ensures that the LLM follows the desired path?I see both meanings used. The former is still plausible, but I see many people using the second…
2 more of the top 3 · 4 posts in this stretch
-
Planning and tooling are suitably designed for execution. That's why they fail to decide on the resulting coding accuracy. If the product is user-facing (i.e. all products), a blind verification harness that is specifically designed to behave like a real user should be the decider.
-
Love your analogy.Cars performance is hugely dependent on use case and overall setup (suspension, engine (NA, turbo, supercharger, etc), coef of drag, etc).Maps well onto the LLM surface.
-
-
2
Research community discusses harness design and benchmarking trade-offs
Hacker News users engage with the empirical harness study, debating the generalizability of findings across model families, the definition of terms like 'bash capable' and 'harness,' and whether existing benchmarks capture real-world performance differences.
“we definitely need more principled studies on the role of harnesses. I'd also say that there aren't too many benchmarks where the more complicated harnesses consistently outperform extremely simple agents.”
— lieret -
The conclusions:> Planning improves success at additional cost for weaker models but mainly reduces cost, with small decreases in success rate, for stronger models.> Predefined tools raise success rates for models with weak bash control, whereas bash-only yields higher success at lower cost for bash-capable models, most clearly on shell-centric…
2 more of the top 3 · 8 posts in this stretch
-
Haven't gone through full PDF as its very detailed, few things have resonated with me so far.Basically if a Car A is performing better (be it speed, milage or in general sense) than Car B, then it is not necessarily because its engine. It could be because of better tires, better gearbox, lighter body, better usability of features, etc.You can…
-
Quite inline with what I had found with my Claude code sessions over the last year. I wrote about this a few months ago.https://rahulmax.com/notes/how-i-keep-the-ai-bill-down/In their case, context management pays off more the tighter your window. Their gap between managing and not managing is 35.7 points of success rate at 32k and 2.7 points at…
-
-
1
Red-teaming study examines blocking classifiers for malign coding agents
Alex Remedios and colleagues release research on red-teaming auto mode—the production blocking monitors in Claude Code and OpenAI's Codex—testing their robustness when agents are persistently misaligned, not just accidentally harmful.
-
first by HN Best, 9d ago · also HN Frontpage, arXiv cs.AI
-
1 outlet first by arXiv cs.AI, 9d ago · read ↗
-
-
background
Empirical study of harness components appears on arXiv — Run-Ze Fan and colleagues publish an empirical study examining individual components of coding harnesses—planning, action space, and other elements—to understand which factors drive coding agent performance, moving beyond treating harnesses as monolithic systems.
-
background
Researchers publish SoL-Pi harness optimization study — Haozhe Liu and colleagues at NVIDIA, NTU, and MIT announce SoL-Pi, which applies recursive auto-research loops at the harness layer to optimize coding agent efficiency. The work identifies four mechanisms—Action Fusion, Online Context Compact, ObservationPack, and Evidence-Preserving Reducer—that survive selection and transfer across diverse environments.
What people are saying 6 voices from 1 site · best of 12 · verbatim
- What does 'bash capable' mean formally, and how do you measure it?
- Do SoL-Pi's findings transfer to open-source models and to the current frontier (Qwen, DeepSeek, latest Claude/GPT)?
- How much of the measured improvement actually matters in user-facing products, versus lab benchmarks?
- Sep 18
-
so then what I’m really interested in are benchmarks of OSS models vs closed source frontier models, using Claude Code as a harness
-
nice work and paper, seeing more harness benchmarks emerge and we definitely need more. I ran mouse on the Frontier Harness benchmark and scored the highest pass rate, however I'm not convinced the results there actually translate to meaning the "best" harness in practice.Always looking for more harness evals, although I'm going broke running them…
-
Cool study, we definitely need more principled studies on the role of harnesses. I'd also say that there aren't too many benchmarks where the more complicated harnesses consistently outperform extremely simple agents. But I'm also biased, because I wrote https://github.com/swe-agent/mini-swe-agent/ , which is probably the most minimal agent out…
-
As far as I can tell, the paper says "bash capable", without ever describing what that means. How would one know whether a given model is "bash capable" or not?I would have to imagine, that Luna would very much fall into the camp of "bash capable". At which point- it seems to me that adding any tools beyond just Bash requires some rigorous testing…
-
This is done on Nemotron models + mistral, so its not very relevant to the current frontier of cheap chinese models + big models from Claude/GPT. Big miss not having qwen or deepseek in this research.
-
Todo/task-tracking tools (TaskCreate/Get/Update/List, TodoWrite) are no longer available on Opus 4.8, Sonnet 5, Fable 5, Mythos 5, and newer models; set CLAUDE_CODE_ENABLE_TODO_TOOLS=1 to bring them back"Anthropic appears to agree frontier models don't need in-session planning tools.