SoL-Pi findings show 44–49% token reduction and ~$5–$13 hourly savings
3 Sep 18 6:40 PM · 7d ago · 1 post · 4 comments · 1 source · development 3 of 3
Detailed results from the SoL-Pi study show token traffic falls 44.7–49.0%, API costs drop by roughly one third, with estimated hourly savings of $8.75–$13.50 against native Codex and Claude Code harnesses and $4.36–$5.71 against the Pi baseline. Performance matches across GPT-5.6 Sol and Opus 5 on the 51-task EdgeBench evaluation.
“estimated hourly savings are $8.75-$13.50 relative to native Codex and Claude Code harnesses, and $4.36-$5.71 relative to Pi.”
SoL-Pi researchers, Authors · hn ↗Haozhe Liu Lead researcher, SoL-Pi studyRun-Ze Fan Lead researcher, harness components studyNVIDIA, NTU, MIT Collaborating institutions
The whole story articlespostscomments the bright band is this development · numbered dots are the others · click one to jump
Reported in the same hours no headline names this development itself — these 2 claims were published in its stretch
-
first by arXiv cs.AI, 3d ago
-
first by arXiv cs.AI, 3d ago
What people said 4 voices · verbatim
-
Something that totally confuses me is the use of the term “harness.”Is it the harness that enables the LLM to use tools and implement plans, or is it the short leash that, through many guardrails, ensures that the LLM follows the desired path?I see both meanings used. The former is still plausible, but I see many people using the second…
-
Planning and tooling are suitably designed for execution. That's why they fail to decide on the resulting coding accuracy. If the product is user-facing (i.e. all products), a blind verification harness that is specifically designed to behave like a real user should be the decider.
-
Love your analogy.Cars performance is hugely dependent on use case and overall setup (suspension, engine (NA, turbo, supercharger, etc), coef of drag, etc).Maps well onto the LLM surface.
-
so then what I’m really interested in are benchmarks of OSS models vs closed source frontier models, using Claude Code as a harness
All 3 developments of SoL-Pi: Researchers Cut Coding Agent Costs by a Third →
Hacker NewsNewswiresMastodon