Commenters call for better harness benchmarks and model-agnostic testing
2 Sep 17 12:08 AM · 9d ago · 1 post · 3 comments · 2 sources · development 2 of 7
Discussion surfaces a gap in available benchmarking: no reliable source comprehensively compares harnesses across open source models. Commenters note that common metrics like token efficiency don't capture full performance picture, and that benchmarking methodology is itself contested—some view harness as polish that prevents errors rather than fundamentally affecting intelligence.
“We really need better harness benchmarks. It seems there's no reliable source that benchmarks the main harnesses against all open source models.”
nojsmatt_d HarnessTax research author/submitterYashjain413 HN commenterSupermancho HN commenternojs HN commenterlukax HN commenter
The whole story articlespostscomments the bright band is this development · numbered dots are the others · click one to jump
What people said 4 voices · verbatim
-
We really need better harness benchmarks. It seems there's no reliable source that benchmarks the main harnesses against all open source models.I also wish the discussion around Pi did not always use cost/token count as the metric. It's amazingly token efficient, but how does it stack up again opencode and others if you don't care about token…
-
My own findings are in line with this research:Having a coding harness is critical but the differences between them are overstated.Personally, I’ve replaced OpenCode with a thin wrapper around Pydantic-AI as the pythonic analogue to Pi-Agent for headless use via HermesThey’d all do the job - I just prefer to compartmentalize for access…
-
Say I'm using Claude Code or GPT Codex's harnesses but also sending some queries to the respective Anthropic and OpenAI models via OpenRouter.Do harnesses and therefore sending the queries directly to the LLM providers have caching and other benefits that OpenRouter does not provide? Would I get any of those benefits if I simply proxied any…
-
Claude Code/Codex charge the user for their extremely bloated one-size-fits-all system prompts (including safety instructions and other stuff users dont want).In my experience if you're using OpenAI/Claude models and paying API costs, almost every other harness beats Claude Code/Codex in cost.
All 7 developments of HarnessTax study: how much does the harness matter for… →
Hacker NewsMastodonNewswiresLobsters