HarnessTax study: how much does the harness matter for coding agents?
Research publication examines the role of harness design in coding agent performance, sparking debate on benchmarking and model-specific optimization.
What to know
- HarnessTax research quantifies the role of harness design versus model selection in coding agent performance, challenging assumptions about which factor matters more.
- Commenters disagree on whether harness differences are substantial or overshadowed by other factors like concurrent execution, subagents, and model-specific tool design.
- No comprehensive benchmark currently compares harnesses across models and metrics; existing methodology gaps make it difficult to draw reliable conclusions.
- Harness importance scales inversely with model size: smaller models require sophisticated harness work for context management, while large models need minimal tooling.
The dispute Whether harness differences are material or overstated, and whether a single benchmark can meaningfully measure harness quality across different models and use cases. · positions read across 22 posts and comments
Harness is critical but differences between them are often overstated; what matters most is matching tools to the target model's fine-tuning.
-
“Having a coding harness is critical but the differences between them are overstated.”
corv · Hacker News ↗
The research methodology is flawed because it treats harness as independent of broader agent architecture; concurrent execution, subagents, and delegation matter more.
-
“The term "harness" here is being overloaded for the term "agent", which is worrying…Optimal harnesses use concurrent execution + subagents.”
Supermancho · Hacker News ↗
Harness importance scales inversely with model size; small models need sophisticated context and memory management, large models need only basic tooling.
-
“The harness becomes more and more important, the smaller the model is as you need to offload context management as well as memory to the harness.”
calgoo · Hacker News ↗
No reliable comprehensive benchmark exists; better harness evaluation requires function-level metrics rather than generic benchmarks.
-
“We really need better harness benchmarks. It seems there's no reliable source that benchmarks the main harnesses against all open source models.”
nojs · Hacker News ↗
matt_d HarnessTax research author/submitterYashjain413 HN commenterSupermancho HN commenternojs HN commenterlukax HN commenter
How it unfolded 7 developments, newest first · click a bar or a number to jump articlespostscomments
-
7
Commenter surprised by Pi harness efficiency matching larger models
A user expresses surprise that the Pi harness achieves similar efficiency to Claude Code and Codex, and wonders whether the same holds for smaller 9-32B models that would presumably require more steering.
“The most astonishing thing to me is that Pi harness is basically as efficient as the Codex/Claude.”
— Otterly99 -
I think it’s more likely that models have random behavior, and so the results in these dimensions are just random.
2 more of the top 3 · 11 posts in this stretch
-
I find that in many cases, you ARE the harness. Case in point, Terence Tao utilized the simple chat interface to find the Jacobian Conjecture counterexample. This is arguable no harness at all. I also find that many people will disagree with me but most of the time, they just want a button to press that will solve the problem. If that's the work…
-
> Claude Fable 5 solves 97.8% of attempts in Claude Code, 96.7% in Codex and 96.7% in Pi, yet Claude Code costs about twice as much as Pi ($1.33 vs $0.67). If the harness tax is much lower than the subscription subsidy, it makes sense to not use custom harnesses like pi. Right? It’s a shame i would really love to use alternative, less buggy…
-
-
6
Commenter proposes function-based benchmarking approach for harnesses
A commenter who has been thinking about harness evaluation suggests that general benchmarking will be insufficient, proposing instead that benchmarks track gain and loss of specific functions alongside ability impact. References their own recent writing on the topic.
“I don't think yet that a general benchmark will capture what is needed, but a gain/loss of function along with improvement / loss along the ability of that function seems to be a good path forward.”
— tomrod -
I've been thinking about this a lot in the past few weeks - https://www.roderick.dev/writing/2026-08-28-obsessing-harnes...I don't think yet that a general benchmark will capture what is needed, but a gain/loss of function along with improvement / loss along the ability of that function seems to be a good path forward.
-
-
5
Commenters report practical harness preferences and inverse scaling relationship
Users share experience that harness importance scales inversely with model size: smaller models require more sophisticated harness work for context and memory management, while large models need only basic tooling and manage context themselves. One commenter reports switching from OpenCode to Pi harness found it simpler and preferable.
“The harness becomes more and more important, the smaller the model is as you need to offload context management as well as memory to the harness.”
— calgoo -
Harness and benchmark for the harness feels like a chicken and egg problem. The harness is to optimize the interaction results with the models. Any benchmark for harness has to focus on the goals that the harness was trying to optimize for unless we are only focussing on generic harnesses.At this point when all the models have been trained on all…
1 more of the top 2 · 2 posts in this stretch
-
I really disliked how opencode works IMO; the harness tries to do to much in my mind. Switching to Pi was a breath of fresh air for me, and I even use hax for some of my local needs where i dont want to have the giant pile of fertilizer that is NPM or PIP installed.The harness becomes more and more important, the smaller the model is as you need…
-
-
4
Model-specific tool design emerges as critical harness factor
A commenter surfaces evidence that harness effectiveness depends heavily on matching tool interfaces to what the target model was fine-tuned on. Claude models perform better with custom edit functions, while GPT models prefer patch-based approaches. Newer models reportedly work better with naive tool calls but worse with custom tools resembling default ones.
“What matters more is that you use the tools that the target model was fine-tuned on…newer models are better at naive harness tool calls and worse at custom tools that look similar to default tools.”
— lukax -
What matters more is that you use the tools that the target model was fine-tuned on.E.g. for editing files with Claude models you should use Edit(file_path, old_string, new_string, replace_all) but with GPT models you should use apply_patch_call(patch) (where patch is a custom patch string with custom grammar).It appears newer models are better at…
1 more of the top 2 · 2 posts in this stretch
-
A good benchmark would require a decent number of smaller scoped one off tasks to larger multi step refactors, and also one shot full project of simple to complex varieties. In addition to a series of “conversational” ambiguity filled one-liners.
-
-
3
Commenters debate harness importance and context consumption trade-offs
Hacker News discussion reveals disagreement over how much the harness matters. One commenter argues harness is critical for context management, noting that different harnesses can consume vastly different amounts of context despite similar model call counts. Another challenges the framing, arguing the analysis conflates harness with agent design and overlooks factors like concurrent execution and subagents.
“The harness is basically the underlying source of truth…the hidden cost is really around context. Two different harnesses can make a similar number of model calls while consuming a very different amount of context.”
— Yashjain413 -
I think it’s really important, especially when you look at everything the tool does, from the execution loop and context management to feedback. The harness is basically the underlying source of truth.With coding agents, what I’ve noticed is that a simple task can often be handled with a fairly simple harness. But the hidden cost is really around…
-
-
2
Commenters call for better harness benchmarks and model-agnostic testing
Discussion surfaces a gap in available benchmarking: no reliable source comprehensively compares harnesses across open source models. Commenters note that common metrics like token efficiency don't capture full performance picture, and that benchmarking methodology is itself contested—some view harness as polish that prevents errors rather than fundamentally affecting intelligence.
“We really need better harness benchmarks. It seems there's no reliable source that benchmarks the main harnesses against all open source models.”
— nojs -
We really need better harness benchmarks. It seems there's no reliable source that benchmarks the main harnesses against all open source models.I also wish the discussion around Pi did not always use cost/token count as the metric. It's amazingly token efficient, but how does it stack up again opencode and others if you don't care about token…
2 more of the top 3 · 4 posts in this stretch
-
My own findings are in line with this research:Having a coding harness is critical but the differences between them are overstated.Personally, I’ve replaced OpenCode with a thin wrapper around Pydantic-AI as the pythonic analogue to Pi-Agent for headless use via HermesThey’d all do the job - I just prefer to compartmentalize for access…
-
Say I'm using Claude Code or GPT Codex's harnesses but also sending some queries to the respective Anthropic and OpenAI models via OpenRouter.Do harnesses and therefore sending the queries directly to the LLM providers have caching and other benefits that OpenRouter does not provide? Would I get any of those benefits if I simply proxied any…
-
-
1
HarnessTax research published examining harness impact on coding agents
A research study titled "HarnessTax: How Much Does the Harness Matter for Coding Agents?" was published and shared on Hacker News, reaching 214 points with 87 comments. The study investigates the relative importance of the harness—the execution framework and context management layer—versus the underlying model in coding agent performance.
“The term "harness" here is being overloaded for the term "agent", which is worrying…single-harness analysis is shallow and misleading.”
— Supermancho, HN commenter · source -
first by HN Best, 9d ago · also HN Frontpage
-
The term "harness" here is being overloaded for the term "agent", which is worrying. Putting that aside, there are many factors that matter. The "harness" context, the execution pattern (parallel vs sequential), the ability to delegate to other models, etc.Optimal harnesses use concurrent execution + subagents and are not stuck on one model. Cost…
-
What people are saying 8 voices from 2 sites · best of 22 · verbatim
- Do smaller models (9-32B range) require significantly more harness steering than the research suggests?
- How should harness benchmarks be structured to capture function-specific performance rather than generic cost/token metrics?
- Sep 19
-
We really do need better benchmarks and for models too- Most people use a harness because of its subscription (most companies pay Anthropic) - All model benchmarks are biased and gamed, harness benchmarks are too few to matter - Everyone is just guessing, acting on sample sizes of 1 and trust me bro vibes
- Sep 18
-
To clarify, this is true for Claude. A ChatGPT sub can be used in other harnesses.
-
I came to conclusion that all I really care about from the harness is interoperability. I want to be able to switch providers, models at any point in any session or between sessions in a single action, and I want my transcripts to be in one single format from the beginning till the end of time.I want one place where I define project prompt, one…
-
If you use the subscription your usage is heavily subsidized over the api billing. If you use the subscription you can’t use other harnesses. So to use another harness you have to give up the subscription subsidy,
- Sep 17
-
sub subsidy? are you talking about another adjustment that happened after their billing retraction?
-
I also test harness with "Ship Harness Bench", jcode with its browser integration gives the best results
-
The problem is that Pi, on any third party harness, cannot really compete with codex or cc due to subscriptions.
- Sep 16
-
Claude Code/Codex charge the user for their extremely bloated one-size-fits-all system prompts (including safety instructions and other stuff users dont want).In my experience if you're using OpenAI/Claude models and paying API costs, almost every other harness beats Claude Code/Codex in cost.