Model-specific tool design emerges as critical harness factor
4 Sep 17 1:27 AM · 9d ago · 2 comments · 1 source · development 4 of 7
A commenter surfaces evidence that harness effectiveness depends heavily on matching tool interfaces to what the target model was fine-tuned on. Claude models perform better with custom edit functions, while GPT models prefer patch-based approaches. Newer models reportedly work better with naive tool calls but worse with custom tools resembling default ones.
“What matters more is that you use the tools that the target model was fine-tuned on…newer models are better at naive harness tool calls and worse at custom tools that look similar to default tools.”
lukaxmatt_d HarnessTax research author/submitterYashjain413 HN commenterSupermancho HN commenternojs HN commenterlukax HN commenter
The whole story articlespostscomments the bright band is this development · numbered dots are the others · click one to jump
What people said 2 voices · verbatim
-
What matters more is that you use the tools that the target model was fine-tuned on.E.g. for editing files with Claude models you should use Edit(file_path, old_string, new_string, replace_all) but with GPT models you should use apply_patch_call(patch) (where patch is a custom patch string with custom grammar).It appears newer models are better at…
-
A good benchmark would require a decent number of smaller scoped one off tasks to larger multi step refactors, and also one shot full project of simple to complex varieties. In addition to a series of “conversational” ambiguity filled one-liners.
All 7 developments of HarnessTax study: how much does the harness matter for… →
Hacker NewsMastodonNewswiresLobsters