Commenter proposes function-based benchmarking approach for harnesses
6Sep 17 8:34 AM · 9d ago · 1 comment · 1 source · development 6 of 7
A commenter who has been thinking about harness evaluation suggests that general benchmarking will be insufficient, proposing instead that benchmarks track gain and loss of specific functions alongside ability impact. References their own recent writing on the topic.
“I don't think yet that a general benchmark will capture what is needed, but a gain/loss of function along with improvement / loss along the ability of that function seems to be a good path forward.”
I've been thinking about this a lot in the past few weeks - https://www.roderick.dev/writing/2026-08-28-obsessing-harnes...I don't think yet that a general benchmark will capture what is needed, but a gain/loss of function along with improvement / loss along the ability of that function seems to be a good path forward.