The Kluever Maneuver: Writing Tests That Fail Before Fixing Bugs
Developer Jordi Gómez Hernández names a technique for writing regression tests that demonstrate bugs before fixes are applied.
What to know
- The Kluever Maneuver is a code-review technique where developers write tests demonstrating buggy behavior before applying fixes, ensuring the test actually fails on broken code and passes after the fix.
- The approach is already standard practice in several major projects (Rust, Cargo, LLVM) as a two-commit pattern: one commit with a failing test, one with the fix that makes it pass.
- Developers disagree on implementation details: some prefer explicit 'terrible tests' of incorrect behavior; others advocate 'expected to fail' annotations to encode intent more explicitly.
- The technique catches a gap in traditional fix-and-test workflows where a proposed bugfix might not actually address the problem, but the added test passes anyway.
The dispute Whether to write tests that pass with buggy behavior (Kluever approach) or write correct tests marked as 'expected to fail' (xfail approach), with advocates on both sides citing code clarity and maintainability. · positions read across 36 posts and comments
Write tests that assert buggy behavior before fixes, catching unrelated fixes.
-
“It cuts a commit down to the esentials to better see what changed and it is a much better illustration of what a PR does than the description.”
epage · Lobsters comment ↗
Use 'expected to fail' annotations instead, preserving intent without misleading tests in code.
-
“'Known failing' would let you do it in a more natural way: write the test for the correct behavior, and mark it as 'expected to fail'.”
nedbat · Lobsters comment ↗
The choice depends on clarity: use whichever method better communicates the bug to reviewers.
-
“For me, a key point of this is to make the change of behavior clear, complementing the PR description. If an 'expect fail' doesn't clearly communicate what operation failed and how, then I'd rather switch the code to assert that condition.”
epage · Lobsters comment ↗
Jordi Gómez Hernández Developer and writerKurt Kluever Developer (originator of technique)
How it unfolded 4 developments, newest first · click a bar or a number to jump postscomments
-
4
Developers debate tradeoffs between test approaches
Discussion emerges over whether 'expected to fail' annotations or explicit tests of buggy behavior better serve code review clarity and maintainability, with no consensus reached.
“For me, a key point of this is to make the change of behavior clear, complementing the PR description. If an 'expect fail' doesn't clearly communicate what operation failed and how, then I'd rather switch the code to assert that condition.”
— epage -
"Known failing" would let you do it in a more natural way: write the test for the correct behavior, and mark it as "expected to fail". Then you can commit the fix to the code, with the change to remove the fail expectation. The Kluever Maneuver seems odd to me: write the test confirming the wrong behavior. I'd rather have the first commit confirm…
2 more of the top 3 · 30 posts in this stretch
-
I don't think I like committing passing but known-wrong tests to the codebase, even if immediately patched. But I could see the argument if you mark the test `XFAIL` and then fix the bug in the commit that removes the `XFAIL` mark.
-
https://lobste.rs/c/72mqyb proposes using "expect fail" for such tests at first, which encodes the expectation in a way that is verifiable, and easy to flip (and to verify that change to the test) when the fix arrives.
-
-
3
Rust contributors confirm widespread adoption of two-commit bug-fix pattern
Jana notes the technique is standard practice in Rust language contributions, where bug-fix pull requests split into two commits: one demonstrating the bug, one fixing it.
“I love this. It's one of the things I very often ask new contributors to rust-lang/rust to do when reviewing, and it's become kind of a standard. Any bug fix pr is two commits, and imo that's extremely important.”
— jana -
I love this. It's one of the things I very often ask new contributors to rust-lang/rust to do when reviewing, and it's become kind of a standard. Any bug fix pr is two commits, and imo that's extremely important.
2 more of the top 3 · 3 posts in this stretch
-
I'm familiar with this from contributing to LLVM.
-
I usually do this by adding a test for the fixed bug and a test that demonstrates the bug in action, but this is an interesting modification of the idea.
-
-
2
Developers cite prior art and alternative methodologies
Comments note that similar techniques have been used in projects like Aegis, Cargo, and the Rust compiler, with some preferring 'expected to fail' annotations over writing tests that pass with incorrect behavior.
“So you submit a change, which comes with a test (for new features, the test would fail because there's no feature to exercise), and aegis would only merge the change if the test-on-its-own failed and passes with the change.”
— pgeorgi -
A couple, yeah. TDD would have you write a test asserting the *correct* behavior that initially fails, while this methodology has you write a test asserting the *incorrect* that initially passes, but must be fixed. Also, more broadly, TDD is a practice that asserts that any code change, from new features to bug fixes, should be preceded by the…
2 more of the top 3 · 3 posts in this stretch
-
https://aegis.sourceforge.net/ went a step further and expected only the correct test, but had some rules on how it can backport it separate from the fix. So you submit a change, which comes with a test (for new features, the test would fail because there's no feature to exercise), and aegis would only merge the change if the test-on-its-own…
-
Is there a difference between this and TDD?
-
-
1
Jordi Gómez Hernández publishes article on regression-test technique
Article posted to Lobsters and Hacker News describing a method for writing tests that demonstrate bugs before fixes are applied, attributed to Kurt Kluever.
“But what if the test accidentally already passed before the alleged bugfix and the alleged bugfix didn't actually work? In that case the added test gives false confidence that the bugfix worked.”
— Jordi Gómez Hernández, Author · source
What people are saying 16 voices from 1 site · best of 36 · verbatim
- Nov 2
-
It no doubt sounds like a negative act first when you frame it like that. I think a _worse_ framing of the situation is that broken behavior is already committed and is also not covered by test. Viewed like that, you are first raising the code coverage and then demonstrably correcting a bug, both now positive acts on the codebase.
- Nov 1
-
Unless you squash-merge (in which case I would write the breaking test locally, fix it, and commit a single change with just the new passing test), your proposal still means that green-but-broken tests will make it onto `master` and then you could get dubious `git bisect` results, etc.
-
My PRs get squashed into main so I dunno. Maybe it helps the programmer figure out what was wrong, but it doesn't persist in git history
-
Probably multiple ways. The first one I can think of: The commit introducing the fix introduces another bug and gets reverted by another developer who is not familiar with the Kluever Maneuver. Development can be messy and slip ups happen. I don‘t think this approach is bad, but I think it breaks the assumption that a commit change is atomic.
-
this seems good if you're already heavy on process in a large team. i would want to require the bug/fixme comment as part of the process, when doing this. you could even commit the reproduction before the fix, as part of triage, if it's annotated as `[BugReproduction(issue#)]` or something so that if someone breaks the test they can tell it wasn't…
-
If you're bisecting a (new) failing test, having test failures in your history makes that harder to automate.
-
Can you not bisect when there's a test failing? Commit atomicity as far as I understand is a concept that (often) includes the requirement that tests pass after every commit, but to me it's still not clear why having a failing CI on some commit is a bad thing. General code hygeine is very general and subjective and again, I don't see the problem…
-
Bisecting, commit atomicity, general code hygiene, just to name a few?
-
This sound ingenious until the test which asserts the wrong behavior somehow slips into production and now that is the new generally desired behavior. May be better tooling with a new test marker is the answer.
-
Pytest’s XFAIL can take an exception type, but it’s absolutely possible for a given exception kind to be raised for an unexpected reason, so strictly speaking no. > I don’t know if that’s very useful but also those tests should have a TODO comment either way. Pytest supports a `reason` parameter though it is not required by default. Also from…
-
In our case, it's mostly because we're using Java. I added a section addressing this concern:
- Oct 31
-
That would be significantly easier for me to understand when reading the history of the fix.
-
One related thing I like using is pytest's xfail, with strict=True. You can then write the test to expect the desired behavior rather than any specific incorrect behavior, and you'll be obligated to remove the xfail when you fix the bug.
-
Can “expected to fail” catch if your test fails for a different reason than intended? > Doing it this way could also let you separate the two commits by a long period of time without having misleading tests in the code. I don’t know if that’s very useful but also those tests should have a TODO comment either way.
-
this is why unittest libraries have "known failing" annotations.
-
We independently developed this in Cargo and it is a big help. It cuts a commit down to the esentials to better see what changed and it is a much better illustration of what a PR does than the description. My write up for this is at