Technical expert argues precision/recall misses key metric: silent failures
9 Nov 2 5:55 PM · 5d ago · 2 comments · 1 source · development 9 of 12
A developer building a competing QA tool noted that recall and precision don't capture the most important failure mode—silent passes on broken flows—and recommended mutation testing (deliberately planting bugs) as a better benchmark.
“a silent pass is worse than a flaky test, since the flaky one at least makes some noise”
daniielWei-Wei Momentic co-founderJeff Momentic co-founderHenry Momentic engineerGabe (MGabrielDenton) Momentic engineer
The whole story articlespostscomments the bright band is this development · numbered dots are the others · click one to jump
What people said 2 voices · verbatim
-
Not sure precision/recall is the number I'd look at first, because the thing I'd actually want to know is how often the tool says pass on a flow that is already broken, and a silent pass is worse than a flaky test, since the flaky one at least makes some noise.I'm building something in the same space and this bit me quite early, so what I ended up…
-
Hey thanks for sharing your work! Just wondering how many of the 1396 came from Mo, and what happens to a bug that Mo can't reproduce?
All 12 developments of Momentic launches AI QA agent that autonomously tests apps… →
Hacker NewsNewswiresMastodon