Commenter requests benchmarks with codebase access, citing agent efficiency gains
12 Nov 3 12:57 AM · 5d ago · 3 comments · 1 source · development 12 of 12
A user with prior experience at another company suggested Momentic benchmark scenarios where agents have read access to source code, noting this setup produced faster debugging and code correlation without benchmarking overhead.
Wei-Wei Momentic co-founderJeff Momentic co-founderHenry Momentic engineerGabe (MGabrielDenton) Momentic engineer
The whole story articlespostscomments the bright band is this development · numbered dots are the others · click one to jump
What people said 21 voices · verbatim
-
Good to see Momentic's demo. We at Vostride are building something similar, but we feel the verification layer should be open for verification, too.We built agent-qa (https://github.com/vostride/agent-qa), an open-source self-improving QA agent. The harness becomes outdated every 3 to 6 months. Unfortunately, the big players are unable to maneuver…
-
I drive a real browser against websites as my job, so I've been on the buyer side of this. What decides whether I keep an agent in the loop isn't how many bugs it finds, it's whether I trust the red result without redoing the check myself.Two things bit me early. Agent written assertions drift to whatever is stable to assert: page copy, a fixed…
-
The haukebri point deserves more weight than it's getting: for a professional tester, recall is not the metric — trust is. One noisy finding per run and the report gets ignored, which means every real finding in it dies too. A tool that finds 3x more bugs at the cost of reader trust has negative value, because the reader's attention is the scarce…
-
The benchmark is the part I would push on, and I say that from assessment work. A test built by the people selling the tool, with bugs labelled by the same team, will usually flatter the tool, not because anyone cheated, but because the cases reflect what the builders already think counts as a bug.Planted and labelled bugs are also cleaner than…
-
One thing I've been thinking about as coding agents take on more implementation work is the difference between finding bugs and verifying that the right thing was built.An app can work perfectly from a functional standpoint while still deviating from the original requirements or architectural decisions. We've seen agents quietly simplify…
-
Cool benchmark setup, but I’d sanity-check the evaluation a bit more before trusting the $/bug and precision/recall comparisons.In particular: how is “bug” defined and de-duplicated across agents/runs? Are the 1,396 human-labeled bugs the same ones used for judging (i.e., any leakage via traces/dataset construction)? And since precision is partly…
-
Funny seeing Saleor in your benchmarks. I was digging through that codebase recently for something else and ran into a nasty bug along those same lines.They have a price recalculation task meant to only run on dirty listings, except someone forgot to assign the filtered queryset back to the variable. So it just recomputed every single listing on…
-
The recording and steps part is what caught my eye. A bug report with just a description usually turns into a bunch of back and forth before anyone can reproduce it, so having the actual steps there sounds like it would save a lot of that. I am curious how that holds up on the borderline ones though. When a report is not clearly a bug, does that…
-
I'm curious to see benchmarks when you have access to source code. While traditional QA does not get access to the codebase, with LLMs being primary authors of the code, there are no real practical barriers to giving code access to the testing agents.At my last job, we did this - gave access to codebase to codex/CC, allowed it to drive the app via…
-
The point about LLMs reinforcing their own assumptions matches something I keep running into: when the same model writes the code and the tests, the tests tend to encode the implementation, so they pass by construction. What has helped more than a better model is separating the roles, e.g. tests written from the spec or ticket before the code…
-
Hey there! Thanks for watching our demo! Would love to run Mo on some other, public, benchmarks and share results. I'm curious how you're evaluating your framework's ability to find bugs vs just execute flows in an app? I also took a look at your site and it seems like you're product is more akin to Momentic regression tests, as opposed to Mo, as…
-
One metric I'd be interested in is defects discovered per hour of test maintenance. Traditional end-to-end tests can become expensive when interfaces change frequently.For AI-driven testing, I'd want to compare:1. Real defects discovered before release 2. False positives requiring investigation 3. Maintenance effort after UI changes 4. Bugs that…
-
That's where UI-only testing gets tricky. A flow can look fine even when the request returns 200 with an error in the body, or a background job gets accepted but never actually completes.Seeing network requests helps, but for async flows you'd also need some way to verify the final application state. Does Mo have access to that kind of…
-
When an agent finds a bug in a mobile app, how much does it see below the UI? A lot of the failures I run into aren't visible on screen: a request that returns a 200 with an error body, a retry loop, a call that quietly fails. Do the agents get the network traffic for a failed flow, or only screenshots and the view hierarchy?
-
How are you handling the failure modes when the agent gets stuck in a retry loop? For example, if an element doesn't load and the agent just keeps trying to click it over and over. Are you using a hard max-steps limit, or do you have a smarter way to detect when the agent's state has stopped changing and kill the run?
-
For this I created a concept called Guide coding and created a vs code extension that still keeps you in then loop. Have a look
-
Nice! Let us know if we can help. Just to save you some time, I will say that in order to speed up data labelling, we pinned a commit a little bit behind latest main, since some of the bugs would already be documented in fixes or GitHub issues. So some of the bugs may have already been fixed!
-
Most of the thread is about web, so a mobile question: do the agents run on real devices, or only simulators and emulators? A lot of the bugs I chase only show up on specific Android OEM builds or mid-range phones.
-
Thanks for putting Saleor in. We're going through the voucher ones now, starting with the editor crashes, will fix what's real, and can report back.
-
So you compare a naked “find a bug” with real harness build of tears , know how to test , lol . Is that an assumption that people stop building harness ?
-
When a user marks a finding as "not a bug," does Mo learn from that for the next run or does it report the same one again?
All 12 developments of Momentic launches AI QA agent that autonomously tests apps… →
Hacker NewsNewswiresMastodon