conv.

← The whole story
AI
Google releases Android Bench 2.0 with long-horizon AI coding tasks

Android Bench 2.0 reveals significant performance gap on long-horizon tasks

1 Sep 17 12:00 PM · 9d ago · 4 articles · 1 source · development 1 of 2

Results show the highest pass rate for long-horizon tasks is around 28%, dramatically lower than the ~91% pass rate on the original benchmark's simpler tasks. Analysis reveals AI excels at writing new code but struggles with refactoring and migrations due to architectural complexity, and cross-platform app porting remains unsolved by all tested models.

Android Bench 2.0 reveals significant performance gap on long-horizon tasks
android-developers.googleblog.com

“The highest pass rate for LHTs is around 28%, much lower than the ~91% for the original tasks in the benchmark.”

Google Android team

Google Android team Benchmark creator and publisher

The whole story articlesposts the bright band is this development · numbered dots are the others · click one to jump

Peak 4 pieces in 3h at Sep 17, 11 AM; 7 pieces over 9 days (4 articles · 3 posts) Sep 17, 11 AM — 4 pieces · 4 articles — Google News 4Sep 17, 2 PM — 1 piece · 1 post — Mastodon 1Sep 17, 5 PM — quietSep 17, 8 PM — quietSep 17, 11 PM — quietSep 18, 2 AM — quietSep 18, 5 AM — quietSep 18, 8 AM — quietSep 18, 11 AM — quietSep 18, 2 PM — quietSep 18, 5 PM — quietSep 18, 8 PM — quietSep 18, 11 PM — quietSep 19, 2 AM — quietSep 19, 5 AM — quietSep 19, 8 AM — quietSep 19, 11 AM — 1 piece · 1 post — Hacker News 1Sep 19, 2 PM — quietSep 19, 5 PM — quietSep 19, 8 PM — quietSep 19, 11 PM — quietSep 20, 2 AM — quietSep 20, 5 AM — quietSep 20, 8 AM — quietSep 20, 11 AM — quietSep 20, 2 PM — quietSep 20, 5 PM — quietSep 20, 8 PM — quietSep 20, 11 PM — quietSep 21, 2 AM — quietSep 21, 5 AM — quietSep 21, 8 AM — quietSep 21, 11 AM — quietSep 21, 2 PM — quietSep 21, 5 PM — quietSep 21, 8 PM — 1 piece · 1 post — Hacker News 1Sep 21, 11 PM — quietSep 22, 2 AM — quietSep 22, 5 AM — quietSep 22, 8 AM — quietSep 22, 11 AM — quietSep 22, 2 PM — quietSep 22, 5 PM — quietSep 22, 8 PM — quietSep 22, 11 PM — quietSep 23, 2 AM — quietSep 23, 5 AM — quietSep 23, 8 AM — quietSep 23, 11 AM — quietSep 23, 2 PM — quietSep 23, 5 PM — quietSep 23, 8 PM — quietSep 23, 11 PM — quietSep 24, 2 AM — quietSep 24, 5 AM — quietSep 24, 8 AM — quietSep 24, 11 AM — quietSep 24, 2 PM — quietSep 24, 5 PM — quietSep 24, 8 PM — quietSep 24, 11 PM — quietYesterday, 2 AM — quietYesterday, 5 AM — quietYesterday, 8 AM — quietYesterday, 11 AM — quietYesterday, 2 PM — quietYesterday, 5 PM — quietYesterday, 8 PM — quietYesterday, 11 PM — quietToday, 2 AM — quietToday, 5 AM — quietToday, 8 AM — quietToday, 11 AM — quietToday, 2 PM — quiet 1–2
1–2Android Bench 2.0 reveals significant…
Sep 18Sep 19Sep 20Sep 21Sep 22Sep 23Sep 24yesterdaynow · 4:30 PM ET

What was reported 1 claim about this development

  1. 2 outlets Android Bench 2.0: Pushing the frontier with challenging long-horizon tasks

    first by blog.google, 9d ago · also 9to5Google

    1 more headline

All 2 developments of Google releases Android Bench 2.0 with long-horizon AI… →

Google NewsMastodonHacker News