Users report Mercury 2.5 underperforms for practical tasks despite speed advantage
4 Sep 23 11:31 PM · 2d ago · 4 comments · 1 source · development 4 of 5
Early adopters and commenters shared negative assessments of Mercury 2.5's practical utility. One user stated the model performs on par with 14B models at best, while another noted attempting to use it despite attractive speed and pricing but finding it unsuitable even for basic tasks.
“I have tried using Mercury 2.5 for a lot of my tasks.. but this model just isn't there. It seems to be on par with any 14B model at max.”
freakynitMercury 2.5 Language modelArtificial Analysis Benchmarking organization
The whole story articlespostscomments the bright band is this development · numbered dots are the others · click one to jump
What people said 5 voices · verbatim
-
Chat Jimmy clocks at 17K tokens per sec burning LLM into the Chip - https://chatjimmy.ai/ - Source:
-
I have tried using Mercury 2.5 for a lot of my tasks.. but this model just isn't there. It seems to be on par with any 14B model at max. Even GPT-OSS-20B performs way better than this in my own attempts to use it.I really really wanted to use this because it offers incredible speeds and pricing combinations. But nop.. I still am not using it.. not…
-
At some point the bottleneck becomes tool calling.. and as such, it's preferably if the model is co-hosted (in the same datacenter, at least) with your code repository and all other reference/context it needs (full documentation for most ecosystems, maybe even a copy of common crawl to minimize web fetch usage, etc)
-
I used this a few days ago and thought something must be wrong with how fast it was responding. "Mercury 2.5 is below average in intelligence, but well priced when comparing to other models of similar price." this is so funny. So when you have a stupid model that is fast - what do you use it for?
-
I'm still sad that we haven't seen a new Taalas style chip a la https://chatjimmy.ai/. Smaller models are good enough now to make that insane burst of tokens so useful.
All 5 developments of Mercury 2.5 LLM debuts at 770 tokens per second with… →
Hacker NewsNewswiresMastodon