pancomplex releases Jev playing Pokémon Red, sparks debate on model capabilities
A developer built a live AI agent using a small, fast language model to play Pokémon Red, raising questions about whether the project actually demonstrates intelligence or just speed.
What to know
- Jev, a small fast language model, can play Pokémon Red in real time with open-sourced code, but debate centers on whether the project demonstrates real intelligence or just efficient inference.
- The model frequently gets stuck in loops, makes poor strategic decisions, and relies on significant game scaffolding (pathfinding, textual milestones), limiting what the project reveals about reasoning capability.
- Observers suggest Jev works best for simple single-shot decisions; combining it with a smarter reasoning model would be necessary for complex multi-step problem-solving.
- The project sparked broader discussion about whether video game benchmarks are meaningful tests of AI advancement.
The dispute Whether Jev's ability to play Pokémon at all demonstrates meaningful progress in AI reasoning or is mostly a speed/cost benchmark obscuring poor decision-making and reliance on game scaffolding. · positions read across 22 posts and comments
The project reveals meaningful capability worth studying, not just speed metrics.
-
“After watching a few minutes of this it makes me think that maybe we should be a little more interested in how smart it is.”
ralusek · Hacker News ↗
Poor decision-making and repetitive behavior show the model isn't actually ready for practical use despite speed.
-
“Then I saw just how bad the decision are and how it would get stuck in strange loops of going in and out of the same door to no end.”
stusmall · Hacker News ↗
The scaffolding and guidance baked into the harness make it hard to assess what Jev actually understands.
-
“Cool project, comes with a little too much guidance in the harness though IMO (pathfinding, textual milestones etc).”
ac2u · Hacker News ↗
This is a replay of old AI milestones; the real story is that people focus on speed/cost rather than capability.
-
“we already went through the "look this AI can play pokemon terribly" phase like a decade ago.”
binlog · Hacker News ↗
pancomplex Developer, project authorJev Small language model
How it unfolded 6 developments, newest first · click a bar or a number to jump articlespostscomments
-
6
Hacker News moderator notes parallel Pokémon-playing AI discussion
A second related post about a world model playing Pokémon emerged on HN at the same time, suggesting broader interest in AI game-playing benchmarks.
“Koga's lieutenant tells Jev that "Strength isn't the key for Pokemon! It's strategy!" Jev then wipes the whole gym with a single overleveled Charizard. Fitting and appropriate on many levels.”
— bulatb, HN commenter · source -
I've been watching GPT_Plays_Games on twitch and with Astra now it can beat pretty much all of them with very few mistakes, with vision only. One downside is that when pathfinding it can only plan high level paths, wait for result, then try again if it fails. Its a ~10-30 second loop.Has anyone tried a combined LLM + Jev? So the LLM directs the…
2 more of the top 3 · 12 posts in this stretch
-
N
Show HN: Jev Plays Pokémon Red: https:// jev-pokemon.vercel.app/ Discussion: http:// news.ycombinator.com/item?id=4 9845172
-
This is really cool, I love the live streaming too. I did something similar yesterday, built a tiny RTS game and has Jev make decisions every 2 seconds. I am using Luna to tweak the strategy.I wonder for real games, like online Pokemon, multiplayer games, how they will ban this kind of AI "cheating".1.
-
-
5
Commenter questions scaffolding and suggests pure reasoning test
A reviewer noted the project includes significant guidance (pathfinding, textual milestones), making it hard to assess true capability. They proposed combining Jev with standard vLLM and testing it on a model with no prior Pokémon knowledge to see its reasoning process.
“Cool project, comes with a little too much guidance in the harness though IMO (pathfinding, textual milestones etc).”
— ac2u -
Cool project, comes with a little too much guidance in the harness though IMO (pathfinding, textual milestones etc). (The author is very upfront about this in their README though)I think if it was combined with a regular vLLM it could be really interesting, especially watching the reasoning logs.Bonus points if it was one of the latest open models…
2 more of the top 3 · 3 posts in this stretch
-
H
Show HN: Jev Plays Pokémon Red Link: https:// jev-pokemon.vercel.app/ Discussion: https:// news.ycombinator.com/item?id=4 9845172
-
Jev is for single shot classification, not multi-step RL environments with delayed reward and explore/exploit. My guess is it would go through the door with high confidence every time unless you change the input to add the history.
-
-
4
pancomplex confirms limitation and suggests hybrid approach
The author acknowledged the model's reasoning shortcomings, noting that combining Jev with a smarter reasoning model for strategic thinking would be better than using Jev alone, which 'sometimes can be kinda dumb.'
“a mix of models like Jev for simple stuff + a smarter reasoning model for more strategic thinking is the optimal solution. This experiment however is purely Jev. Which sometimes can be kinda dumb.”
— pancomplex -
The exact message I sent my friend this morning:> the most interesting thing about this jev stuff> is that people are seemingly like> completely disinterested in how smart it actually is> I haven't even heard it mentioned a single time how it actually compares to other LLMs coming up with their own classifications. Just: it's fast and cheapAfter…
2 more of the top 3 · 3 posts in this stretch
-
This entire conversation around Jev seems weird to me. Like... we started from neural nets that could do basic decision making and classifications pretty well, then trained larger and larger language models to get to where we are now. Now suddenly everyone is going crazy because someone trained a smaller model that is adequate at making decisions?…
-
Like others have mentioned in this post, I think a mix of models like Jev for simple stuff + a smarter reasoning model for more strategic thinking is the optimal solution. This experiment however is purely Jev. Which sometimes can be kinda dumb.
-
-
3
Observers note poor decision-making and unfulfilled promise
Commenters reported watching the stream revealed significant limitations: the model gets stuck in repetitive loops (going in and out of the same door), makes bad strategic choices, and the project feels more like a technology demo than a working solution. One observer suggested it's heading in the right direction but not ready for practical use.
“For a couple minutes I was in awe of how quick and cheap it was. Then I saw just how bad the decision are and how it would get stuck in strange loops…”
— stusmall -
This is so interesting to watch. For a couple minutes I was in awe of how quick and cheap it was. Then I saw just how bad the decision are and how it would get stuck in strange loops of going in and out of the same door to no end.This seems like a technology heading in the right direction but not quiet there yet. Excited for what they are cooking…
-
-
2
Hacker News commenters question whether project demonstrates real intelligence
Discussion surfaced disagreement about what the project reveals. Some praised the speed and cost efficiency, while others noted the model makes poor decisions, gets stuck in loops, and relies heavily on scaffolding (pathfinding, milestones). Observers questioned whether this represents meaningful progress in AI reasoning.
“the most interesting thing about this jev stuff is that people are seemingly like completely disinterested in how smart it actually is…”
— ralusek -
This is kinda chill to have in the background. I wish there were livestreams showing live reasoning of top models which are currently trying to solve cancer or whatever. Imagine the pogs in chat when it does.
1 more of the top 2 · 2 posts in this stretch
-
Looking at the diagram in the gh repo, it looks like this is entirely jev. Are there any examples of people having a big model like Fable handle high level goals?
-
-
1
pancomplex releases Jev Pokémon Red project with live stream
Developer published a project letting Jev, a small language model, play Pokémon Red. The system makes decisions quickly and cheaply, with the game streamed live including token-level visibility. Code is open-sourced on GitHub.
“Given Jev can make decisions really fast (but not fast enough to play Doom yet sadly), I wanted to try and push it to play a more complex game than Tetris.”
— pancomplex -
2 outlets Show HN: Jev Plays Pokémon Red
first by HN Best, 1d ago · also HN Frontpage
-
That's actually fun to watch. Did you experiment with nicknaming before you turned it off? I'd be a little curious to see how it behaves.
-
What people are saying 9 voices from 1 site · best of 22 · verbatim
- How much does the pathfinding and textual milestone guidance inflate the appearance of capability?
- Would Jev perform differently if tested without access to prior Pokémon game knowledge?
- What would happen if the model received its own previous decisions as context to improve multi-step reasoning?
- Today
-
Where does the "goal" come from? Is it just known based on different positions in the game? Otherwise it seems like it would require text generation.
-
The seminal (lol) Twitch Plays Pokémon was twelve years ago, so just posting this amazing moment of internet history/lore just in case folks don’t know or have forgotten:
-
Jev is dumb a rock, so it's basically just a random selection of a smart model's suggested plans?
- Yesterday
-
i need to come up with something more creative and interesting than video games as benchmarks
-
Koga's lieutenant tells Jev that "Strength isn't the key for Pokemon! It's strategy!"Jev then wipes the whole gym with a single overleveled Charizard.Fitting and appropriate on many levels.
-
He confirms what I long suspected, all you need is to cheese Charizard and flamethrower
-
The choices Jev has here feel very railroady, it seems like something very significantly dumber could beat the game with these options.
-
The decisions are quick, but look good as random w/ tons of back and forth. Are you at least feeding back some of its previous decisions on to state?
-
Dare we have two Pokemon-playing-AI threads at the same time?Teaching a World Model to Play Pokemon -