Creator positions mini-AGI as alternative to frozen commercial models
1 Sep 20 8:42 PM · 5d ago · 1 article · 2 posts · 3 sources · development 1 of 1
volotat articulates the core problem mini-AGI addresses: existing language models are frozen after training, cannot be trained from scratch on personal hardware, and forget previously learned knowledge when fine-tuned on new data. Mini-AGI's architecture enables continuous learning from a stream of data without catastrophic forgetting on modest hardware.
“Every language model you can actually own today is a model somebody else trained and then froze… The moment you try, it forgets what it knew before.”
volotatvolotat Developer, mini-AGI creator
The whole story articlesposts the bright band is this development · numbered dots are the others · click one to jump
What was reported 1 claim about this development
-
first by HN Best, 5d ago · also HN Frontpage
1 more headline
- Mini-AGI – dynamic continual learning model trained from scratch on 8GB VRAM HN Frontpage · 5d ago
What people said 12 voices · best of 13 · verbatim
-
I am not sure about eventually, but it learns on the steady paste so far. The main thing to keep in mind is that it learns on the single STREAM of data. Not randomized batched samples. Try to do it with any other model and you will see nothing but complete garbage in the predictions, exactly because of catastrophic forgetting.And here are the…
-
I built a smaller scale replication. Most of your claims prove out but the lever is essentially Biderman et als "learn less forget less". My first pass actually made this mistake initially and the findings didn't replicate. After controlling for step density I replicated your findings. Here is a write…
-
I have not looked carefully but it seems like this is over-promising on avoiding catastrophic forgetting.The "trunk learning rate" is set at 0.1x the learning rate for the experts, so learning on different subjects disproportionately happens in the experts, and the trunk portion is comparatively more stable. But the population of experts can grow…
-
Mini-AGI is a totally inappropriate name - it seems what this project is shooting for, but not delivering on, is being a language model with "continual learning".Where it seems to fail, by design, on this goal is in delivering continual learning that is more than just "memorization with LRU catastrophic forgetting".That said, props to the author…
-
I'd love to explore a question. If I have a standard elementary school math textbook, along with all the results of every correct and incorrect answer my child got on the textbook exercises and class tests, is it possible to train a model that can help me figure out what concepts my child is struggling with and have that model provide help?
-
If you actually scroll through the transcript he links to, you will see that something that looks like it could be training is happening, but no coherent responses are coming out at any point. At least not that I saw skimming through.That might explain why there are no benchmarks of any kind.
-
Seems interesting, I've been messing with a lot of continuous learning approaches lately and it's cool to see something that's built from the ground up for avoiding catastrophic forgetting. Worth a clone for sure
-
Interesting. I wonder how much could be gained from using tokenization, which makes the model work at a semantic level rather than a syntactic level? I think it’s a force multiplier, but idk if it works here.
-
Nobody will throw rocks, I think most people are curious/suspicious about the big players and wants more hands-on since we suspect that this all will come down in cost soon enough.
-
Is this architecture actually able to generalize or is it mostly based on memorization? Have you tried some basic tasks that require generalization? e.g. number addition etc?
-
I read your github page and there's mention of continual learning but no argument how catastrophic forgetting is avoided. I think you should explain that better.
-
This is the first thing I see in my life that really looks like proto-AGI, it deserves its name.
All 1 developments of Developer releases mini-AGI, a continual learning model… →
Hacker NewsMastodonNewswiresLobsters