Microgpt in pure C achieves 10M tokens/sec on Apple M5
A dependency-free GPT implementation in C demonstrates character-level transformer training and inference at 10M tokens per second on Apple silicon.
What to know
- Microgpt is a complete GPT implementation (training + inference + sampling) in pure C with zero external dependencies, fitting in a single file.
- Achieves 10.1M tokens per second on Apple M5 Pro with NEON acceleration; 6.9M tok/sec on AMD Ryzen 5 with AVX2.
- A 4,192-parameter model trained on 32k names generalizes better than memorizes, beating larger trigram models on unseen data.
“Training and inference use separate forward passes. gpt_forward stores activations for backprop; gpt_forward_infer is a specialised single-token path whose logits match it to within fp32 rounding.”
Vixhal Baraiya · GitHub repository
Vixhal Baraiya Developer
The record 1 articles and posts · last 30 days
What people are saying 11 voices from 1 site · verbatim
- Aug 20
-
In the README, the C version gets "452× faster than Python", the 2500x one is using OpenBLAS which uses lower level things than what's in C[1].[1] see eg
-
Of course, an interpreted language implementation, versus a compiled language designed to port UNIX.Even if counting that it is basically calling into C or C++ libraries, there is lots of boilerplate that the CPU has to consume, that just won't happen on the pure C implementation.
-
I often wonder if there aren't some low hanging fruit in the nature of the python ecosystem surrounding AI/ML these days. I'm sure there are a lot of eyeballs looking at the problem, but every time I fire up a chat and see the python loadout, it "feels" like there might be work yet still to be done to make things as fast as possible.And then…
- Aug 19
-
This is pretty cool. Implemented something similar myself (a really small language model with ~10M params) just to teach myself the ML behind the LLMs. Did not implement it in C obviously, just use PyTorch, but it's interesting to go through the c file and see how he has implemented stuff I took for granted in Python in C.Anyway, I just tested…
-
It could probably go quite a bit faster using Arm SME. The entire network could fit into the ZA register.
-
Check out this port of microgpt to C, posted 5 months ago. It got a 2500x speedup over the python version.
-
> The most atomic way to train and inference a GPT in pure, dependency-free C.What sense of the word "atomic" is meant here?
-
Honestly not sure this is impressive. I ported microgpt to zig as a learning exercise, then moved scalar engines to NEON/metal just to see what happened. Besides metal being slower (I probably did something wrong, but it could be due to the fixed costs of memory transfer into the GPU not being worth it due to the small model).Anyways, it was also…
-
This is not an LLM obviously , it's just for generating random names. But interesting to think of the possibilities of truly tiny language models if there were connected together.
-
Model is 4K parameters - I don't know enough about that size of model to know if this impressive or not.
-
And the 5 years old AMD Ryzen 5 5600H is doing 7M?Am I reading this right? Then I need to try this on Strix Halo