PrismML releases Ternary Bonsai 2 27B, 9x compressed model retaining 98% performance
New quantized model runs 27B-class AI on consumer hardware with near-lossless capability preservation.
What to know
- PrismML's Ternary Bonsai 2 27B achieves 9x model compression while retaining 98.2% of benchmark performance—particularly important for coding, tool use, and multimodal tasks where small errors compound.
- The 5.9GB model runs on consumer hardware (RTX 4090, Mac M5 Max) and in browsers via WebGPU, enabling local AI deployment without cloud dependency.
- Community testing reveals genuine capability gains over the previous Bonsai generation but also confirms severe degradation on longer tasks—marketing claims of 'near-lossless' performance face scrutiny.
- Unclear technical advantage: commenters question how this ternary approach meaningfully differs from existing Q2-level quantization of the same Qwen base model.
The dispute Whether 'near-lossless' performance claims hold up: some users report failure modes on longer sequences, conflicting with the headline 98.2% aggregate benchmark retention figure. · positions read across 42 posts and comments
This is a genuine advance for local inference, especially for coding work on consumer hardware.
-
“Running at about 7-8 tok/s (~60 tok/s prefill) on a Mac Mini M2 16GB. So far feels smarter than Bonsai 1 27B, it's slightly larger than the Q1_0 quant. Super exciting stuff :)”
jedbrooke · Hacker News ↗
The model degrades badly on longer tasks and the marketing 'near-lossless' claim is misleading.
-
“Use it for any longer task and they fall apart spectacularly and in interesting ways.”
Aurornis · Hacker News ↗
The technical novelty is unclear; comparisons to standard quantization methods (Q2) and the special sauce are missing from documentation.
-
“I took a quick glance at Bonsai's blog posts, and don't really see them comparing themselves to "typical" quants or explaining what's the special sauce that makes them better?”
adrian17 · Hacker News ↗
PrismML Model developer and publisherAurornis Community tester
.png)
How it unfolded 4 developments, newest first · click a bar or a number to jump articlespostscomments
-
4
Discussion centers on benchmark claims and practical use cases
Commenters questioned the novelty of the compression technique relative to existing quantization methods, noted unclear comparisons to standard quants, and discussed whether the model's real-world value lies primarily in coding tasks versus general-purpose use.
“Models that can run with good speed on affordable consumer hardware for coding only is the dream.”
— blactuary -
Is it just me or are local models improving much more quickly than the frontier models? If so, that would be a very welcome development, with local/private inference becoming available to everyone.
2 more of the top 3 · 38 posts in this stretch
-
I managed to run it with RTX 3070 (8GB VRAM) following the "Quickstart" on HF model card with minor modifications (modify the architecture 86 for your own hardware): git clone https://github.com/PrismML-Eng/llama.cpp && cd llama.cpp cmake -B build -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=86 && cmake --build build -j Then downloaded & verified…
-
The fact that you get 98.2% of the benchmark performance after going from 16 bits per weight to 1.76 bits per weight really shows you how much space is wasted in these models. Wow.
-
-
3
Users identify technical limitations and quantization comparisons
Community members raised questions about model degradation on longer tasks, requested comparisons with standard Q2 quantizations of the same base model, and noted that existing llama.cpp implementations require PrismML's custom fork to support the ternary weights.
“Use it for any longer task and they fall apart spectacularly and in interesting ways.”
— Aurornis -
If you want to try out out the GGUFs from https://huggingface.co/prism-ml/Ternary-Bonsai-2-27B-gguf#th... be aware that you need Prism's llama.cpp fork to get them to work, from https://github.com/PrismML-Eng/llama.cpp/releases/tag/prism-...This should work: cd /tmp # Get the Prism macOS runtime curl -fL…
1 more of the top 2 · 2 posts in this stretch
-
I really wish people would stop saying N times smaller than something when making a comparison; that makes no sense - it's 1/9th (11.11%) the size. You don't get a smaller quantity by multiplying by a number greater than 1.0. You could instead reverse the subjects being compared - "the original model is 9x bigger than this new smaller, efficient…
-
-
2
Community explores model deployment in browsers and on local hardware
Developers report successfully running the model in browsers via WebGPU kernels and on consumer hardware including Mac Mini M2 and NVIDIA GPUs, with observed throughputs of 7-8 tokens/second on M2 and 143 tokens/second on RTX 5090.
“These are small enough that you can run them entirely in the browser…”
— Aurornis -
These are small enough that you can run them entirely in the browser https://huggingface.co/spaces/webml-community/ternary-bonsai...Remember to clear the downloaded weights afterward.Like the last model, it's amazing they work as well as they do. Use it for any longer task and they fall apart spectacularly and in interesting ways.
-
-
1
PrismML releases Ternary Bonsai 2 27B model
PrismML announced Ternary Bonsai 2 27B, a quantized 27B-parameter model using ternary {−1, 0, +1} weights with FP16 group-wise scaling. The model achieves a 5.9GB footprint (1.76 effective bits per weight), more than 9x smaller than full-precision Qwen3.8 27B, while retaining 98.2% of aggregate benchmark performance across reasoning, math, coding, vision, and agentic tool use.
“Against its full-precision counterpart, Ternary Bonsai 2 27B is more than 9x smaller while retaining 98.2% of aggregate benchmark performance.”
— PrismML -
first by Lobsters, 8d ago · also PrismML News, Simon Willison, HN Best, HN Frontpage
1 more headline
- Introducing Bonsai 2 27B: Near-Lossless Compression in a 9x Smaller Footprint PrismML News · 8d ago
-
Love this for the folks with 16gb graphics cards - 3.8 27b has been incredible but not quite runnable on anything less than 32gb - will try loading this up on my 16gb intel b50 and see how it goes - not sure these quants can be accelerated by the XPU cores yet but maybe in time!
-
What people are saying 18 voices from 2 sites · best of 42 · verbatim
- How does Ternary Bonsai 2 actually compare to standard Q2 quantizations of the same Qwen model in real-world coding tasks?
- Why does the model fail catastrophically on longer tasks if it truly retains 98.2% of performance?
- What is the 'special sauce' of ternary quantization that distinguishes it from existing low-bit quantization approaches?
- Sep 19
-
> How does Qwen compare against hosted models like Claude Sonnet? Qwen3.8 27B is _very_ roughly on par with Opus 4.5 or maybe 4.6, I'd say, at least in larger quants like Q6 and above. There are some differences: 1. Opus 4.5 and 4.6 probably had more raw intelligence and world knowledge. 2. Qwen3.8 27B is more persistent and detail oriented. 3…
-
Reference point for Bonsai is Qwen 3.8-27B though. Flash-Next can be reasonably called 4.0-Experimental-Alpha1, and it is in most senses larger than 27B (although it pays some MoE penalty relative to absolute size).
-
As a /r/LocalLlama lurker, I must say that the general consensus of actual users there is Bonsai models are not as good as authors claim them to be, getting lost in long horizon tasks, and their numbers seem cherry-picked. Still, there's no free lunch.
-
I've been comparing local Qwen3.8-family models on my 12GB 3060 lately and I'm so impressed by 3.8-Flash-Next that nothing else really compares yet. I wondered if Bonsai 2 might be similarly good, because it would definitely be faster and I'm using a 3-bit quant of Flash-Next anyway, but I was mostly unimpressed. Context: I've been using the…
- Sep 18
-
I only use local stuff so I cannot compare with hosted precisely, but I can tell about improvement experience. My definition of local is very specific: Ryzen9 5800HX (not very last-gen, I looked what I can buy used as a buildbox) bumped up to 64GiB system RAM, bumped to NVMe storage that can saturate the connector in the miniPC, iGPU use allowed…
-
Compute scale though, not just parameter scale. And with distillation, some of the RL compute for large-scale models gets reused for smaller ones. 30B-class models do seem to improve on popular tasks at the pace comparable to 2T-class models on their larger set of tasks, with a lag. Breadth is obviously another question. At Bonsai size there are…
-
Qwen 3.8 27B is about on par with Sonnet 5 on benchmarks. I haven't used Sonnet 5 in a long time, but I have kicked the tires with 3.8 27b and it's pretty good. But if you're specifically looking for a local model you can run that will get real work done, DeepSeek 4.1 Flash is probably the closest, for a loose definition of "local". I recenely saw…
-
This is an excellent point. As a mathematician, I always imagine DNN layers, and in particular those of LLMs, to be akin to overspecified linear combinations, and it's all about extracting an orthogonal basis of it that minimally describes the space. In high dimensions, things get interesting in this regard: If you pick two random ternary (only…
-
Bonsai 2 27B · Radeon RX 7900 XTX - 89 tokens/sec generation with speculative decoding - 81 tokens/sec at 20k context - 474 tokens/sec ingestion at 20k — about 42 seconds - 10.1 GiB peak VRAM with a 24k context window ROCm 7.2.3 · PQ2_0 · Qwen Q4 MTP, draft length 2---- versus ----Qwen3.8-27B IQ3_S · Radeon RX 7900 XTX - 79 tokens/sec generation…
-
The pareto line is hit much quicker for a given data/task/size/specialization. At the frontier, they're tearing books apart to keep the parameter scaling relevant. Obviously, there's work on training efficiency/distribution/clustering/hardware advances, but these are capex heavy ventures, i.e. "datacenter rollout" scale. For fixed size models…
-
How does Qwen compare against hosted models like Claude Sonnet? My previous experiments with local models have been underwhelming, but I realize things are advancing rapidly. (I realize Sonnet is several ranks below state of the art, but it’s the one I use most frequently.)
-
Let’s see, so if you get the same 1/9th the size compression ratio with GLM-5.3-Flash, then you’d end up with a ~72GB model that’s about as good as GPT-5.6 Sol (high), according to artificialanalysis.aiWhich is within reach of some higher end consumer hardware, especially with layer offloading.You have to wonder what kind of trouble the “labs” are…
-
There's a chap called Bijian Bowen who does very quick agentic coding challenges for new models (very soon after release?) mainly for toy games or websites. He just did one for this model and included a comparison with the base model Qwen 3.8 which shows the "near-lossless" claim should be taken with a grain of salt. It is an interesting model if…
-
Fyi, if you're trying to run this under AMD/HIP:PTQ1_0 has no optimized MMQ-Path in their llama-cpp fork, try running PTQ2_0 (needs a bit more vram, but is about 2x faster on my 6700 XT)
-
Testing on a MBP m4 pro 24gb~100t/s prefill, ~15t/s, dropping to ~10t/s later with 64k context.The issue is I have yet to find a useful agentic local llm that I can run on this machine.Just given a relatively simple task on a swift app, took 25 minutes, brainstorming like crazy but can not decide on what to do. Eventually I killed it. GPT 5.6…
- Sep 17
-
LLM quants seem to eerily converge to modern/not so modern graphics techniques. You wouldn't think it would apply but it's obvious in hindsight. In fact mining graphics ideas is probably a good inspiration for efficient LLM architecture.For example, the Hadamard activation transform used here feels a lot like multiplying Fourier basis ala DFT…
-
Thanks for all of your exploration in public Simon.Commenting because the fix I proposed was merged in roughly 49 commits after the PrismML Fork. The “tensor API is not supported” warning occurred because llama.cpp’s startup probe fails to compile a matmul2d kernel: Metal’s tensor headers require language version 4.0, but ggml-metal-device.m…
-
> Ternary Bonsai 2 27B uses ternary {−1, 0, +1} weights with FP16 group-wise scaling, for 1.76 effective bits per weightIf I recall correctly, a recent post [1] has shown that Q2 quants (with like 2.6 bpw) of the same base Qwen model sit at the edge between "noticeably worse" and Q1's "useless". I took a quick glance at Bonsai's blog posts, and…