Discussion centers on benchmark claims and practical use cases
4 Sep 17 6:42 PM · 9d ago · 1 post · 10 comments · 2 sources · development 4 of 4
Commenters questioned the novelty of the compression technique relative to existing quantization methods, noted unclear comparisons to standard quants, and discussed whether the model's real-world value lies primarily in coding tasks versus general-purpose use.
.png)
“Models that can run with good speed on affordable consumer hardware for coding only is the dream.”
blactuaryPrismML Model developer and publisherAurornis Community tester
The whole story articlespostscomments the bright band is this development · numbered dots are the others · click one to jump
Reported in the same hours no headline names this development itself — these 1 claim were published in its stretch
-
first by Lobsters, 9d ago · also PrismML News, Simon Willison, HN Best, HN Frontpage
1 more headline
- Introducing Bonsai 2 27B: Near-Lossless Compression in a 9x Smaller Footprint PrismML News · 9d ago
What people said 24 voices · best of 38 · verbatim
-
Is it just me or are local models improving much more quickly than the frontier models? If so, that would be a very welcome development, with local/private inference becoming available to everyone.
-
I managed to run it with RTX 3070 (8GB VRAM) following the "Quickstart" on HF model card with minor modifications (modify the architecture 86 for your own hardware): git clone https://github.com/PrismML-Eng/llama.cpp && cd llama.cpp cmake -B build -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=86 && cmake --build build -j Then downloaded & verified…
-
The fact that you get 98.2% of the benchmark performance after going from 16 bits per weight to 1.76 bits per weight really shows you how much space is wasted in these models. Wow.
-
LLM quants seem to eerily converge to modern/not so modern graphics techniques. You wouldn't think it would apply but it's obvious in hindsight. In fact mining graphics ideas is probably a good inspiration for efficient LLM architecture.For example, the Hadamard activation transform used here feels a lot like multiplying Fourier basis ala DFT…
-
This is an excellent point. As a mathematician, I always imagine DNN layers, and in particular those of LLMs, to be akin to overspecified linear combinations, and it's all about extracting an orthogonal basis of it that minimally describes the space. In high dimensions, things get interesting in this regard: If you pick two random ternary (only…
-
Thanks for all of your exploration in public Simon.Commenting because the fix I proposed was merged in roughly 49 commits after the PrismML Fork. The “tensor API is not supported” warning occurred because llama.cpp’s startup probe fails to compile a matmul2d kernel: Metal’s tensor headers require language version 4.0, but ggml-metal-device.m…
-
Qwen 3.8 27B is about on par with Sonnet 5 on benchmarks. I haven't used Sonnet 5 in a long time, but I have kicked the tires with 3.8 27b and it's pretty good. But if you're specifically looking for a local model you can run that will get real work done, DeepSeek 4.1 Flash is probably the closest, for a loose definition of "local". I recenely saw…
-
Bonsai 2 27B · Radeon RX 7900 XTX - 89 tokens/sec generation with speculative decoding - 81 tokens/sec at 20k context - 474 tokens/sec ingestion at 20k — about 42 seconds - 10.1 GiB peak VRAM with a 24k context window ROCm 7.2.3 · PQ2_0 · Qwen Q4 MTP, draft length 2---- versus ----Qwen3.8-27B IQ3_S · Radeon RX 7900 XTX - 79 tokens/sec generation…
-
Compute scale though, not just parameter scale. And with distillation, some of the RL compute for large-scale models gets reused for smaller ones. 30B-class models do seem to improve on popular tasks at the pace comparable to 2T-class models on their larger set of tasks, with a lag. Breadth is obviously another question. At Bonsai size there are…
-
> Ternary Bonsai 2 27B uses ternary {−1, 0, +1} weights with FP16 group-wise scaling, for 1.76 effective bits per weightIf I recall correctly, a recent post [1] has shown that Q2 quants (with like 2.6 bpw) of the same base Qwen model sit at the edge between "noticeably worse" and Q1's "useless". I took a quick glance at Bonsai's blog posts, and…
-
How does Qwen compare against hosted models like Claude Sonnet? My previous experiments with local models have been underwhelming, but I realize things are advancing rapidly. (I realize Sonnet is several ranks below state of the art, but it’s the one I use most frequently.)
-
There's a chap called Bijian Bowen who does very quick agentic coding challenges for new models (very soon after release?) mainly for toy games or websites. He just did one for this model and included a comparison with the base model Qwen 3.8 which shows the "near-lossless" claim should be taken with a grain of salt. It is an interesting model if…
-
I've been comparing local Qwen3.8-family models on my 12GB 3060 lately and I'm so impressed by 3.8-Flash-Next that nothing else really compares yet. I wondered if Bonsai 2 might be similarly good, because it would definitely be faster and I'm using a 3-bit quant of Flash-Next anyway, but I was mostly unimpressed. Context: I've been using the…
-
Testing on a MBP m4 pro 24gb~100t/s prefill, ~15t/s, dropping to ~10t/s later with 64k context.The issue is I have yet to find a useful agentic local llm that I can run on this machine.Just given a relatively simple task on a swift app, took 25 minutes, brainstorming like crazy but can not decide on what to do. Eventually I killed it. GPT 5.6…
-
> How does Qwen compare against hosted models like Claude Sonnet? Qwen3.8 27B is _very_ roughly on par with Opus 4.5 or maybe 4.6, I'd say, at least in larger quants like Q6 and above. There are some differences: 1. Opus 4.5 and 4.6 probably had more raw intelligence and world knowledge. 2. Qwen3.8 27B is more persistent and detail oriented. 3…
-
Let’s see, so if you get the same 1/9th the size compression ratio with GLM-5.3-Flash, then you’d end up with a ~72GB model that’s about as good as GPT-5.6 Sol (high), according to artificialanalysis.aiWhich is within reach of some higher end consumer hardware, especially with layer offloading.You have to wonder what kind of trouble the “labs” are…
-
I only use local stuff so I cannot compare with hosted precisely, but I can tell about improvement experience. My definition of local is very specific: Ryzen9 5800HX (not very last-gen, I looked what I can buy used as a buildbox) bumped up to 64GiB system RAM, bumped to NVMe storage that can saturate the connector in the miniPC, iGPU use allowed…
-
Fyi, if you're trying to run this under AMD/HIP:PTQ1_0 has no optimized MMQ-Path in their llama-cpp fork, try running PTQ2_0 (needs a bit more vram, but is about 2x faster on my 6700 XT)
-
The pareto line is hit much quicker for a given data/task/size/specialization. At the frontier, they're tearing books apart to keep the parameter scaling relevant. Obviously, there's work on training efficiency/distribution/clustering/hardware advances, but these are capex heavy ventures, i.e. "datacenter rollout" scale. For fixed size models…
-
If you want to download the gguf to your regular huggingface cache directory instead of to /tmp, you can download the model and run the server in one step: export HF_TOKEN=xxx # optional, speeds up the download ./llama-prism-b10685-7dffb15/llama serve \ -hf prism-ml/Ternary-Bonsai-2-27B-gguf:PTQ1_0 \ --port 8331 -ngl 99 -fa on -c 32768
-
As a /r/LocalLlama lurker, I must say that the general consensus of actual users there is Bonsai models are not as good as authors claim them to be, getting lost in long horizon tasks, and their numbers seem cherry-picked. Still, there's no free lunch.
-
I used that to Generate an SVG of a pelican riding a bicycle:https://tools.simonwillison.net/markdown-svg-renderer?url=ht...It took 18 minutes 20 seconds. Pretty decent for a 5.5GB model file.
-
Reference point for Bonsai is Qwen 3.8-27B though. Flash-Next can be reasonably called 4.0-Experimental-Alpha1, and it is in most senses larger than 27B (although it pays some MoE penalty relative to absolute size).
-
Thanks for the headstart, I saw hf also has PQ2_0 and able to finetune the command and in a MBA M4 24GB averages around 10t/s with the command../llama-prism-b10685-7dffb15/llama-server \ -m Ternary-Bonsai-2-27B-PQ2_0.gguf \ --port 8331 -ngl 99 -fa on -c 65536 --jinja \ --temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.0
All 4 developments of PrismML releases Ternary Bonsai 2 27B, 9x compressed model… →
NewswiresHacker NewsMastodonLobsters