Users identify technical limitations and quantization comparisons
3Sep 17 6:13 PM · 8d ago · 2 comments · 1 source · development 3 of 4
Community members raised questions about model degradation on longer tasks, requested comparisons with standard Q2 quantizations of the same base model, and noted that existing llama.cpp implementations require PrismML's custom fork to support the ternary weights.
prismml.com
“Use it for any longer task and they fall apart spectacularly and in interesting ways.”
Aurornis
PrismMLModel developer and publisherAurornisCommunity tester
The whole story articlespostscommentsthe bright band is this development · numbered dots are the others · click one to jump
If you want to try out out the GGUFs from https://huggingface.co/prism-ml/Ternary-Bonsai-2-27B-gguf#th... be aware that you need Prism's llama.cpp fork to get them to work, from https://github.com/PrismML-Eng/llama.cpp/releases/tag/prism-...This should work: cd /tmp # Get the Prism macOS runtime curl -fL…
I really wish people would stop saying N times smaller than something when making a comparison; that makes no sense - it's 1/9th (11.11%) the size. You don't get a smaller quantity by multiplying by a number greater than 1.0. You could instead reverse the subjects being compared - "the original model is 9x bigger than this new smaller, efficient…
.png)