Researcher demonstrates gzip as functional language model using compression
A programmer shows that the standard compression tool gzip can generate text by treating compression as prediction, without neural networks or learned parameters.
What to know
- Nathan Pemberton published a technical demonstration that gzip, the standard compression utility, can generate text by exploiting the mathematical equivalence between compression and prediction, without using neural networks or learned parameters.
- The approach uses beam search to score candidate text continuations by their compressed length when added to a primed context, with better compression indicating higher predicted probability.
- Generated output shows the model captures patterns from training data but lacks the coherence of neural language models; commenters note technical limitations including incomplete search of the possibility space and that other compressors like bzip2 fail to produce readable text.
The dispute Whether the beam search approach explores sufficient possibility space to demonstrate gzip's true capacity as a language model, or whether results represent only a lower bound on performance. · positions read across 31 posts and comments
The compression-prediction equivalence is intellectually elegant but practically limited compared to neural models.
-
“This is fun, but historically people have gone a bit overboard with saying that models like this, or n-gram language models, are anywhere close to large neural network models.”
montebicyclelo · Hacker News ↗
The compression principle opens interesting research directions beyond text generation, including classification and code generation.
-
“The prompt (or really, the tokens) would then be the compressed version of the codebase or the text.”
berkes · Hacker News ↗
“every prediction model is inherently a compressor, and all compression algorithms are prediction models.”
Language Modeling is Compression paper · Blog article
Nathan Pemberton Researcher/author
How it unfolded 4 developments, newest first · click a bar or a number to jump articlespostscomments
-
background
Nathan Pemberton publishes gzip language model experiment — A technical article demonstrates that gzip can perform language modeling by exploiting the compression-prediction equivalence: sequences that compress well are those the model predicts as likely. The author primes gzip with a Shakespeare corpus, uses beam search to score candidate continuations by their compressed length, and generates text without neural networks or learned weights.
-
4
Commenter tests other compressors; bzip2 fails to generate coherent text
A user reports attempting to adapt the gzipt code to use bzip2 and zstd compression algorithms with AI assistance, finding that bzip2 produces only nonsensical character sequences rather than text resembling language.
-
W
"Can gzip be a language model?" tl;dr: "It turns out, kind of? "
2 more of the top 3 · 24 posts in this stretch
-
I think language itself is compression, so the arxiv paper tracks for me.Viz. if Language is compression (of thought / culture / the tacit je ne sait quois of being-to-being communication etc.), then definitionally, Language Modelling must also be Compression.Except, language is an arbitrarily lossy compressor, who's "compression-prediction…
-
N
Can gzip be a language model?: https:// nathan.rs/posts/gzip-lm/ Discussion: http:// news.ycombinator.com/item?id=4 9797323
-
-
3
Commenter cites prior work on compression-based classification
A commentator notes that gzip-based text classification (comparing compressed file sizes across topic domains) was explored by researchers at Waikato University and references the Hutter Prize competition focused on compression.
-
Yes: you can classify a test file by topic with gzip as follows: gzip -9 sports.txt testfile.txt gzip -9 politics.txt testfile.txt gzip -9 business.txt testfile.txt (ass. sports.txt politics.txt and business.txt are text docs pertaining from the sports, politics and business domains, respectively, and have equal size)The test file belongs to the…
-
-
2
Commenter explores compression-as-prompt concept in reverse
A Hacker News commenter extends the logic, proposing that reproducible language models could treat generated code or text as compressed versions of themselves, where the prompt (or tokens) becomes the compression.
“The prompt (or really, the tokens) would then be the compressed version of the codebase or the text.”
— berkes -
2 outlets Can gzip be a language model?
first by HN Best, 2d ago · also HN Frontpage
-
I've been pondering on something related: can an LLM be a chat?Some models are reproducible, in that the same prompt will generate the same output. Say that we could wire up such a model to generate some code.In that case, we could create a prompt that generates, say, an entire codebase, or a large piece of text. The prompt (or really, the tokens)…
2 more of the top 3 · 3 posts in this stretch
-
3blue1brown did a series on this topic: https://www.youtube.com/watch?v=l6DKRf-fAAM https://www.youtube.com/watch?v=GlYgs6v2YfU (i think one more is yet to release)
-
This is fun, but historically people have gone a bit overboard with saying that models like this, or n-gram language models, are anywhere close to large neural network models. There is certainly a connection though.
-
-
1
Commenter questions search completeness of beam approach
A Hacker News user raises a technical concern about whether the beam search explores enough of the possible sequence space to find truly optimal compressions, noting the search may only provide a lower bound on gzip's effectiveness as a plausibility tester.
“It turns out, kind of? It's not exactly coherent text, but it clearly knows something about the text. Much more than I expected gzip to know.”
— Nathan Pemberton · source -
give it a normal text prompt, and it continues that prompt by searching for the byte sequences that compress best. One moment, how are we supposed to know how well that search was done? There is no way to search a meaningful part of the search space.So the result only gives us some lower bound of how well gzip works as a "plausibility tester" of a…
2 more of the top 3 · 3 posts in this stretch
-
I'm more interested in the converse question: how well does an LLM perform as a compressor, compared to gzip (ignoring its insanely lower speed)?
-
This tracks perfectly with Winrar being more profitable than OpenAI... coincidence? I think not!
-
What people are saying 15 voices from 1 site · best of 31 · verbatim
- How does the performance of gzip compare to other compression algorithms like bzip2 or zstd for language modeling?
- Could this approach work with video compression algorithms that encode semantic meaning like motion vectors?
- Yesterday
-
Funny, this kind of knowledge used to be a common theme in ML/DS roles. Funny to see that people are thinking about it now by talking to AI (the article does look very AI written). Funny that people can enter very deep rabbit holes because they didn’t study something and guess they are making progress on something that is commonly known. Dunning…
- Sep 22
-
You might also want the topic files to be compressed against each other to get a baseline matrix and then multiply any results by the inverse, assuming equal priors on the topics.
-
Compressing something and next token prediction are very related. Once you understand the connection, things like ts_zip and hutter prize make a lot more sense.
-
There is an older paper that also looks at gzip for ML: https://arxiv.org/abs/2212.09410Video from tsoding where he implements the algorithm:
-
Reminded me Google's earlier paper, "Language Modeling Is Compression" https://arxiv.org/abs/2309.10668Learning being a compression is also recently proposed as Gibbs compression proposition.See Gibbs randomness-compression proposition
-
zip2zip paper by Geng et al. also exploited LZ/LZW and made the rounds a while back; novel approach that uses zip content as output compression adapter
-
That's a bit of a stretch. The LZ77 algorithm merely finds repeated items within a sliding window and encodes them into Huffman codes. A basic model should at least be capable of classification and prediction, like very early-stage machine models. The former is a fixed, quantifiable algorithm, whereas the latter becomes difficult to quantify and…
-
We used a similar technique for a class project (N decades ago) to test this:https://en.wikipedia.org/wiki/Baconian_theory_of_Shakespeare...By looking at mutual information from different authors on the same topic vs same author on different topics. As I recall, it convincingly disproved the hypothesis.
-
Nitpick: Doing it exactly like this is flawed because you let the compressibility of your references taint the result; what you would prefer is the compressed size of testfile given sports.txt/... as a dictionary without accounting for the compressed size of that, no?Really interesting approach though.
-
aka Normalized compression distance (NCD). Its close cousin: Normalized Google distance (NGD) is also super interesting!
-
Back in the day - maybe two decades ago - I implemented language detection like this.I seeded gzip compressors’ dictionaries with Wikipedia articles in different languages.I would then try to use said dictionaries on any random text, and the one that was best able to compress it, was the correct language.Absolutely totally not the best approach…
-
There are some deep connections between machine learning, compression, and cryptography with information theory as a common thread.Also, I’ve never seen “ass.” Used to shorten “aside” — I typically use N.B. but perhaps only for important ones.
-
Would this work with video compression? Video codecs encode a lot of meaning; they use motion vectors to track the movement of objects on screen, for example.
-
I was curious to see how this would work with bzip2 and zstd. The source is public at https://github.com/nathanrs/gzipt, and I asked MiMo-V2.6-Flash to fork and modify it. The answer is that bzip2 produces sequences that don't resemble human language: gzipt \ --corpus data/tinyshakespeare.txt \ --prompt $'MENENIUS:\n' \ --length 200 \ ; MENENIUS…
-
R. Hendricks, D. Chugtai, and J. Dunn, "Lossless compression via optimized middle-out bitstream processing," Pied Piper Inc., Palo Alto, CA, Tech. Rep. 42, Apr. 2014.