Commenter tests other compressors; bzip2 fails to generate coherent text
4 Sep 22 5:51 AM · 1d ago · 3 comments · 1 source · development 4 of 4
A user reports attempting to adapt the gzipt code to use bzip2 and zstd compression algorithms with AI assistance, finding that bzip2 produces only nonsensical character sequences rather than text resembling language.
Nathan Pemberton Researcher/author
The whole story articlespostscomments the bright band is this development · numbered dots are the others · click one to jump
What people said 20 voices · best of 24 · verbatim
-
W
"Can gzip be a language model?" tl;dr: "It turns out, kind of? "
-
I think language itself is compression, so the arxiv paper tracks for me.Viz. if Language is compression (of thought / culture / the tacit je ne sait quois of being-to-being communication etc.), then definitionally, Language Modelling must also be Compression.Except, language is an arbitrarily lossy compressor, who's "compression-prediction…
-
N
Can gzip be a language model?: https:// nathan.rs/posts/gzip-lm/ Discussion: http:// news.ycombinator.com/item?id=4 9797323
-
I was curious to see how this would work with bzip2 and zstd. The source is public at https://github.com/nathanrs/gzipt, and I asked MiMo-V2.6-Flash to fork and modify it. The answer is that bzip2 produces sequences that don't resemble human language: gzipt \ --corpus data/tinyshakespeare.txt \ --prompt $'MENENIUS:\n' \ --length 200 \ ; MENENIUS…
-
Reminded me Google's earlier paper, "Language Modeling Is Compression" https://arxiv.org/abs/2309.10668Learning being a compression is also recently proposed as Gibbs compression proposition.See Gibbs randomness-compression proposition
-
That's a bit of a stretch. The LZ77 algorithm merely finds repeated items within a sliding window and encodes them into Huffman codes. A basic model should at least be capable of classification and prediction, like very early-stage machine models. The former is a fixed, quantifiable algorithm, whereas the latter becomes difficult to quantify and…
-
We used a similar technique for a class project (N decades ago) to test this:https://en.wikipedia.org/wiki/Baconian_theory_of_Shakespeare...By looking at mutual information from different authors on the same topic vs same author on different topics. As I recall, it convincingly disproved the hypothesis.
-
Compressing something and next token prediction are very related. Once you understand the connection, things like ts_zip and hutter prize make a lot more sense.
-
Back in the day - maybe two decades ago - I implemented language detection like this.I seeded gzip compressors’ dictionaries with Wikipedia articles in different languages.I would then try to use said dictionaries on any random text, and the one that was best able to compress it, was the correct language.Absolutely totally not the best approach…
-
There is an older paper that also looks at gzip for ML: https://arxiv.org/abs/2212.09410Video from tsoding where he implements the algorithm:
-
Funny, this kind of knowledge used to be a common theme in ML/DS roles. Funny to see that people are thinking about it now by talking to AI (the article does look very AI written). Funny that people can enter very deep rabbit holes because they didn’t study something and guess they are making progress on something that is commonly known. Dunning…
-
aka Normalized compression distance (NCD). Its close cousin: Normalized Google distance (NGD) is also super interesting!
-
Nitpick: Doing it exactly like this is flawed because you let the compressibility of your references taint the result; what you would prefer is the compressed size of testfile given sports.txt/... as a dictionary without accounting for the compressed size of that, no?Really interesting approach though.
-
zip2zip paper by Geng et al. also exploited LZ/LZW and made the rounds a while back; novel approach that uses zip content as output compression adapter
-
There are some deep connections between machine learning, compression, and cryptography with information theory as a common thread.Also, I’ve never seen “ass.” Used to shorten “aside” — I typically use N.B. but perhaps only for important ones.
-
You might also want the topic files to be compressed against each other to get a baseline matrix and then multiply any results by the inverse, assuming equal priors on the topics.
-
R. Hendricks, D. Chugtai, and J. Dunn, "Lossless compression via optimized middle-out bitstream processing," Pied Piper Inc., Palo Alto, CA, Tech. Rep. 42, Apr. 2014.
-
Would this work with video compression? Video codecs encode a lot of meaning; they use motion vectors to track the movement of objects on screen, for example.
-
My goodness, gzip is intelligent! Enjoy your job while you have it; put on some hip hop music and practice your pivoting.
-
Fun topic but generated article text and then not even actually using gzip? Rubs me in a weird way.
All 4 developments of Researcher demonstrates gzip as functional language model… →
Hacker NewsNewswiresMastodon