Search engine developer optimizes index compression with novel byte-plane technique
6 Sep 21 9:08 AM · 3d ago · 31 comments · 1 source · development 6 of 6
A developer shared technical details on value block compression work for a search index, using zigzag delta encoding and byte-plane rearrangement with LZ4 compression to nearly double compression rates and save approximately 4 TB of disk space.
“Fiddling with value block compression in the marginalia index. Very cursed stuff.”
marginaliamarginalia Search engine developerdallen Software engineer and researchernemin Software developermotet-a Software developer
The whole story postscomments the bright band is this development · numbered dots are the others · click one to jump
What people said 24 voices · best of 34 · verbatim
-
Fiddling with value block compression in the marginalia index. Very cursed stuff. A bit simplified, but I'm basically zigzag delta encoding[1] the data and then reshuffling it in byte planes, and then compressing them with LZ4. So if I have for the sake of simplicity 64 longs that have been zigzag delta encoded, the data is rearranged into ```…
-
Revving up after being ill for five days. There is a bit of a backlog at work, but it's not too bad, and a lot of it is fairly mundane and perfect for ramping up. At home, I have completed shutting down all I had on Digital Ocean and migrating the stuff to less objectionable places. I made a boring word game and am considering ways to spice it up.
-
Good luck on your HAM radio exam! I am not supposed to talk about this, for Elmers will come from the hilltops screaming, but by a logistical accident I managed to pass the General license exam before I had ever taken the Technician license exam. This was all at a HAM radio conference, and upon learning that I didn't have a Technician license exam…
-
Celebrating the equinox. Human societies thrive on celebration and the way the Solar system[1] is aligned with our moon gives us several natural times to do so, and the summer/fall equinox is a good one because there's lots of great stuff to eat. Highly recommended. 1: I learned last week that Solar system is the technically correct…
-
It's about 1.1 TB per index partition with 8 regular partitions, and two smaller ones. Total available storage is ballpark of 90 TB, but it's also got to keep the crawl data and a bunch of other stuff, and have enough storage to allow constructing the index. At least part of the motivation for the change is to reduce the amount of additional space…
-
Work: in the throes of AI knowledge management/context management. Interesting stuff - we started having discussions Descarte-style of "how do I know what I know". Home: building a toy app for workout recovery.
-
I've really enjoyed getting more in-depth into poker theory and practice so have been slowly building a poker solver. Along the way I'm writing all the “standard library” bits that I think are missing in C++, or could be simpler and better. I'm currently on a yak shave of writing a small build system from scratch for my projects too. My goal is to…
-
On my way to finish the Stuxnet book this week. Completely blown away by so many things and I have so many questions. First of all, how did these (Symantec, Kaspersky) private companies get the sinkholes/honeypots?? I've been reading the original threads on the orange site from when this unfolding and its really fascinating. I would love to put…
-
I'm pre-training a small language model to become more competent in my field. Once I'm done with that, I'll be moving on to post training. If all goes well, I'll have a post-trained model ready by the end of the month. I might be making things harder for myself by choosing a non-standard ML framework (I'm using `burn` in Rust). I did so to force…
-
I'm visiting my mom for her birthday, so I'm planning to finally migrate my family mail server to an OpenBSD VPS running [OpenSMTPD](https://www.opensmtpd.org/) (I was using [Mail-in-a-Box](https://mailinabox.email/) before). Then I can personally update her mail client and make sure everything works for her, without having to troubleshoot from 5k…
-
I am winding down after months of feeling immense pressure to be productive at all times. I'll be playing WoW Forever beta when not working or being with the family. We (me and the wife) are also looking to enroll into a math undergrad at a uni in barcelona. 1-2 classes per semester
-
Ah, nice! Migrating away from DO is also on my list, this week I'm moving my family mail server to Hetzner — then I'll have my personal mail server and personal website, but those are less risky. I was also sick with covid, and I'm finally feeling better.
-
Reverse engineering my Sony A7IV camera, and trying to coordinate with someone with the A7V model to extract its firmware encryption keys, too.
-
You run into someone you know once in a while!
-
I had done a tilemap bases project a couple years ago and used bevy_ecs_tilemap directly and it was quite annoying. For this much smaller scoped project I wanted to remove complexity, and bevy_ecs_LDTK takes care of all of the LDTK specific quirks of working with their project files. Less work for me to get something user friendly over to my…
-
Work: We are replacing our [config compiler](https://www.brautaset.org/posts/alien-artefacts.html), as the old architecture is no longer viable with the size of configs we see today. We’ve been running both the legacy compiler and the new validator in parallel, comparing the results and applying fixes, aiming for parity. We started weeks ago at…
-
Motivated by a few customer requests, I've been playing around with an idea to provide a good, privacy-focused and quick "autocomplete" and "did you mean X?" set of features in Uruky. I'm currently inclined to build a corpus of popular Wikipedia titles and some other open sources (only up to ~5M phrases) with weights and match against that. That…
-
> I'm currently inclined to build a corpus of popular Wikipedia titles and some other open sources (only up to ~5M phrases) with weights and match against that. That way there's no privacy invasion, no logging necessary, etc., and it could be fast without hitting search providers a lot more frequently (thus making it prohibitively expensive). Yeah…
-
**Work**: Back to it after a week in Iceland for an off-site blitz. Body is mostly recovered from jet lag now, and kinda unpacking that week and getting back into the swing of managing the rest of my team **Home**: My mother shared a link to a random webapp that lets her play Skip-Bo, one of her favorite games, online with others (standard…
-
**For Work** I was supposed to release an update to [Thetix](https://thetix.ai) but it needed more time to cook. Planning on releasing today or tomorrow. **For Fun** Figure out how to include Jev like models into [Abject](https://abject.world). I was also going to release a new coding agent called slangmaker last week but it needs some more time…
-
> Personal: Rounding out my super rough proof of concept for a 2D platformer in Bevy using LDTK for level editing. Trying to build enough dev tools to hand it over to my non-programmer friend who’s working with me. Can you share why you chose LDTK instead of Tiled? I'm planning to start a new game with Bevy this fall, if I can, and I was leaning…
-
Work: managing my team through a few critical deployments; Doing some consulting for a sister business and planning a trip to Southern California (maybe) to look at their facilities. I'm also getting the pieces together for our big projects in 2027. Personal: working on a talk for the local developer meet up in October; Going to talk about what I…
-
Trying to figure out how to get those corrugated iron sheet things to hang over my firewood and not fly away in the wind (without making lots of little holes that would let through water). Screws with waterproof gaskets connected to some plank underneath maybe? I have no idea what I'm doing :) Will probably make some half-hearted crappy attempt so…
-
**Work:** Working through issues with an EDR alerting pipeline. My choices are 1. Hack something on top of the existing system 2. move my queries somewhere else 3. build yet another scheduled query manager for just this system **Personal:** Rounding out my super rough proof of concept for a 2D platformer in Bevy using LDTK for level editing…
All 6 developments of Lobsters community shares weekend and weekly plans →
Lobsters