Serce explains why whitespace detection is deceptively complex
A deep dive into why checking for whitespace isn't as simple as comparing a character to a space.
What to know
- Whitespace detection is far more complex than checking for a space character—Unicode defines 25 whitespace characters with different purposes and visual properties.
- Major programming runtimes (Java, Python, JavaScript) implement conflicting definitions of whitespace, creating interoperability challenges and potential bugs.
- Even characters named 'space' can look empty or not, and some contain the word 'space' but aren't whitespace characters at all, making intuitive classification unreliable.
Serce Developer and author
How it unfolded 2 developments, newest first · click a bar or a number to jump posts
-
1
Article explains why Unicode whitespace definition matters
The piece notes that Unicode's White_Space property includes 25 characters, each serving distinct purposes—figure space for table digits, thin and hair space for variable gaps, no-break space for line-wrap control, and even OGHAM SPACE MARK that can render as a visible line in certain fonts.
“Yet the code looks a lot more complex than a simple c == ' ' check.”
— Serce, Author · source -
Perfect hashing is ready cute! The implementation is very small and simple even though it's very fast: it's just the proof that it does actually work which is complicated. :D
2 more of the top 3 · 14 posts in this stretch
-
> What adds more complexity is that Unicode is not frozen in time, and is constantly evolving. And this affects the definition of whitespace. For example, Unicode 6.3 had reclassified MONGOLIAN VOWEL SEPARATOR (U+180E) from a space separator to a format character, removing its White_Space property. The Unicode whitespace set had changed from 26…
-
I think it helps to have an intuition for how high-performance multipliers work to sanity check numbers like that. Here's my hand-wavy intuition for why the 3 logical stages map to roughly 3 cycles of latency in high performance implementations. If you look at industry papers on multipliers you'll see that it's not _too_ far from the truth…
-
-
background
Article details how runtimes disagree on whitespace definitions — The analysis reveals that Java excludes no-break space (U+00A0) from its whitespace set, while Python includes it and also recognizes 29 characters total. JavaScript uses yet another approach with regex character classes, each runtime implementing its own definition rather than following a single standard.
-
2
Serce publishes analysis of whitespace detection complexity
A developer published an article examining the Guava library's whitespace-checking method from Google, exploring why the code is far more complex than a simple space check. The piece traces the problem to Unicode's definition of 25 whitespace characters and conflicting implementations across runtimes.
“Yet the code looks a lot more complex than a simple c == ' ' check. This made me curious and sent me down a small rabbit hole…”
— Serce
What people are saying 11 voices from 1 site · best of 14 · verbatim
- Nov 1
-
For some reason, I had it in my head that it was 1 clock cycle now, but apparently my memory is crap. It used to be 39 cycles back when I was doing assembly on the 386. And floating point was all done in software, unless you were lucky enough to have a 8087 (80387 or whatever) co-processor.
-
Just looked it up. 3-4 cycle latency. Still impressive. (Unless you interpret that as saying that the clock speed is too low.)
- Oct 31
-
If that was the case they'd have chosen the optimal (in hamming weight) value 0x51032000. Or the smallest value 0x8fd3817. 1682554634=0x6449bf0a was likely the result of a random search; the probability of a random 32-bit multiplier to generate a perfect hash is ~2^-14, so it wouldn't be out of place.
-
There's a bit of probability involved in the proof that you can find a suitable hash function.
-
A 64-bit integer multiply is 3 cycles of latency on a modern core, not 1 like an integer add. Integer multiply units have been fully pipelined for a long time, though, so each IMUL port can start a new multiply per cycle. That's the usual latency vs throughput thing. For a long time x86 cores only had 1 IMUL port, but both Intel and AMD released…
-
How is the proof complicated? They compare the ...charAt(HASH) == c. If only whitespace is in that table, then it can't possibly yield false positives. How do you check for false negatives? You just test that those couple dozen values in the table all hash correctly. Or am I missing something?
-
IIUC: On modern CPUs, integer multiplications are single-cycle operations. Intel double pumps them, so they're effectively a half cycle. (I'm sure that there are lots of clever cheats to get there.)
-
From memory (recollection of a university lecture about 18 years ago), the usual trick for picking hash functions for perfect hashing is: - pick a function at random from some family of hash functions - you need that function's outputs to be distinct for all of the elements you care about - which isn't a very strict property - so when the randomly…
-
Some multiplications are faster than others, as compilers will try to use additions, shifts, and the special address-offset computation instructions to avoid using the general-purpose mulitplication circuits. But I didn't notice any regularities in 1682554634 that would enable fast multiplication.
-
> I think it’s usually just brute forced, right? Quite possibly, but I'm not sure how and why they arrived at exactly this number since it's one of many options.
- Oct 30
-
> The magic 1682554634 number does look magical, and it's unclear to me how it was chosen (please do share if you know the answer). I think it’s usually just brute forced, right?