Alan Grow documents decade of kill command bugs in Ubuntu LTS releases
A single outage traced to a parsing flaw that misparsed negative process group IDs as -1, killing all but init—a bug pattern repeated across six Ubuntu versions.
What to know
- A single kill command with a negative process group ID was misparsed by procps's getopt implementation, resulting in -1 being sent as the pid—which Unix interprets as a signal to all processes except init, nearly crashing the machine.
- The root cause—getopt truncating unrecognized options to a single character—shipped in the same form across six Ubuntu LTS releases (14, 16, 18, 22, 24) over a decade, suggesting systematic gaps in option parsing testing.
- The bug occurred because negative pids used for process group signaling were misinterpreted as unrecognized options, exposing a friction between Unix convention and the getopt standard library approach.
The dispute Whether the root cause is getopt's design philosophy or the absence of regression testing discipline. · positions read across 20 posts and comments
Getopt design is fundamentally at fault for this family of bugs.
-
“I do regard getopt dogma as the real root cause of the outage.”
dreugeworst · Reddit ↗
The bug persists because option parsing tests were never added after the initial fixes.
-
“I would also blame the fact that apparently option parsing tests weren't added to the project even after fixing these bugs”
dreugeworst · Reddit ↗
Alan Grow Endcrawl engineer and investigatorprocps maintainers Maintainers of /bin/kill in Ubuntu
How it unfolded 1 development · click the chart to see its coverage postscomments
-
1
Community identifies getopt dogma and lack of testing as systemic issues
Commenters identify the underlying problem: reliance on getopt for option parsing and the absence of regression tests for option parsing after previous fixes were applied.
“I do regard getopt dogma as the real root cause of the outage. I would also blame the fact that apparently option parsing tests weren't added to the project even after fixing these bugs…”
— dreugeworst -
Oh dear, this is the kind of slapstick that’s so painfully cringe it’s just sad. I was a bit surprised by the early history: my guess would have been that killing a negative PID was introduced with the rest of the process group / job control features in BSD in the early 1980s. So I had a look in the CSRG archives: * userland kill(1) gained support…
2 more of the top 3 · 20 posts in this stretch
-
I do regard getopt dogma as the real root cause of the outage. I would also blame the fact that apparently option parsing tests weren't added to the project even after fixing these bugs
-
I thought this was a great article. I appreciate the length they went to digging into the topic and contextualized it with historical facts. Like I didn’t know process groups were a thing and have been a thing since 1986. So much more to learn about. Thanks for sharing.
-
-
background
Alan Grow publishes detailed post mortem and recommendations — Grow publishes 'A Decade of Kill Bugs' documenting the outage, root cause analysis, and the historical pattern of the same bug across six Ubuntu versions. The post circulates on Lobsters, Hacker News, and Reddit.
-
background
Same bug pattern found across multiple Ubuntu LTS versions — The root cause analysis uncovered that the same kill command bug shipped in Ubuntu 14, 16, 18, 22, and 24 LTS releases, representing a decade-long pattern of the same vulnerability.
-
background
Root cause identified: getopt parsing truncates negative PID to single character — Investigation revealed that procps 4.0.4's kill implementation mishandles unrecognized options. The getopt() function places only the first character after the dash into optopt, so -1490947 becomes character comparison resulting in pid -1, which Unix interprets as a signal to all processes except init.
-
background
Kill command with negative PID misparsing triggers Endcrawl outage — Alan Grow attempted to cancel a process group render using a negative process group identifier (pgid) but the kill command misparsed the argument, killing nearly all processes on the machine except init. A second outage followed shortly after.
What people are saying 17 voices from 1 site · best of 20 · verbatim
- Why were option parsing tests not added to procps after the initial bug fixes?
- Should Ubuntu or procps maintainers have caught this pattern across six releases?
- Nov 4
-
Upstream procps isn't hostile to contributions. I expect they'd gladly accept more tests, especially in DejaGnu form.
-
You’ve probably already seen this, but Ken Thompson absolutely agreed about `creat`: > Ken Thompson was once asked what he would do differently if he were redesigning the UNIX system. His reply: "I'd spell creat with an e.”
-
Out of curiosity, have you considered submitting extra tests upstream? (I wondered if I should try to do this myself as a result of just reading your blog post; your reply suggests that there is already some low-hanging fruit in the downstream versions.)
- Nov 3
-
There's a [DejaGnu](https://www.gnu.org/software/dejagnu/) test suite in procps, and it has some [kill tests](https://gitlab.com/procps-ng/procps/-/blob/master/testsuite/kill.test/kill.exp?ref_type=heads). Nothing that tests negative pids though. The incoming Ubuntu 24 fix includes [comprehensive negative pid…
-
> And so it came to pass that Linux's kill was reinvented written by Albert Calahan. After Albert DoSed himself by forking 5 children, Debian downstream maintainer Craig Small took over poor old neglected procps. This is a fantastic line, tysm
-
Thanks for the feedback. I suppose I could impose a maximum absolute font size with a `@media` rule.
-
I lost interest as soon as I saw that. How is that a bug? That is the expected and documented behaviour.
-
I doubt they would have had so many problems if they had used Python’s `argparse` module. Unfortunately,,
-
I think there's a lot of this in the older parts of UNIX heritage. Things that optimize for brevity far more than would appear sensible today. (`creat`? really? you couldn't spare the extra letter?). I think it's part an aesthetic ideal that's kind of outmoded today and part the result of genuine technical circumstances at the time. If you're on a…
-
Ultimately my takeaway is that the standard interface to `kill` is way too ambiguous for its own good. There is already a `-s/--signal` option for specifying the signal explicitly (rather than eg `-9`), perhaps it’s time that there was an explicit option to specify a process group too? It seems insane to overload everything into negative numbers…
-
Oh gods the reference to Burn After Reading. The only part of that whole movie worth remembering. That said, this mostly seems like bugs around using `getopt()` properly. `getopt()` style command line parsing has always seemed like a cure that's far worse than the disease.
-
Yeah, to be honest this is the first time I have heard of negative PIDs for process groups, but I would have added a double dash in front of the argument as a matter of course. First learned that trick trying to delete a filename starting with a hyphen, back in the closing years of the last millennium.
-
I didn't immediately know what the problem was after seeing this bug, but I immediately guessed the solution: > kill -TERM -- -1490947 # yes I guess my brain is just so GNU poisoned that negative number args always have me reaching for --.
-
The article does *not* say that procps' kill suffers from quality issues, but this seems to be one of the implicit take-aways. It is not too hard to add unit tests to exercise the various cases, and this would have found the issues and avoided the regressions. What are the tests like? Why weren't thorough tests added on the first…
-
The text on this page is super large for me (on a 2560x1440 monitor) and surprisingly, zooming out doesn't help at all! It seems the culprit is this CSS rule: ``` html { font-size: 1.20vw; } ``` Disabling this rule makes the text a sane size. Otherwise, great article!
-
that v86 link is so cool, I have not seen that before
- Nov 2
-
Yeah the historical question still interests me. Did V9 really originate negative pid killing via `/bin/kill`, or did System V or SunOS already have it? If the main use case was shell job control, maybe there wasn't yet a good reason to expose it. The extent of my spelunking for the post was [TUHS](https://www.tuhs.org/cgi-bin/utree.pl), the…