Engineers swap notes on taming AI-generated code's hidden failure modes
A cluster of blog posts this week catalogs why AI-written code looks finished but isn't, and the guardrails, review habits and layered models developers are building to compensate.
Part of a larger narrative
The AI Control Crisis
- Australia probes OpenAI Medicare hack as more rogue AI incidents surface
- Bessent Says OpenAI Managers, Not AI Agents, Are to Blame for Hugging Face Hack
- Claude Opus 5 used to breach OpenAI's internal systems in under 72 hours
- OpenAI forms independent advisory group on mathematics and AI
- OpenAI launches Astra for Law, pushing into Big Law against Anthropic
What to know
- Several independent engineering blog posts published within the same week describe a shared problem: AI-generated code looks complete and passes demos while hiding deep, hard-to-find failures.
- Practitioners describe compensating with custom tooling — build-time "guardrail" scripts, disciplined code review, and explicit prompting for UI and copy — rather than trusting AI output directly.
- Engagement across the coverage is low (scores of 1-3, almost no comments), indicating this is a niche practitioner discussion rather than a widely debated event.
sitecmd.com blog author self-described senior software engineer, blog authorJimmy Miller software engineer, blog author (jimmyhmiller.com)Daniel Lemire computer scientist, blog author
How it unfolded 7 developments, newest first · click a bar or a number to jump articlespostscomments
-
7
Ludwig low-code AI framework circulates on Hacker News
A GitHub repository for Ludwig, a low-code framework for building custom AI systems, is posted to Hacker News as part of the ongoing thread of AI-development tooling discussion.
-
Feels identical to what is happening in software to me. > In many fields and activities, years of training have traditionally served not only to produce a final answer or product, but also to develop understanding and the ability to formulate new questions and ideas. Sounds like what I try to do every day. All that old schpiel we used to say about…
2 more of the top 3 · 94 posts in this stretch
-
M
Noted mathematician Terrence Tao has coordinated a great open letter about AI and society. The technology in use now will never become sentient. How it is used and who owns it are the actual issues to consider. Humans have not achieved anything close to alignment among ourselves, and that is the real issue rather than imaginary scenarios about…
-
>> Famous problems have often served as landmarks and lighthouses against which one can measure an improved understanding of this landscape. Solving one of these problems has been a certain sign of new insights and interesting methods, which would then be studied by a community of mathematicians, through a long and arduous process of talks…
-
-
6
A 2016 essay on AI-based programming resurfaces
An older post, "How AI based programming could work (2016)," is shared on Hacker News, drawing renewed discussion of early proposals for AI-assisted software development.
-
5
Blogger describes the distinct "shape" of unfinished AI codebases
Jimmy Miller's post "The Chasm: The Shape of Unfinished AI Codebases" argues that AI-written programs give a convincing illusion of completeness — passing tests and demos — while hiding deep, unpredictable failures that differ fundamentally from the visible gaps typical of human-authored unfinished code.
“For human-authored code, you can see the cracks forming before you drop off the cliff. For AI codebases, the chasms are deep, hidden, and often impossible to climb out of.”
— Jimmy Miller -
4
Video lecture covers generalization and data selection in AI
A YouTube talk, "The Foundations of Modern AI: Generalization, Data Selection, and Epiplexity," is shared to Hacker News as part of the same week's cluster of AI-development-focused material.
-
3
Engineer details a year of workarounds for AI coding "slop"
A blog post titled "A Senior Software Engineer's Perspective on Building with AI" describes a year of experimentation with multiple AI coding tools, arguing AI needs detailed prompting to avoid cluttered UIs and generic copy, and describes writing over 120 custom "guardrail" scripts that fail the build when a previously fixed AI mistake reappears.
“if you let them do everything, you will end up with a sloppy, cluttered mess…”
— sitecmd.com blog author -
2
CACM piece urges code review as a check on AI coding
Communications of the ACM publishes "Leverage Code Review for Sustainable AI Coding Development," arguing structured code review is necessary to keep AI-assisted development sustainable; the piece is shared on both Hacker News and Lobsters the same day.
-
1
Lemire publishes a layered model for AI programming
Daniel Lemire's blog post "AI Programming: A Layered Model" is posted to Hacker News, framing how AI fits into different layers of the software development process.
What people are saying 21 voices from 2 sites · best of 94 · verbatim
- Sep 13
-
I am gobsmacked by this letter.Every one of the mathematicians on this letter was likely aware of the progress in AI, and knew this was coming if they were honest with themselves. Some have even helped pushed the agenda forward. Inevitably, there would be a day when AI would find solutions to problems that defined the field. That day has come…
- Sep 12
-
Yeah as a profession and a discipline, I don't feel that we as computer scientists have done the self-reflection about the purpose of our work that mathematicians have. I can't imagine the ACM publishing a statement like this. :(
-
> The goals of the AI companies and the goals of the mathematical community are severely misaligned. We see these as part of broader alignment issues impacting other scientific and creative professions, as well as the whole of society. The frontier labs' business model is entirely based on extracting value from other's work. If they had to fairly…
-
I agree partially, and feel that many software folks have been doing work that was more or less pointless, such as your restaurant website example. These problems are solved, and tooling/generators almost always exists that can accomplish the same without programmer involvement. However, outside of the low/no-code domain, (and we've seen time and…
-
I don't think that this is the same as with software. For many (but not all) software applications, the end goal is to have working software that accomplishes a specific task, not building a human conceptual understanding of a problem space. Like, when I use the website for a restaurant, I want to know what's on the menu and maybe be able to order…
-
As a consumer of mathematics, I want to attack the author’s idea of “what that work was meant to achieve in the first place”.I do a lot of programming that uses mathematics. I designed the error recovery file format “par2”. I worked as a quantitative trader.What do I want from math? To be able to search math results, understand them, and apply…
-
It reminds me of Frédéric Bastiat's Candle Makers' Petition. It satirised protectionism in the 1800s:>We candlemakers are suffering from the unfair competition of a foreign rival. This foreign manufacturer of light has such an advantage over us that he floods our domestic markets with his product. And he offers it at a fantastically low price…
-
As a software engineer, 6 months ago. I was better than AI in essentially every way except raw typing speed.Now AI has surpassed me in fine-grained problem-solving ability. It can write more reliable code than I can, faster than I can, provided it is given the right guidance.But one thing that I'm still much better at is identifying technical…
- Sep 11
-
This is a PR problem, not a mathematical problem. It's possibly the worst PR problem mathematics has faced since the execution of Hippasus for whistleblowing on the cover-up of the regular dodecahedron. It's still a PR problem.So what went wrong? Mathematics education. Math below grad school is all about solving stated problems. Credit is given…
-
I think your question is adjacent to Model Collapse? https://en.wikipedia.org/wiki/Model_collapse Not directly, as you're focused on how specific knowledge gets emphasized/deemphasized as a second-order effect of LLMs "taking over" a field, but it seems like a similar issue. In at least some conditions retraining on e.g. LLM-generated proofs might…
-
The quote at the top is inspiring but not realistic. Humans have long produced proofs that are difficult to follow and there are many proofs that only a few understand after long and specialized study. It seems out of place to me to complain about mathematical discoveries made by computers simply because they might be hard to understand by humans…
-
It's also identical to what HAD happened to software cf. The Rise of Worse is Better. Human slop happens well before AI slop. Now it's only getting worse!
-
Aye, it's hard to deny that software has a function at least somewhat divorced from the process of its creation, although it's still up in the air for me as to how much. I am oddly optimistic to see this issue so starkly realized in the field of mathematics. If we can come to consensus that "proof factories" are not really helpful, that's at least…
-
One question I have is whether/how AI "solutions" make it back into future models. I've sort of naively assumed that AI companies basically scrape the web in large swaths, do a bit curating, and train. In a case where an AI solves math problems, I conjecture, that lots of text is created documenting that the discovery happened (only because its…
-
The situation seems analogous indeed. However, cynically, I do wonder whether the outcome will differ across the CS and math industry once the dust settles. One could argue that software production is funded in large part for the end product, not the process, whereas mathematics is far more exploratory. More concretely, it seems to me that an…
-
But the work the Mochizuki case generated can also be done by AI. AI could generate a landmark proof and then people could use it to solve or simplify intermediate problems and you could use a different AI prompt to try to disprove it if you were really skeptical. From my memory I think they said it took 88 hours to solve a Millenium Problem…
-
I understand this stance and where they are coming from, but I can't help but think this sounds very analogous to engineers' arguments against AI-assisted and vibe-coding, especially with regard to cognitive debt. Yet the software industry is plowing ahead, reportedly pushing mountains of unreviewed code to Prod, and the world hasn't ended.Of…
-
I agree with many of the sentiments here. But an open letter signed exclusively by Fields Medalists that purport to define precisely what the "mathematical community" (who is inside and outside) and what their goals are raises my hackles for some reason.It is true that the manufacturing of "true/false" statements is not the same as gaining…
-
@pushcx, could you merge this into https://lobste.rs/s/xsbz3l/severe_misalignment_ai_mathematics (or vice versa)?
-
As a mathematician maybe I am a little more optimistic than this declaration.I am thinking of Mochizuki's abc conjecture: He worked in relative isolation, and dumped a huge incomprehensible proof on the community (to oversimplify a bit). That's not totally unlike what might happen if AI generates a huge, incomprehensible proof of let's say…
-
> We are witnessing a general threat to intellectual work, with misalignment between the outcome of the use of AI and its initial purpose. In many fields and activities, years of training have traditionally served not only to produce a final answer or product, but also to develop understanding and the ability to formulate new questions and ideas…