TMLR editor tests desk-rejected authors on their own papers, most fail
Nihar B. Shah, editor-in-chief of Transactions on Machine Learning Research, interviewed authors of ten rejected papers to see if they understood their own work.
What to know
- TMLR editor-in-chief Nihar B. Shah interviewed authors of ten already desk-rejected papers to test whether they understood their own technical content.
- Results: 3 authors dodged the interview, 3 could not answer basic questions, 3 answered basic questions but struggled with technical ones, and only 1 succeeded.
- Shah frames the exercise as relevant to concerns about AI-generated or heavily AI-assisted submissions and author accountability for correctness and integrity.
- The post has spread across Mastodon and Hacker News, drawing reactions ranging from praise for the accountability check to alarm at how poorly most authors performed.
Shah's interviews are a needed and admirable accountability check against AI-generated or AI-assisted papers.
-
“What a hero: Nihar B Shah interviewed authors of papers that would have been desk-rejected to see if they understood their own papers.”
ionica@mathstodon.xyz · Mastodon ↗
The finding that authors can't explain their own papers is genuinely alarming for the integrity of academic publishing.
-
“jfc”
inthehands@hachyderm.io · Mastodon ↗
The breakdown of results shows publication itself is a weaker signal of genuine researcher contribution than assumed.
-
“The value of publication as a signal of researcher contribution is much weaker when authors cannot explain or defend their paper.”
marick@mstdn.social · Mastodon ↗
Nihar B. Shah Editor-in-chief, Transactions on Machine Learning Research (TMLR)
How it unfolded 5 developments, newest first · click a bar or a number to jump posts
-
5
Hacker News discussion of the TMLR post continues
The Medium post circulates further on Hacker News under the title "Asking Authors About Their Own Papers (TMLR)", with additional submissions and a small comment thread.
-
This is the journal's policy on LLM use by authors [1]:> LLMs may be used as general-purpose assistive tools. Whichever tools are used, authors are fully responsible for content on which they are listed as (co-) authors. This includes, but is not limited to, content generated by LLMs that could be construed as plagiarism or scientific misconduct…
2 more of the top 3 · 22 posts in this stretch
-
T
What happens if an Editor asks an author about their use of AI? https:// medium.com/@TmlrOrg/asking-aut hors-about-their-own-papers-3d2e04e5dee0
-
I answered requests to be a peer reviewer. (I'm not sure why I was selected, I don't have many publications or credentials.) I saw a lot of papers with hallucinated references. I also remember one paper that described a methodology that I don't think the authors really performed, I think it was just academic fraud where they pretended to have…
-
-
4
Hachyderm user reacts with alarm to the article's rationale
A hachyderm.io post quotes the article's justification—linking author accountability requirements to concerns about AI-generated submissions—and reacts with dismay.
“jfc…”
— inthehands@hachyderm.io -
❝Most journals and conferences require authors to ensure correctness and integrity of their papers and take responsibility for them. This is particularly relevant given potential AI generated or heavily AI-assisted submissions. When authors could not answer questions about the technical parts, and sometimes even basic questions about the paper, it…
-
-
3
Mathstodon user hails Shah's interviews under #FuckAI
A mathstodon.xyz post praises Shah for interviewing authors of already desk-rejected papers to check their comprehension, tagging the post #FuckAI.
“What a hero: Nihar B Shah interviewed authors of papers that would have been desk-rejected to see if they understood their own papers.”
— ionica@mathstodon.xyz -
I
What a hero: Nihar B Shah interviewed authors of papers that would have been desk-rejected to see if they understood their own papers. https:// medium.com/@TmlrOrg/asking-aut hors-about-their-own-papers-3d2e04e5dee0 # FuckAI
-
-
2
Commenter summarizes the interview outcomes
A Mastodon post breaks down the results of Shah's ten interviews: three authors dodged the interview, three could not answer basic questions, three answered basic questions but struggled with technical ones, and one succeeded.
“The value of publication as a signal of researcher contribution is much weaker when authors cannot explain or defend their paper.”
— marick@mstdn.social (quoting the article) -
M
Editor-in-chief of Transactions on Machine Learning Research wanted to interview authors of ten (already desk-rejected) papers about the content of their own paper. The expected ensues. 3 dodged the interview. 3 could not answer basic questions. 3 could answer basic questions, but struggled with technical ones. 1 success "The value of publication…
-
-
1
Nihar B. Shah publishes account of author interviews
TMLR editor-in-chief Nihar B. Shah posts "Asking Authors About Their Own Papers" on Medium, describing how he interviewed authors of ten desk-rejected papers to test whether they understood the technical content of their own work, citing concern over AI-generated or heavily AI-assisted submissions.
-
first by Mastodon, 8d ago
-
What people are saying 18 voices from 1 site · best of 25 · verbatim
- Sep 20
-
This triggers a much bigger question for me - why does nothing happen to these authors based on this feedback? I don't mean "oh you missed a meeting or withdrew your paper so you get fired" by any means, but why isn't there a way for academic committees or employers to look up PhDs and see "oh interesting this person has a history of submitting…
-
> When authors could not answer questions about the technical parts, and sometimes even basic questions about the paper, it is difficult to see how they could have verified the paper's contentsIf you cannot answer, you did not author the paper, meaning you are misrepresenting your contribution, and there is already a process for this. And this is…
-
That’s a great “story from the trenches.”I’m wondering if anyone just flat-out admits they used LLMs in their work. I have no problem, doing that, myself, but I also have the luxury of not having my livelihood on the line, and not being overly-concerned about what people think of me.Eventually, I assume that AI will affect every aspect of the…
-
Academia is in a time for reckoning.Actors who already went through the process (or otherwise) gained sufficient reputation or credentials to self-market their own paper can skip journals entirely. That was what OpenAI did. With a sufficiently powerful AI model and correctional pipelines, generating a paper is trivial given some insight.I think we…
-
LLMs of course, but think the real issue here is incentives. Clearly the participants are incentivized to publish garbage.That's going to crater the signal to noise ratio of papers so best that be fixed asap. How though...
-
In the same vein, the Symposium on Theory of Computing (STOC) for 2027 has made some interesting changes to its Call For Papers (https://acm-stoc.org/stoc2027/stoc2027-cfp.html), notably requiring that papers be submitted beforehand to a preprint repository and that authors submit a 20-30 minutes video presentation explaining their results.
-
This could be renamed "Asking authors about their own pull requests" and the outcome would be exactly the same.
-
Curation is the new skill. The dismissal of a result on the basis of authorship is one of the reasons blind reviews are there. It sounds like we need more scientists. In al seriousness, this is a skill not just for science. Even at work, the amount of slop is rising exponentially and people are half-treating the symptom with its source, using AI…
- Sep 19
-
It strikes me as akin to plagiarism. If the purported author can’t even answer basic questions about the paper, how can they plausibly claim to have written it?In the case where someone uses AI to write the paper and then deeply familiarizes themself with it, it may go undetected, but then it’s also presumably less of an issue since they have…
-
> Accepted authors could "vouch" for "dead" papers in case they were "auto-killed"?The problem isn't with the papers here though, it is the author's understanding of the paper that is in question. A paper written by some hypothetically awesome AI would be a good paper but just not really the proclaimed author's paper.I think this highlights a dual…
-
> Separately, our group has been exploring approaches along these lines to make such evaluations more scalableActually, that sounds like an interesting idea for peer review in general, to include an interview between referees and authors. If it saves one round of rebuttals/reactions, it needn't even consume a lot more of everyone's time if you're…
-
> All reviewers, Action Editors and Editors-in-Chief for TMLR are unpaid volunteers.He is not an unpaid volunteer.He's an associate professor at the prestigious Carnegie Mellon University. He is not paid by the journal, but he is paid a salary by the university, and the university expects that a small part of his academic work is to serve as an…
-
Is it really “research” - as in expanding human knowledge - if nobody understands it? The point is deepening human understanding, not producing research papers
-
IMHO (not a paper writer, but read a lot during my grad school years), the Genie is out of the bottle. The only way forward, as I see it, is using LLMs for reviews also. Basically, filter all submitted papers with an LLM and ask it to summarize it, find the biggest weaknesses and main strong points, etc. that a human can then use to review the…
-
The Medium comments on this post are also on point. Running the same experiment with accepted papers is a good control. Running a similar experiment with reviewers would be interesting, but more obnoxious because they are not being paid.I would keep a private blacklist (shadow ban) the authors who wasted several hours of a reviewer's time to prove…
-
I'm not familiar with the world of academic publishing, so I want to ask: how is the industry making sure that submissions aren't at least partially AI-generated?Is it standard practice for authors to have to defend their submissions via interview like this? If not, why not?Does the vetting process vary with the quality of the publisher?As an…
-
One larger problem here is the value of a research paper is rarely the specific knowledge it adds but in the process of researching that adds to the collective knowledge+experience of those involved, especially training graduate students. AI papers shortcut this entirely. Academia has a lot to answer for this too by making papers the currency of…
-
Peer review has its historical issues, but the landscape of science and science-publishing has changed. New problems of authorship and authorial-understanding are now challenged by LLMs writing (at least) good sounding papers - some of which might be of acceptable quality in subject (I am not against AI in the sciences; some of the math work has…