Hacker News discussion of the TMLR post continues
5 Sep 18 3:16 AM · 5d ago · 1 post · 1 source · development 5 of 5
The Medium post circulates further on Hacker News under the title "Asking Authors About Their Own Papers (TMLR)", with additional submissions and a small comment thread.
Nihar B. Shah Editor-in-chief, Transactions on Machine Learning Research (TMLR)
The whole story posts the bright band is this development · numbered dots are the others · click one to jump
What people said 22 voices · verbatim
-
This is the journal's policy on LLM use by authors [1]:> LLMs may be used as general-purpose assistive tools. Whichever tools are used, authors are fully responsible for content on which they are listed as (co-) authors. This includes, but is not limited to, content generated by LLMs that could be construed as plagiarism or scientific misconduct…
-
T
What happens if an Editor asks an author about their use of AI? https:// medium.com/@TmlrOrg/asking-aut hors-about-their-own-papers-3d2e04e5dee0
-
I answered requests to be a peer reviewer. (I'm not sure why I was selected, I don't have many publications or credentials.) I saw a lot of papers with hallucinated references. I also remember one paper that described a methodology that I don't think the authors really performed, I think it was just academic fraud where they pretended to have…
-
> Accepted authors could "vouch" for "dead" papers in case they were "auto-killed"?The problem isn't with the papers here though, it is the author's understanding of the paper that is in question. A paper written by some hypothetically awesome AI would be a good paper but just not really the proclaimed author's paper.I think this highlights a dual…
-
Academia is in a time for reckoning.Actors who already went through the process (or otherwise) gained sufficient reputation or credentials to self-market their own paper can skip journals entirely. That was what OpenAI did. With a sufficiently powerful AI model and correctional pipelines, generating a paper is trivial given some insight.I think we…
-
This triggers a much bigger question for me - why does nothing happen to these authors based on this feedback? I don't mean "oh you missed a meeting or withdrew your paper so you get fired" by any means, but why isn't there a way for academic committees or employers to look up PhDs and see "oh interesting this person has a history of submitting…
-
The Medium comments on this post are also on point. Running the same experiment with accepted papers is a good control. Running a similar experiment with reviewers would be interesting, but more obnoxious because they are not being paid.I would keep a private blacklist (shadow ban) the authors who wasted several hours of a reviewer's time to prove…
-
That’s a great “story from the trenches.”I’m wondering if anyone just flat-out admits they used LLMs in their work. I have no problem, doing that, myself, but I also have the luxury of not having my livelihood on the line, and not being overly-concerned about what people think of me.Eventually, I assume that AI will affect every aspect of the…
-
I'm not familiar with the world of academic publishing, so I want to ask: how is the industry making sure that submissions aren't at least partially AI-generated?Is it standard practice for authors to have to defend their submissions via interview like this? If not, why not?Does the vetting process vary with the quality of the publisher?As an…
-
> When authors could not answer questions about the technical parts, and sometimes even basic questions about the paper, it is difficult to see how they could have verified the paper's contentsIf you cannot answer, you did not author the paper, meaning you are misrepresenting your contribution, and there is already a process for this. And this is…
-
IMHO (not a paper writer, but read a lot during my grad school years), the Genie is out of the bottle. The only way forward, as I see it, is using LLMs for reviews also. Basically, filter all submitted papers with an LLM and ask it to summarize it, find the biggest weaknesses and main strong points, etc. that a human can then use to review the…
-
It strikes me as akin to plagiarism. If the purported author can’t even answer basic questions about the paper, how can they plausibly claim to have written it?In the case where someone uses AI to write the paper and then deeply familiarizes themself with it, it may go undetected, but then it’s also presumably less of an issue since they have…
-
Peer review has its historical issues, but the landscape of science and science-publishing has changed. New problems of authorship and authorial-understanding are now challenged by LLMs writing (at least) good sounding papers - some of which might be of acceptable quality in subject (I am not against AI in the sciences; some of the math work has…
-
> Separately, our group has been exploring approaches along these lines to make such evaluations more scalableActually, that sounds like an interesting idea for peer review in general, to include an interview between referees and authors. If it saves one round of rebuttals/reactions, it needn't even consume a lot more of everyone's time if you're…
-
In the same vein, the Symposium on Theory of Computing (STOC) for 2027 has made some interesting changes to its Call For Papers (https://acm-stoc.org/stoc2027/stoc2027-cfp.html), notably requiring that papers be submitted beforehand to a preprint repository and that authors submit a 20-30 minutes video presentation explaining their results.
-
One larger problem here is the value of a research paper is rarely the specific knowledge it adds but in the process of researching that adds to the collective knowledge+experience of those involved, especially training graduate students. AI papers shortcut this entirely. Academia has a lot to answer for this too by making papers the currency of…
-
Curation is the new skill. The dismissal of a result on the basis of authorship is one of the reasons blind reviews are there. It sounds like we need more scientists. In al seriousness, this is a skill not just for science. Even at work, the amount of slop is rising exponentially and people are half-treating the symptom with its source, using AI…
-
> All reviewers, Action Editors and Editors-in-Chief for TMLR are unpaid volunteers.He is not an unpaid volunteer.He's an associate professor at the prestigious Carnegie Mellon University. He is not paid by the journal, but he is paid a salary by the university, and the university expects that a small part of his academic work is to serve as an…
-
LLMs of course, but think the real issue here is incentives. Clearly the participants are incentivized to publish garbage.That's going to crater the signal to noise ratio of papers so best that be fixed asap. How though...
-
Is it really “research” - as in expanding human knowledge - if nobody understands it? The point is deepening human understanding, not producing research papers
-
This could be renamed "Asking authors about their own pull requests" and the outcome would be exactly the same.
-
That would fail, humans and agents could create new “author” accounts by the swarm or have paid author accounts.
All 5 developments of TMLR editor tests desk-rejected authors on their own… →
Hacker NewsMastodon