« #AI reviewers are here — we are not ready » – by Giorgio F. Gilestro
#ArtificialIntelligence promises rapid and polite feedback on papers — but we must first review the reviewer.
https://www.nature.com/articles/d41586-025-03909-5 (paywall)
#OpenScience #Sciecne
1/7
« #AI reviewers are here — we are not ready » – by Giorgio F. Gilestro
#ArtificialIntelligence promises rapid and polite feedback on papers — but we must first review the reviewer.
Published in some Nature stuff
https://archive.is/VuiRP (no paywall)
#OpenScience #Sciecne
2/7
« #AI reviewers are here — we are not ready » – by Giorgio F. Gilestro
The noise in human peer reviews is crucial because it arises from the variability in human experience and practical knowledge […]
🤔 Human experience + practical knowledge = implicit knowledge?
Somehow, a “knowledge“ that you acquired but you cannot explicitely describe.
It reminds me Lecture on Ethics by Ludwig Wittgenstein…
« suppose this human wrote all they knew in a big book, then this book would contain the whole description of the world; and
what I want to say is, that this book would contain nothing that we would call an ethical judgment or anything that would logically imply such a judgment.
It would of course contain all relative judgments of value and all true scientific propositions and in fact all true propositions that can be made.
But all the facts described would, as it were, stand on the same level and in the same way all propositions stand on the same level.
There are no propositions which, in any absolute sense, are sublime, important, or trivial. »
– Ludwig Wittgenstein, Lecture on Ethics (1965)
https://www.wittgensteinproject.org/w/index.php/Lecture_on_Ethics
4/7
« #AI reviewers are here — we are not ready » – by Giorgio F. Gilestro
Researchers understand, for instance, which experiments are feasible within realistic budget and time constraints, and which protocols are robust or temperamental. But an LLM can only reflect the literature, and published claims can be exaggerated. Moreover, specialists know when older approaches in their field have been, or should be, superseded by techniques that are just starting to appear in an LLM’s training data set.
🤔 This sounds the same criticism as Wittgenstein’s book, right?
« #AI reviewers are here — we are not ready » – by Giorgio F. Gilestro
🤔 It makes me think that as humans we’re individually super smart to invent mechanical tools for (most of the time) compensating our collective dumb failures.
Guess what? For easing this LLM review, I bet the journals’ll ask to submit the paper formatted by another LLM. Somehow a Long Looping Mediocre (LLM) process…
Reading « #AI reviewers are here — we are not ready » – Good reading!
🤔 It makes me think:
AI reviewers: Yet another example of stupid collective human behaviour?
Instead of rethinking in depth what peer-review means, we pile up technical patches* that fix each other when the bug is by design.
• Peer-review = a mean for a end
• End = build a trust
• Trust = Be convinced that’s “good“
• good is more than correct!
• And good is about a judgment
So we’re back on “Lecture on Ethics” by Wittgenstein. 🤩
*patches: becoming more and more resource hungry…
Next patch: conquer another planet?
