https://academia.stackexchange.com/questions/204370/what-should-i-do-if-i-suspect-one-of-the-journal-reviews-i-got-is-al-generated Stack Exchange Network Stack Exchange network consists of 183 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers. Visit Stack Exchange [ ] Loading... 1. + Tour Start here for a quick overview of the site + Help Center Detailed answers to any questions you might have + Meta Discuss the workings and policies of this site + About Us Learn more about Stack Overflow the company, and our products. 2. 3. current community + Academia help chat + Academia Meta your communities Sign up or log in to customize your list. more stack exchange communities company blog 4. 5. Log in 6. Sign up Academia Stack Exchange is a question and answer site for academics and those enrolled in higher education. It only takes a minute to sign up. Sign up to join this community [ano] Anybody can ask a question [ano] Anybody can answer [an] The best answers are voted up and rise to the top Academia 1. 1. Home 2. Questions 3. Tags 4. 5. Users 6. Unanswered 2. Teams Stack Overflow for Teams - Start collaborating and sharing organizational knowledge. [teams-illo-free-si] Create a free Team Why Teams? 3. Teams 4. Create free Team Teams Q&A for work Connect and share knowledge within a single location that is structured and easy to search. Learn more about Teams What should I do if I suspect one of the journal reviews I got is Al-generated? Ask Question Asked today Modified today Viewed 10k times 28 I've submitted a paper to one of the well-reputable journals and received two reviews asking for a revision. One of the reviews is on-topic and helpful. The other, however, I suspect was generated by an Al based on the abstract only. I don't want to debate the decision (the second reviewer is fair), but I wonder if I should mention my suspicions to the journal editor. Should I? I doubt I can prove Al's use. All I have is a strong suspicion. Plus, I don't want to hurt my article. Yet, using Al for generating (not merely styling) reviews seems like such a bad practice that maybe it's worth letting the editor know. My suspicions are based on several things: * First, the style is a bit off. The review consists only of very long questions like: "In terms of the paper's stated goal of contributing to the literature, how well does it provide [contribution mentioned in abstract] for [use-case mentioned in abstract]?". There are no suggestions, no feedback, and no statements. It's only broad questions suspiciously rephrasing each line of the abstract (and heavily re-using entire phrases from it). * Second, the list of suggested articles is strange and includes existing papers (only names) from irrelevant fields. * Third, out of curiosity, I've run the text via several online "detect AL" tools, and all of them conclude it's Al. It's not valid proof, just a fun point. * peer-review * ethics * generative-ai Share Improve this question Follow asked 7 hours ago pintor's user avatar pintorpintor 30122 silver badges55 bronze badges New contributor pintor is a new contributor to this site. Take care in asking for clarification, commenting, and answering. Check out our Code of Conduct. Add a comment | 3 Answers 3 Sorted by: Reset to default [Highest score (default) ] 31 I think you should contact the editor with basically the same content as your post here: explain why you think the review could be AI generated, including the limitations of each point of evidence. I think most editors see their job as letting reviewers guide their decision rather than reviewing the reviews, so I would start from the assumption that the editor would not have considered checking the reviews for suspicion of gen AI. Explain why you think this is concerning, which I think at minimum would include the integrity of the peer review process (you and your colleagues are expecting papers to be reviewed by humans, not AI) and the protection of your intellectual property (having an AI review your paper involves inserting your work into a 3rd party tool which may incorporate it into the training data and is a definite breach of the confidentiality that reviewers are sworn to). I would check the journal's website to see if they have an explicit statement about use of AI in peer review; if so, you can refer to that as well. Finally, make clear that you're not protesting the decision for a revision and plan to work on revising and resubmitting your paper. I'd include a note thanking the editor and the reviewers for their time. From there, let the editor decide what should be done next; it's their call, your only duty here is to raise their awareness. It's possible they'll want to bring in another reviewer. It's also possible they'll decide it's okay. At that point you'll have to decide whether you're satisfied with their effort and whether that affects your opinion of the journal as an appropriate place to submit your work. Share Improve this answer Follow edited 6 hours ago answered 7 hours ago Bryan Krause's user avatar Bryan KrauseBryan Krause 103k2424 gold badges297297 silver badges376376 bronze badges 3 * 4 Yes (YESSS!), editors should be made aware of this so that appropriate policies can be developed. It goes beyond the concerns of a particular author or paper and requires deep thought. - Buffy 6 hours ago * 3 Thank you! I've checked peer-reviewing guidelines, but it says nothing about explicitly prohibiting Al-use for reviewers. At least not at first glance. Though, implicitly, it can be tied to copyrights (as you mentioned). However, I found that the journal has an ethics team. Do you think it's better to ask them instead of the editor? - pintor 6 hours ago * 3 @pintor Not too surprising, this genAI stuff is still pretty new and publishing moves slowly, and I agree with you that it's likely already covered by standard confidentiality policies even if not explicit. I'd start by working with the editor. You might possibly ask the editor if this is something the ethics team could help out with, but I'm guessing they exist more as a check on the editorial team when editors run amok (or, cynically, they just exist to quash ethical complaints). Going to them directly may be seen as an accusation against the editor, which you don't intend to make. - Bryan Krause 5 hours ago Add a comment | 2 Edit in response to discussions here. I think my answer is wrong. @BriahKrause and @Buffy and @leonos are right. The OP should contact the editor immediately. Leaving this post up so the back and forth is visible. --------------------------------------------------------------------- Interesting puzzle. Since the (possibly AI prepared) review is generally positive, treat it as if it were legitimate. Since the reviewer asks for no changes you need not argue about whether they are appropriate. If any of the suggested references are in fact relevant, include them. After the paper is accepted you could contact the editor to voice your suspicions, with whatever supporting evidence you have. Edit after reading @BrianKrause 's answer. Clearly you will revise your manuscript in response to the good suggestions from the unquestionably legitimate review. If you think the work would benefit from a second good review then you should tell the editor now about your suspicions. Share Improve this answer Follow edited 6 hours ago answered 7 hours ago Ethan Bolker's user avatar Ethan BolkerEthan Bolker 32.6k11 gold badge7676 silver badges119119 bronze badges 5 * 5 This suggestion feels unethical. Only raise the problem if the review was negative? Either using "AI" to write a review is a problem, or it is not, this should be independent of the outcome of the artificial review. - leonos 6 hours ago * 1 @leonos Perhaps. My thought was that the ethics of the situation did not require the OP to risk losing the chance to publish by possible antagonizing the editor. There are plenty of lazy referees out there. I hope the OP will report back here when the dust has settled. - Ethan Bolker 6 hours ago * 4 in my opinion, this is a clear ethical issue and the authors should send a clear and immediate message that it is not acceptable - particularly because the journal might not care that much. Imagine if journals could just dispense from finding reviewers and artificially generate reviews at no additional time cost...a gold mine for predatory journals. It's quite concerning that the editor did not even notice or care about that report in the first place. - leonos 6 hours ago * 1 @leonos You're right. See my edit. - Ethan Bolker 5 hours ago * 1 Ethan - I saw it! Thanks for the edit and for being open to changing your mind! - leonos 5 hours ago Add a comment | 0 I suspect one of the journal reviews I got is Al-generated I wonder if I should mention my suspicions to the journal editor. I don't think that's a relevant criteria. Whether the review is useful is a more important criteria. You could focus on the lack of constructive feedback or how the comments seemed to closely align with your abstract without offering substantial insights or suggestions for improvement. Expressing gratitude for the helpful feedback from one reviewer while politely pointing out the absence of actionable feedback from the other might prompt the editor to consider the nature of the review. Share Improve this answer Follow answered 4 hours ago Julien Reszka's user avatar Julien ReszkaJulien Reszka 13322 bronze badges New contributor Julien Reszka is a new contributor to this site. Take care in asking for clarification, commenting, and answering. Check out our Code of Conduct. 2 * 3 Strongly disagree; it's highly relevant, for the reasons I point out in my answer. - Bryan Krause 4 hours ago * 4 In addition, the peer-reviewing system relies on the implicit trust that the editor will choose reviewers that are knowledgeable experts of the relevant field. At this stage, the language models are not knowledgeable, experts, or even concerned about truth at all. Using one to write a peer-review completely misses the point of peer-review which is to promote scientific accuracy. - leonos 2 hours ago Add a comment | You must log in to answer this question. Not the answer you're looking for? Browse other questions tagged * peer-review * ethics * generative-ai . * Featured on Meta * Seeking feedback on tags update * We're rolling back the changes to the Acceptable Use Policy (AUP) * Design Vision Updates to the Left Navigation Related 130 Asked again to review a paper, when the authors don't wish to modify it 41 How to respond to a perplexing review? 3 Resubmission - Dealing with rude reviewer 16 Should I withdraw a paper if the Editor-in-Chief doesn't like it, but the reviewers think it's great? 24 How to handle a possibly biased editor? 36 Should I point out a flaw in a paper, which will likely result in withdrawal and resubmission, before I start to write the review? 20 Journal that rejected paper to disclose identity of referee to the next journal? 3 Article review: to decline or not decline 16 How to handle reviewing a paper which I suspect has been generated by AI? Hot Network Questions * Retroactive lack of testamentary capacity * What's the largest string OpenSSH can expand from a % token? * Is high-carbs consumption during Z2 rides efficient for weght-loss? * How to separate superglued fingers * Creating horizontal lines connecting points using QGIS Geometry generator * What stops stolen software from offering a valid license? * How to request to stop communication? * Is there any interest in eating a baked donut on Chanuka? * Why is it the indefinite article in 'training of an older generation'? Shouldn't it be 'the' here? * 15 year old crossing the Germany/Netherlands border - do I need parental letter? * Is logistic regression suitable for high frequency values? * How would the government cover up a nuclear war? * What is the specific name of this distribution? * Why does a plastic bag rip when it is raised to a higher position when being held but not when it is stationary? * What happens to non-cooperative shareholders when a company goes private? * Questionable peer review practice in top-tier journal * How does "innate magical power" fit into a magic system powered by emotion and memory? * Cocoa vs cacao pate vs 100% dark chocolate * Why is the time domain low-pass filter the "sinc" shape? * When has the scaffolding been more important than the completed building? * The Emotional Complexity in the 'Oppenheimer' Atomic Bomb Test Scene * Was there any computer since about 1960 without interrupt support? * Means of pretending to be Dominated, thralled, or Charmed * How do I start a priority dispute? more hot questions Question feed Subscribe to RSS Question feed To subscribe to this RSS feed, copy and paste this URL into your RSS reader. [https://academia.sta] * Academia * Tour * Help * Chat * Contact * Feedback Company * Stack Overflow * Teams * Advertising * Collectives * Talent * About * Press * Legal * Privacy Policy * Terms of Service * Cookie Settings * Cookie Policy Stack Exchange Network * Technology * Culture & recreation * Life & arts * Science * Professional * Business * API * Data * Blog * Facebook * Twitter * LinkedIn * Instagram Site design / logo (c) 2023 Stack Exchange Inc; user contributions licensed under CC BY-SA. rev 2023.11.29.1725 Your privacy By clicking "Accept all cookies", you agree Stack Exchange can store cookies on your device and disclose information in accordance with our Cookie Policy. Accept all cookies Necessary cookies only Customize settings