(C) BoingBoing
This story was originally published by BoingBoing and is unaltered.
 . . . . . . . . . .



A university decided to use an AI detector to detect AI cheating. Now it had two problems. [1]

['Rob Beschizza']

Date: 2025-10-09  

A top university in Australia used AI to detect students cheating with AI. Unfortunately, AI detection is snake oil. Of the 6,000 students accused of academic misconduct by the machine, many did nothing wrong.

It was an email from a generic Australian Catholic University (ACU) address titled "Academic Integrity Concern". In it, the university accused her of using artificial intelligence (AI) to cheat on an assignment. "And on top of that, I was getting emails from the academic misconduct board saying I needed to write out an explanation as to why I think this might have happened." It took six months for ACU to clear the 22-year-old of any wrongdoing, but by that point she believes the damage was done.

She was required to prove her innocence, denied her transcripts, and denied a graduate position: "I didn't know what to do. Do I go back and study? Do I just give up and do something that's not nursing in a hospital?"

Because it knows its own AI detection suite doesn't really work,

Turnitin recommends "further scrutiny and human judgement" to determine whether misconduct has occurred. But email chains and documentation obtained by the ABC indicate ACU regularly relied on the report alone. When the paramedic student requested additional evidence, they were sent a copy of their original assessment — identical to the AI report, minus the highlighting. According to internal ACU documents, the university was aware of issues with the AI detector tool for more than a year before it scrapped it.

A funny thing about these tools is that if you search for them at the usual places, all the verifiably human commentary is that it's inaccurate, but all the AI-generated stuff is hype. Those little "AI overview" answers? The links go to press releases put out the AI detection companies and their customers!

If you're wondering why this sort of thing keeps happening, you have to consider the customers: managers and administrators. Their jobs amount to poking around all day in crudely-customized enterprise software, and that goes double for the education sector. They are sunk as deep as it gets in the sketchy tech of procurement culture and always eager to buy another feature. They'll be shaking their heads in uncomprehending denial at critics right up to the point they get out-bullied by lawyers or some other threat. Even then, it's bullet points all the way down.

[END]
---
[1] Url: https://boingboing.net/2025/10/09/a-university-decided-to-use-an-ai-detector-to-detect-ai-cheating-then-it-had-two-problems.html

Published and (C) by BoingBoing
Content appears here under this condition or license: Creative Commons BY-NC-SA 3.0.

via Magical.Fish Gopher News Feeds:
gopher://magical.fish/1/feeds/news/boingboing/