AI Detectors Don’t Work
1. Introduction
I still remember the feeling I had when I watched ChatGPT write an essay for the first time. The best way to explain it was equal parts wonder and dread.
Like many HASS teachers, written tasks formed the backbone of my assessment.
What did this mean for essays, reports, and posters? What did this mean for take-home assignments?
Thankfully, after some frenzied research, I stumbled upon a well-known AI detector.
Problem solved.
I generated some text with ChatGPT and it flagged as a high probability of AI generation. Great.
Then, summoning what little scientific rigour has remained from university, I copied in a blog post I wrote a few years ago. It also flagged as a high probability of AI generation. Not so great.
My equal parts wonder and dread turned to mostly dread.
2. What is AI Detection?
Anyone who’s watched CSI knows that if you want to catch a killer, you dust for prints.
Here’s the problem with AI detection: AI has no fingerprints.
Instead of fingerprints, AI detectors examine the text for certain traits.
Essentially, they look for two things.
Firstly, they look for perplexity – a measure of how predictable writing is. Human writing is more unpredictable than AI-generated content, resulting in a higher perplexity score. The lower the perplexity score, the more likely text is AI-generated.
Secondly, these tools look for burstiness – a measure of sentence length variation. This is like perplexity, but examined at the scale of sentences rather than words.
So AI doesn’t leave fingerprints. Sure, it has certain stylistic similarities, but these can easily be mirrored by human authors.
There have been some high-profile false-positives over the last 12 months. Below is my favourite:
Would fingerprint evidence be so damning if there was a 45% chance it could still be someone else?
Next week we’ll look at how to handle academic honesty in the AI age, but first, you have a homework task!
3. Your Homework Task
As the old saying goes, you can’t learn to ride a bike at a seminar.
Some things, you just have to learn by doing.
What better way to learn about AI detection than by trying to beat the system?
(If you need some help, there is an article in the resources section about how to beat AI detectors.)
Your mission:
Log onto ChatGPT, Bard, BingAI, or any otherLLM.
Generate generic text (“write me a paragraph about___”)
Open up an AI detection tool like Scribbr https://www.scribbr.com/ai-detector/or ZeroGPT https://www.zerogpt.com/
Paste in your text, see results.
Iterate your text in the AI tool (“vary sentence length”, “use less formal language”, “re-write at a grade 5 English level”)
Reflect on the results of your experiment. What changed in your understanding of AI detectors? What stayed the same?
If you’re interested in exploring how to encourage wise use of AI amongst your students, I’ve got just the resource for you.
Last year I put together a small eBook called AI: Cheater or Tutor, and you can get it for free below:
