In January ‘23 the “AI Classifier” market was effectively launched with the introduction of GPTzero via a Tweet. The site was a simple form created by Edward Tian, a student at Princeton University where you could paste text, click submit, and get a quick analysis of a few metrics you probably never heard of: burstiness and perplexity. That tweet has since garnered over seven million views and 1000s of shares, effectively creating a new category of academic integrity tool as a way to identify AI generated text from ChatGPT (and now other generative AI tools) which had been launched less than two months prior.
While GPTzero was not the first classifier, it became one of the most well known in a short time followed by plagiarism detection incumbents like Turnitin and Copyleaks a few months later, and opening the door to additional services to launch purporting the same analysis and detection tools.
What is an AI Classifier?
AI classifiers are essentially groups and collections of text analysis tools working at the same time. Text analysis and writing analytics have been researched for decades. Writing by humans can be classified and quantified in many ways, from simple readability scores which take into account sentence length, word length, and syllables per word to more complex analysis for stylometrics and linguistic analysis.
Modern classifiers focus on features that can be extracted from known human written language and large collections of text (similar to the texts that are used to train large language models). Through this analysis we can statistically infer tone, word frequency and predict the order and flow of words in a sentence.
Two important analysis techniques tie most (if not all) AI classifiers together: perplexity and burstiness.
- “Perplexity…is one of the most common metrics for evaluating language models. It is defined as the exponentiated average negative log-likelihood of a sequence, calculated with exponent base e.” It might be best reflected using the equation:1
- “Burstiness basically measures how predictable a piece of content is by the homogeneity of the length and structure of sentences throughout the text. In some ways, burstiness is to phrases what perplexity is to words.”2
Many commercial models have their own secret sauce and additional models that help to tune their output.
What’s important to know is that each is using Machine Learning and Artificial Intelligence to identify the Artificial Intelligence-ness of text you input to it, providing back to the user that data along their own confidence intervals, gradients, spectrums, or binaries. These outputs are not whether or not it is or isn’t AI. Rather, most state in their disclaimers and end user license agreements that the output scores, percentages, or graphics represent the statistical analysis of whether that text matches their model of AI generated text, or not and can be misclassified due to “edge cases” and other phenomenon.
What are the best known AI Classifiers and what differentiates them?
| Name | Pricing | Free access | API | Plagiarism detection | Other features |
| GPTzero | $10/mo (teacher) | ☑️ | ☑️ | ☑️ (premium) | |
| Turnitin | ~$4+FTE3/ year | 🟪 | 🟪 | ☑️ | Proctoring |
| Copyleaks | ~$4+ FTE/year | ☑️ | ☑️ | ☑️ | |
| Originality.AI | $.01 per 100 words | ☑️ (account required) | ☑️ | ☑️ | Fact Checking |
| Winston.ai | $12/mo (teacher) | ☑️ (account required) | ☑️ | ☑️ | |
| Crossplag | €9.95/mo (teacher) | ☑️ (account required) | 🟪 | ☑️ |
Can I use an AI Classifier for Free?
Many of the most well known AI classifiers offer some level of free access where you can paste a limited text and get an output.
We don’t have any affiliation with, but we do love Originality.AI for its flexibility, pricing and ability to leverage plagiarism detection in a single swoop and some of the most transparent pricing available. If you like them as much as we do, you might use our referral link:
Do AI Classifiers work?
This is a difficult question to answer. AI classification is a science, and it is possible to classify text as likely human or likely AI generated along a spectrum. Many AI classifiers are pretty good at identifying unadulterated AI generated text as AI generated. Anecdotally, you can prompt ChatGPT or another Generative AI, copy that text, paste it into an AI classifier and see “100% AI.”
In that case the answer is “yes.”
However, since the introduction of these new tools, researchers, educators, and students all have worked hard to test and in many cases create exploits, defeat strategies, and studies that pit classifiers against each other and various techniques to elicit a false negative or false positive outcome. Techniques to defeat classifiers include re-prompting, paid and free paraphrasing and humanizing tools,
One popular article by researchers from researchers at the University of Maryland point out the acceleration of AI development and the inability (overtime) of detection tools to keep pace and outline their paraphrasing technique defeating available classifiers4. Along similar research, a March 2024 paper5 discusses these and other techniques while also calling attention to the impact of detection tools on non-native English speakers (NNES) and of false positives on any student facing an accusation.
At Cursive, we actively test, demo, and use various academic integrity tools to keep current. We participated in a non-academic research study in mid-2023 with 30 different sample texts that had been created in various ways, all AI generated and found that simple techniques (including prompting) could be used to bypass detection.
In this experiment we tested 3 classifiers (including OpenAI, Turnitin, and GPTzero) with three bypass techniques.
Along those lines, the answer clearly is “no,” right?
AI classification is a difficult task, made more challenging by the rapid advancement of generative AI in short time and the various macroeconomic forces that incentivize bypassing (not just in academic integrity to earn a grade, but also in search engine optimization (SEO) and content marketing to increase web traffic and sales). These tools, like all detection tools, are not a silver bullet to identifying AI generated content, but they do provide additional context and remain a viable and useful tool in evaluating text (similar to revision history, plagiarism detection, and even stylometric data that can aid in verifying student work).
Conclusion
AI generated content has proliferated across the web in ways we may not yet fully comprehend. In educational settings, we sympathize with teachers who receive AI generated content intentionally submitted in lieu of student created text. “Catching” a student is never fun but can often create opportunities for learning.
Our recommendation for educators in academic integrity questions is that these provide a useful, but not alone conclusive, datapoint for starting a discussion with students about their work.
Good writing is an active process and effort beyond prompting a chatbot is always required. Any insights into time, editing, revision history and other analytics (such as those created by Cursive) can go a long way in improving confidence in student writing adding additional data and transparency to the writing process.
- “Perplexity” https://huggingface.co/docs/transformers/perplexity ↩︎
- The Dummy Guide to ‘Perplexity’ and ‘Burstiness’ in AI-generated content https://medium.com/the-generator/the-dummy-guide-to-perplexity-and-burstiness-in-ai-generated-content-1b4cb31e5a81 ↩︎
- FTE = Full time equivalent. In contractual language this equates to “an enrolled and active student.” ↩︎
- Can AI-Generated Text be Reliably Detected? https://arxiv.org/pdf/2303.11156.pdf ↩︎
- GenAI Detection Tools, Adversarial Techniques And Implications For Inclusivity In Higher Education https://arxiv.org/ftp/arxiv/papers/2403/2403.19148.pdf ↩︎