Did a robot write this post?

You can now use AI to detect AI content -- and to fool the AI content detectors.

The ongoing tragedy of robot-on-robot violence. Source: Midjourney.

Oh, great, just what the world needed – another innovation poised to shove human creativity into the abyss of obsolescence. If you thought spell checkers were the beginning of the end for proper spelling and grammar, let me introduce you to the poster child of digital decadence: generative AI in the writing profession. Gather 'round, dear readers, as we explore how this digital devil in a bytes disguise is single-handedly – or should I say, single-algorithmically – wrecking the once hallowed halls of the writing world.

Remember the days when writers honed their craft through years of rejection, gallons of coffee, and a mountain of crumpled paper? Well, kiss that romantic notion goodbye, because now anyone with a keyboard and a halfway decent dataset can be the next Shakespeare – or at least, a cheaper knockoff version. Generative AI has rolled up its sleeves, or should I say, lines of code, and proclaimed, "Why should writers suffer for their art when a machine can suffer in their stead?"

Does anything in those last two paragraphs strike you as a bit off? Like maybe Dan has drunk way too much coffee this morning and/or needs to tamp down the meds?

I certainly hope so. Because they weren't written by me, they were written by ChatGPT pretending to be me. [1] And if the robot's imitation fooled you, I'm in a lot of trouble.

The emergence of AI tools that can write like humans has caused a lot of angst among both people who do this sort of thing for a living and those whose job it is to evaluate writing (like teachers). Since these tools exploded into our collective consciousness last fall, there have been dozens of stories about students passing off ChatGPT essays as their own, news sites publishing AI-generated stories without informing their readers, and even lawyers using it to write legal briefs that reference fictional case law in their arguments. [2]

ChatGPT fakery has become so prevalent that there's an emerging cottage industry of tools that use AI to detect stuff created by AI. And now, there are even newer AI tools designed to fool the AI fake detectors into believing the AI fakes are real.

Welcome to the absurd predicament in which we now find ourselves. [3]

Never wake a sleeping cat

A common mantra that's emerged over the last few months goes like this: "You need AI to detect/manage/combat AI." Well, maybe.

When I started looking into this I was kind of shocked how many tools have popped up that claim to sniff out content generated by ChatGPT, Google Bard, Bing Chat, and their various robot cousins. There are apparently dozens of these AI content detectors, and they all claim to be nearly 100 percent accurate.

Here's the thing: ChatGPT et al are essentially just prediction tools; they work by analyzing billions of sentences and learning which words commonly appear next to each other. If you wrote "The cat is..." an AI engine would complete that sentence by predicting the next word would be "sleeping." (Because that's what cats spend 99% of their time doing.)

AI content detectors look for these patterns and flag them. Per Search Engine Land:

The two core concepts around this process are called:

• Burstiness: A predictable length and tempo to sentence structure.

• Perplexity: A randomness to the words chosen in a sentence or collection of sentences.

For example, in an essay about the founding of America, it’s highly unlikely that generative AI would include a random, unevenly written anecdote about the first time they ever saw a penguin, so that would likely look like human writing to a detection tool.

(Note to self: Insert more penguin references in my writing.)

I tried out a half dozen or so of these tools: Originality.ai, Smodin, CopyLeaks, GPTZero, ZeroGPT (a blatant knockoff of the latter). I submitted both AI generated content (like the paragraphs that began this blog post) and stuff I'd written myself, both in this blog and elsewhere. And while my tests weren't exactly comprehensive, the biggest takeaway I have from this exercise is that a) many of these tools aren't worth the paper they're not printed on, and b) they almost never agree with other.

The best one by far was Originality.ai, which is also apparently the market leader. I had a hard time fooling Originality. For example, it nailed that blurb I started this post with, and accurately detected nearly every human and robot-written blurb I threw at it (with one notable exception, which I'll talk about in a bit).

GPTZero, on the other hand, determined that there was a 34 percent chance that the ChatGPT blurb was written by AI, but couldn't identify any sentences that had likely AI content inside. ZeroGPT said there was a 99.97 percent chance that essay was not AI generated. Smodin totally whiffed, saying the text was 100 percent of human origin.

I got similarly screwy results when I submitted articles I had written myself. GPTzero said there was a 25 percent chance an article I recently published in Forbes was written by a robot. (Smodin rated it at 11 percent AI.) ZeroGPT, on the other hand, correctly identified a chunk of text from one of my earlier blog posts as human, but then flagged nearly every sentence in it as "likely" or "possibly" AI generated.

Just cough up some dough, man

Then I decided to try one of the anti-AI content detectors. Undetectable.ai claims that it can take text written by robots and "humanize" it by substituting different words, altering the sentence structure slightly, and even introducing grammatical errors.

For example, it took this ChatGPT content:

Now, everything sounds like it was cranked out by a soulless algorithm that got stuck in an infinite loop of clichés and recycled phrases. Congratulations, literature, you're now an eternal Mad Libs game.

And turned it into this more 'humanized' text, replete with typos:

Nowadays everything seems to be written by an emotionless algorithm that keeps repeating clichés and reused phrases over and over again. Well done literature you've turned into a game of fill, in the blanks.

When I ran the 'humanized' text through Originality.ai, it completely flummoxed the AI sleuth.

I did a second test, using another another ChatGPT version of this essay; Originality.ai flagged that one as 72 percent AI. So Undetectable is at least somewhat detectable.

Why, you might ask, would you want to do this? There are legitimate uses for AI-generated text. These tools are really good at summarizing and explaining basic concepts, for example. And there's also an increasing backlash against anything generated by AI. So I could see a use for more 'humanized' versions of that text, though I can't imagine anyone wanting to publish stuff with obvious errors in it.

Bottom line, though, is that our ability to tell the real from the synthetic is not going to get any easier. AI is not a silver bullet for AI.

Or as Opus would say, Hairy Fishnuts.

Did you detect the AI in this post? Fess up in the comments below.

[1] The specific prompt was "write me a 400 word essay on whether generative AI is ruining the writing profession in the snarky style of Dan Tynan, author of 'Cranky Old Man Yells at Internet'." I also asked it to write the same essay in "the style of technology journalist Dan Tynan," removing the words 'snarky' and 'cranky'. The tone changed slightly, but the end result wasn't that much different. Do I really sound like that? Jesus god, just shoot me now.

[2] The New York lawyers in question got fined $5K by the judge who received the ChatGPT brief. "We made a good faith mistake in failing to believe that a piece of technology could be making up cases out of whole cloth," the lawyers' statement said.

[3] Again raising the existential question, How did we all get trapped inside this timeline, and how the hell can we get out? Where is Stephen J. Hawking when we need him?