An AI Detection Tool Wants to Provide ‘Human’ Feedback on Stories, and We Tested It | Exclusive

GPTZero, which can detect whether a TV script or college paper was generated with AI, is branching out with a new feature that offers writers AI-generated feedback trained by human editors.

In other words, the AI tool can mimic the editing skills of a real person who is willing to train it.

In kicking off this initiative, GPTZero partnered with Greg Altman, an Emmy-nominated TV producer; Bill Retherford, a journalist and 4-time Emmy winning documentarian; and Wendy Snyder, a college admission officer at Northeastern University. All three created “digital doubles” that they helped prompt and calibrate and can provide the kind of feedback they would’ve offered had they actually read your work.

The announcement comes on the heels of another AI service, Quilty, which last week promised to provide AI-generated creative and business feedback and analysis for Hollywood scripts — which we found yielded mixed results, and is part of a broader wave of AI assistive tools that have cropped up. But while Quilty’s AI model is a bit of a black box, with the fine-tuning done with a proprietary algorithm, GPTZero is trying to stand out by embracing the “human element” of AI feedback with the digital doubles.

“Every expert evaluates writing against criteria they’ve developed over years,” GPTZero CEO Edward Tian told TheWrap. “What makes a good cold open, what makes dialogue feel authentic. We work with them to capture that metric and apply it consistently.”

As Tian puts it, GPTZero comes at the AI phenomenon from a different direction, since it got its start as a service looking to weed out AI-generated content from written work. And AI detection has become big business; Tian said GPTZero has 17 million registered users and 800,000 daily active users submitting everything from book reports to scripts to see if there’s any AI content. Last month, book publisher Hachette canceled Mia Ballard’s “Shy Girl” after claims arose that AI detectors found much of her work was generated by AI.

But there’s a big difference between spotting what’s generated by AI and offering thoughtful feedback on everything from a screenplay to a TV script.

Which is why I built my own digital double.

How it works

After creating an account in GPTZero, I was able to create a custom reviewer by writing a short description of the kind of editor I wanted to be. In my case, I wanted my double to edit feature news stories with a focus on a compelling headline and introduction, a strong angle that’s backed up with reporting, the context to explain the relevance of a story, and clear and concise language.

I would then upload examples of raw copy stories that I would “edit” to my preference in an effort to calibrate the settings and to make it more like, well, me.

The early examples of feedback remained rough. It didn’t quite catch the core thematic paragraphs (i.e. “nut grafs”) in a story, although it helped with trimming down some of the copy. There were too many suggested edits to quotes, which you typically don’t touch.

But I acknowledge that I was still early in the process, and hadn’t really put in the time to really train the reviewer with my preferences.

Would I use it as a way to speed up the editing process? No. But I could see a scenario where writers looking for my edits could potentially use it as a source for early feedback. Although not until I really, really fine tune the model.

More digital doubles to come

While GPTZero is highlighting the three initial professionals who created a digital double, Tien said the company is working with larger agencies to get introductions to scriptwriters, producers and other professionals to expand the roster.

GPTZero charges you “credits” that you need to pay for each time you run a scan and get feedback, bt the company is still working on a monetization model for the humans whose doubles get tapped for feedback.

But ultimately, even if you create a double, as I did, you don’t have to make it public. I intend to keep mine private and would only share it with one of my reporters if they expressed interest (which I highly doubt).

“Our growth is community-driven, where editors, script writers, journalists, producers and comedy writers share their editing processes with their community,” Tien said. “It’s not character.ai where everyone wants to talk to Einstein. Writing is deeply personal, where targeted edits from writers that you know and trust are far more valuable than general feedback.”

The post An AI Detection Tool Wants to Provide ‘Human’ Feedback on Stories, and We Tested It | Exclusive appeared first on TheWrap.