If you assume spotting AI writing comes naturally, fresh evidence indicates most people overestimate that skill. Scholars at Villanova University discovered that readers had difficulty separating machine-produced stories from those penned by humans and frequently assigned higher scores to the AI pieces.

The project, headed by Dr. Deena Skolnick Weisberg and appearing in Judgment and Decision Making via The Guardian, involved more than 1,600 volunteers who evaluated one of six brief narratives. Three came from a person and three from ChatGPT. Each person received a stated author for their assigned tale, yet the label was sometimes incorrect.

AI versions earned stronger marks while the human label added appeal

Those who received an AI story judged it more engaging and better crafted than those who read a human story. Meanwhile, any tale presented as human-written received better reviews than those labeled AI, no matter the real source.

Weisberg notes the findings indicate AI now produces fluent and compelling fiction beyond common expectations. She stresses this does not render human writers unnecessary.

Most participants could not detect the source

In a second round, volunteers received one human story and one AI story on matching themes and had to name the origins. Success rates stayed near chance levels. Only those who reported greater experience with AI tools performed noticeably better, implying detection improves with practice rather than intuition.

The Commonwealth Short Story Prize faced a similar issue earlier this year when online readers questioned three of five winning entries as AI-made and an automated checker supported the claim. The publisher conducted its own review and reached no firm conclusion. When experienced judges struggle to decide, casual readers face an even greater challenge.