AI vs. Humans – SEO Experiment

Published by Liz Sena on Wednesday, October 18, 2023 in Ai.

Artificial Intelligence (AI) has and will continue to reshape nearly every industry, and marketing is no exception. Since the release of ChatGPT, OpenAI’s Natural Language Processing (NLP) model, massive shifts in business have occurred. Like many marketers, you’re probably worrying about how this progressively improving technology fits into a world run and maintained by human content. If ‘machines’ can write as well as humans, what does that mean for creativity, authenticity, and Search Engine Optimization (SEO)? It’s a fair question, especially because SEO plays a significant role in online content that's readily viewable. Yet, it (SEO) remains relatively unregulated.

Marketers have already taken the dive, with some content marketing experts finding that 82% of marketers claim AI content is "just as good as human content." 

A more important question worth exploring is, “Where does Google stand in managing AI's exploding popularity and mass production of AI-generated content?” How do search engines handle the ranking of human-generated content vs. AI-generated content? 

Reboot, an online marketing company in the U.K. known for its industry-changing experiments, set its sights on answering these questions. 

Before we dive into the experiment, let’s take a look at the components.

Search Engine Optimization (SEO)

Search Engine Optimization is the technology behind what determines which content shows up on search engine results pages (SERPs). As such, it has become a key phrase in a growing empire built on keywords and phrases. Online businesses now focus on using this science to optimize content to push their websites, articles, or blog posts, landing pages, and other content to the top of the search engine results pages (SERPs). 

And how do they do this? It’s all in:

• Identifying and using trending keywords
• Meticulous content structuring and formatting
• Optimizing metadata like titles and descriptions

With AI becoming increasingly advanced, top-notch SEO content is becoming easier to produce. Marketers can provide prompts for AI programs to:

• Optimize an article with the right keywords and structure
• Implement SEO practices into a pre-written article
• Write an entire SEO-focused piece of content in a matter of seconds

All this, within seconds, can significantly decrease the time spent on human-written content while minimizing overhead costs.

AI-Generated Content

Natural Language Processing (NLP) models have hit the ground running with advanced conversational chatbots like ChatGPT and Microsoft Bing AI, both capable of producing hundreds of lines of text in seconds and near-instantaneous revisions.

This software works by taking in a prompt from the user and feeding it into an AI program trained on large amounts of text gathered from internet sources, books, magazines, etc. 

The AI predicts the next most likely word in a sequence, over and over, via an internal ranking system. While this sounds basic, the amount of data used to train it allows it to produce coherent, human-sounding text in seconds with incredibly high accuracy and precision.

Essentially, pattern recognition is what powers AI content, not understanding.

Still, it’s fast, polished, and grammatically clean, which explains why it’s become a go-to tool for 90 percent of marketers and copywriters, as revealed in a survey. So if you need a product description, a blog post, or even a short story, AI can produce it in a matter of seconds.

But that raises a big question: Does Google treat AI-written content differently? Let’s find out.

Google’s Stance on AI-generated Content

Google's position on AI-generated content states clearly that content is at the forefront of the company’s focus, whether human- or AI-generated. Google developers also state that AI content isn’t ranked in any particular way. It's rated with the same paradigm as any other type of content: “Useful, helpful, original, and satisfies aspects of E-E-A-T.” Whether human or AI-generated, the "winning" content will rank highest according to E-E-A-T.

Worth noting is that content created by AI has been around for a long time. For instance, tools like Grammarly and Google Translate, which leverage AI, have been available to writers and readers to improve writing quality and accessibility. Recently, Google went ahead to integrate generative AI directly into Gmail, Docs, and Workspace, with features such as “Help Me Write.”

So, in other words, if the content provides real value and meets Google’s quality standards, it can rank, no matter who (or what) created it.

But how truthful is this claim? Does Google’s algorithm walk the talk? That question was the basis for Reboot’s experiment. Through deep research, statistical testing, and articulate reporting and discussion, Reboot sought to answer that very question. 

AI vs. Humans: The Reboot Experiment

The experiment began in the early months of 2023 with the hypothesis that Google ranked AI- and human-generated content differently on its search engine results. To ensure a fair test, the Reboot team created a controlled environment where all aspects of their content were the same, except for who created it: AI vs. human. 

The researchers used ChatGPT to create eleven meaningless words with no search history or domain existence. They chose "flemparooni" as the keyword for the content. 

Then, using the remaining ten words, they developed and deployed ten basic websites to host the content articles.

They began the writing process after outlining a 7-step guideline to promote replicability and equal optimization across all content. 

Five of their company’s in-house content writers each wrote an article based on the keyword "flemparooni" and the guidelines provided. 

The researchers then used ChatGPT-4, the latest version of OpenAI’s ChatGPT, to feed the keyword, the guidelines, and the prompts needed to output five articles meeting the specified criteria. 

Finally, the team analyzed data based on information collected through keyword position tracking. They also performed tests to determine the significance of the results.

AI vs. Humans: The Results

When the results came in, the findings were telling. Reboot found that of the ten articles created (five by humans and five by ChatGPT-4):

• Four of the top-five ranked articles were human-generated.
• The only AI-generated article in the top-five group ranked third place.
• Four of the bottom five ranked articles were AI-generated.
• The only human-written article in the bottom group ranked ninth.

The researchers concluded that AI-generated content held an average ranking of 6.6, whereas human-written content held an average ranking of 4.4, showing that content created by humans tended to outrank content created by AI. Data is only as good as its significance, however, and this is where the word "Science" in "Data Science" comes in.

So, the researchers didn’t stop there. To prove that the difference was just a coincidence, they decided to run an additional test. They used the Mann-Whitney U test, which is a common tool for determining whether two groups’ rankings differ significantly.

Reboot researchers tested the data using the Mann-Whitney U test. 

Omitting the complex mathematical mumbo-jumbo, the Mann-Whitney test pits the numerical data against a null hypothesis, in this case, that the probability of AI- and human-generated content rank equally. After some behind-the-scenes calculations, the test returns a number indicative of the difference in mediums (or middle numbers) between the two populations (AI-generated vs. human-generated content). 

A higher result indicates websites hosting human-generated articles rank higher than their AI counterparts, and a lower result indicates that AI-generated articles rank higher. 

After a total of 25 comparisons, the test rejected the null hypothesis in about 92% of cases, telling us that the results of the experiment were significant. This means that the results they found didn't happen by chance.

The math behind it is complex, but the outcome was simple: in about 92% of the time, the test found that the two weren’t ranked equally. Furthermore, the results showed that ranking tended to favor human-generated content over AI content, assuming total SEO equality throughout all content.

Conclusion

The results of this experiment are interesting, especially when considering the implications. 

SEO-based companies and marketers focusing on ChatGPT and other AI marketing tools to maximize content optimization may save on the overhead costs involved in human content creation. But this experiment shows their competitors may maintain an edge if they stick with human-generated content.

A theory is only as good as its replicability, of course, and research should continue for more in-depth confirmation. However, the results are a good outlook for human content creators and should be a flag for all those online marketing decision-makers pushing for total content automation.

The answer lies somewhere between human and AI content generation. 

But one should not supersede the other. Instead, marketers should utilize both in harmony to promote engaging, human-to-human knowledge and content while maximizing efficiency through AI ideation, outlining, and keyword generation, among other tasks.

Experiments like Reboot's show that we're not ready for complete automation, and companies that think we are will likely do themselves more harm than good.

Humans and AI Should Work Together for You

Your content must resonate with real people (your target audience) while satisfying a complex algorithm. Unfortunately, AI can't do it alone, and human creators can only do so much in limited amounts of time. 

This is the very problem Scripted’s founders wanted to solve. Scripted is the only platform offering both human and AI content. We unify the two in a way that maximizes speed and efficiency while providing high-quality, branded content that stands out. 

With a pool of carefully selected subject matter experts (SMEs) with industry-specific knowledge and a one-stop platform for all your content creation needs, marketers need look no further. Ready to see the difference a human-first, AI-empowered content strategy can make? Get started today.