Content performance comparison: results from a human vs. AI content marketing experiment

Results breakdown

Using Google Search Console (GSC), our custom Organic Search dashboard in Tableau, and Hotjar Heatmaps and Feedback, we analyzed quantitative and qualitative metrics for our human and AI content pieces. Below is a breakdown of both articles’ performance across three categories.

1. SEO metrics

The human article outperformed its competitor across multiple SEO metrics:

It peaked at 71 clicks on November 11 and, from October to the experiment's conclusion in December, maintained a healthy average of around 34 clicks per day. Our AI piece, comparatively, took an immediate post-launch nosedive, then plateaued, peaking on August 2 with five clicks.

What’s exciting is our human piece saw a steady increase in clicks over time despite several months of AI-induced upheaval in the SEO industry that saw major events like the introduction of Google’s SGE and Gemini. Even with the odds stacked against it from the start, it performed exactly as we hoped it would.

2. Internal performance metrics

We used our custom Organic Search dashboard in Tableau to determine if either piece contributed to our internal metrics. As mentioned in our original experiment write-up, we didn’t foresee movement here because the topic we selected—the impact of AI on various industries—is irrelevant to our ideal customer profile (ICP). 

And yet…

Imagine our surprise at seeing our human piece featured in July’s GSC performance report, an imposter among two other topics very much targeted to our ICP:

Our dashboard revealed two more pleasant surprises:

Of the 4,550 people who clicked on our human piece, 93% of them were new visitors to Hotjar.com (welcome! 👋). But even more importantly, we got three signups—three brand-new Hotjar users—from a piece of (very) top-of-funnel content that wasn’t even created with our ideal audience in mind. 

3. Reader sentiment and behavior

Finally, and perhaps most importantly, we compare audience sentiment and behavior across both pieces using Hotjar Heatmaps and Feedback.

Scroll maps—a type of heatmap in Hotjar—use a color gradient to represent the most and least viewed parts of a page. Red indicates the areas of a page users see the most; blue represents little to no customer interaction.

Scroll maps of both pieces show that the AI piece (right) loses readers’ attention significantly earlier than its human counterpart: the gradient changes from green to blue just a few paragraphs in, while the human piece retains interest for longer.

Readers’ qualitative feedback further reinforced our quantitative scroll map results:

However, not every reader agreed that the human piece was a clear winner:

Others mentioned that both articles have their strengths and weaknesses, depending on factors related to personal content preferences or subject familiarity.

Of the total feedback we received, this was the sentiment breakdown:

And the winner is…

Well, it’s not so simple, even if it looks simple.

At face value, our human piece outperformed—nay, totally annihilated—the AI version in every category.

It would be easy to give our writer the trophy and thank her for single-handedly saving millions of content marketing careers. But even though these results definitely mean something, they were always going to be imperfect.

It’s worth acknowledging, as many readers already have, that hundreds of variables affect these outcomes: maybe if we’d used the paid version of ChatGPT, maybe if we’d spent more time refining the AI article, maybe if the topic were different, maybe if we’d masked the experiment, maybe if our prompts were better, maybe if we’d used a different AI tool, maybe, maybe, maybe.

Then, there’s our own bias: we’re content marketers and we’re nervous about the future; we want to believe the work we do is unique and irreplaceable. Did we unintentionally sabotage ChatGPT from the very beginning? Possibly. 

There’s also one more critical factor worth considering:

We probably don’t feel the same way we did six months ago

Over the past few months, ChatGPT has become our unofficial right-hand robot, a permanent tab in our browser, and we understand its applications for our jobs a lot more than we did in June. 

There’s still absolutely no chance we’d use it for product-led content writing and editing, but there are many ways AI tools make other, more tedious aspects of our jobs easier. Our internal team of content marketing managers, editors, SEO specialists, and team leads have dozens of use cases between us for tools like ChatGPT, GPT-4, Hotjar AI for Surveys, Jasper AI, and YouTube Summarizer. Here are just a few:

Heck, even the writer of our human piece uses GPT-4 to develop angles for her main topic and subtopics, write FAQ sections, and shorten lengthy sentences. (Plus: rumor has it our Editorial team was actually spied suggesting more ways for our writers to use AI 👀—something that seemed pearl–clutchingly unthinkable back in June 2023.)

Will AI replace human writers?

No—but the takeaway from our experiment is not that AI sucks and people are cool. As everyone reading this has probably already learned for themselves, it’s fantastic for some use cases and terrible for others, just like any reliable tool in your stack.

Ultimately, we hope this experiment has accurately outlined

• The differences between working with a human writer vs. ChatGPT

• Real people’s perceptions of 100% human content and AI-assisted content

• The pros and cons of human and AI-assisted content production

One final thing we’ve learned is that the content marketing landscape is not the same as it was a couple of years ago. AI has upended our workflows, probably forever—but is that a bad thing? Maybe not.

What have you discovered about AI over the past six months? Let us know using the Hotjar Feedback widget—it’s that red tag to the right of the page. 👉