This is what you should do to create better A/B tests

You might think you already know what a good A/B test is: just try out a couple of headlines and find out which one generates the most clicks. But there is more. Less is more, in fact.

Almost all big news organisations use A/B testing tools to optimise their content strategy. But testing which headline your audience fancies the most isn’t just about generating more clicks. With smartocto Tentacles you can also test loyalty.

To find out how to do a good A/B test, we spoke to Ivan Zrinjski, who specialises in doing this at the online newsroom at the Croatian newspaper, Jutarnji list (part of Hanza Media).

Check out this webinar about A/B testing and how AI can help

Three tips on A/B testing

Tip 1: Come up with just one extra headline
“You can think about the headline when you’re proofreading your article. If you create too many alternatives, there won’t be a clear winner. It may even overload the system.”

Tip 2: Ask a colleague to suggest another one
“Other editors bring a fresh perspective to your story. They don’t need to provide a perfectly polished alternative, but they can point you in a new direction.”

Tip 3: If that’s tricky, start by changing a single word
If you don’t have another headline ready, simply change one word. “If we have a background article about the war in Ukraine, we interview experts,” says Ivan Zrinjski. “But we found that the word expert in the headline wasn’t the best choice for our readers. The word analyst worked far better. By replacing just one word, you can already make a difference.”

Why less is more in A/B testing, technically speaking

Basically, the clue’s in the name. It’s not A/B/C/D/E/F testing, after all. Smartocto’s data scientist Goran Milovanovic: “First of all you’re talking about different headlines for the same article. The more headlines you would like to test, the more creativity you need - but the angle of the article doesn’t change.” In Milovanovic’ opinion the headline needs to fit the angle of the story. “To have so many headlines is almost impossible. I think the quality of the headline would suffer. Especially because editors need to share their focus with other tasks.”

Besides, and perhaps this is a more important argument, there probably aren’t enough readers to make a statistically significant difference when it comes to four, five or six headlines. “The system will randomly show different headlines to different visitors. It’ll take more time for the system to truly say that A is better performing than B.”

Pop Quiz

Which headline scored best on conversion and engagement?

a. Perpetrator of fatal stabbing incident chose victim at random, police suspect

b. Police suspect victim stabbed to death was chosen at random

Join us in our Smartoctober webinar on 24 October (14:00 Central European Time) to discuss this, and much more besides.

Other things to think about while doing an A/B test:

• Make use of emotive words. If there’s emotion in the headline, more people are eager to read the story. The audience would like to sympathise. Example: Prince Harry devastated the Queen's Initials were removed from his uniform.
• Don’t use question marks too much. It’s better to give the answer. ‘This is what a good A/B test looks like’ works better than ‘what does a good A/B test look like?’
• Use pronouns to speak to your audience. We can improve the example of the previous tip: ‘This is what you should do to make better A/B tests’.
• Be super concrete and detailed. Not: Man dies after car crash, but: ‘Juan (44) died after a bridge collapsed on his car in Madrid’.