This is what you should do to create better A/B tests

You might think you already know what a good A/B test is: just try out a couple of headlines and find out which one generates the most clicks. But there is more. Less is more, in fact.

Almost all big news organisations use A/B testing tools to optimise their content strategy. But testing which headline your audience fancies the most isn’t just about generating more clicks. With smartocto Tentacles you can also test loyalty.

To find out how to do a good A/B test, we spoke to Ivan Zrinjski, who specialises in doing this at the online newsroom at the Croatian newspaper, Jutarnji list (part of Hanza Media).

Check out this webinar about A/B testing and how AI can help

Four tips on A/B testing

Tip 1: Come up with just one extra headline.

“You can think about the headline when you are reading your own article for a spelling check. If you have more than three alternatives, there won’t be a clear winner. It may overload the system.”

Tip 2: Ask a colleague to come up with another one.

“Other editors bring a new perspective to your story. They don’t need to formulate a second alternative precisely, but they can give you a new direction.”

Tip 3: Decide if you should use those three headlines to compare impact or leave one that is not good enough.

“Doing A/B tests is also about time management. Go with two titles max, maybe three, so that you don’t lose too much time and affect your other editorial duties.”

Tip 4: If that’s tricky, start by changing a single word

If you don’t have any alternative headline to work with, you can also add a headline with just one word changed. “If we have a background article about the war in Ukraine, we speak to experts”, Ivan Zrinjski says. “But we found out that the word ‘expert’ in the headline is not the best word for our audience. The word ‘analyst’ is way more popular. By replacing just one word, it can already make a difference.”

Why less is more in A/B testing, technically speaking

Basically, the clue’s in the name. It’s not A/B/C/D/E/F testing, after all. Smartocto’s data scientist Goran Milovanovic: “First of all you’re talking about different headlines for the same article. The more headlines you would like to test, the more creativity you need - but the angle of the article doesn’t change.” In Milovanovic’ opinion the headline needs to fit the angle of the story. “To have so many headlines is almost impossible. I think the quality of the headline would suffer. Especially because editors need to share their focus with other tasks.”

Besides, and perhaps this is a more important argument, there probably aren’t enough readers to make a statistically significant difference when it comes to four, five or six headlines. “The system will randomly show different headlines to different visitors. It’ll take more time for the system to truly say that A is better performing than B.”

Pop Quiz

Which headline scored best on conversion and engagement?

a. Perpetrator of fatal stabbing incident chose victim at random, police suspect

b. Police suspect victim stabbed to death was chosen at random

Join us in our Smartoctober webinar on 24 October (14:00 Central European Time) to discuss this, and much more besides.

Other things to think about while doing an A/B test:

• Make use of emotive words. If there’s emotion in the headline, more people are eager to read the story. The audience would like to sympathise. Example: Prince Harry devastated the Queen's Initials were removed from his uniform.
• Don’t use question marks too much. It’s better to give the answer. ‘This is what a good A/B test looks like’ works better than ‘what does a good A/B test look like?’
• Use pronouns to speak to your audience. We can improve the example of the previous tip: ‘This is what you should do to make better A/B tests’.
• Be super concrete and detailed. Not: Man dies after car crash, but: ‘Juan (44) died after a bridge collapsed on his car in Madrid’.