What makes visitors to a news site click on one headline but ignore another? A unique collaboration between a university, a publisher, and smartocto has yielded interesting insights.

Testing different headlines for an article is one of the most effective actions a newsroom can take to ensure articles are read more widely. The (loyalty) click-through rate not only highlights the appeal of one particular story over others, but a clear title also sets the right expectations for the target audience. And, once you’ve discovered the best headline option, it’s not just your website that will benefit: it will also be displayed across other channels, such as social media and search engines, giving you a much-needed edge over the competition.

A previous data study by smartocto revealed that winning headlines receive, on average, nearly 20% more clicks than the original, even when including articles where the original headline performed better. Here are the results from an analysis of 9,000 headline tests across 57 media titles, which by the way always takes loyalty into account.

This article was first published on INMA.org. Do you want to see our A/B testing tool in action, ask for a demo.

Get a demo

A collaborative study

The new study was presented last week at the INMA Subscriptions Summit in Amsterdam. The participants:

  • Vrije Universiteit Amsterdam led the research
  • Publisher DPG Media provided access to data from four newspaper titles
  • Smartocto provided the headline testing system called Tentacles

Associate Professor Luuk Lagerwerf of Vrije Universiteit categorised the headlines by assigning values to words used in them and comparing their performance against headlines without those words. “Editors typically rely on their experience to craft headlines with specific effects in mind. I wanted to determine the precise impact and see if we could define it in an academic manner.”

Get more info

How do they do headline testing at DPG Media? Each editorial team has its own approach. In the video below, Tijs Hofmans from the tech website Tweakers (part of DPG Media) explains how he formulates hypotheses before conducting an A/B test.

Key findings

One research question was whether quotes in a headline lead to a higher click-through rate. The answer, according to this study, is no. Similarly, posing a question in a headline does not necessarily yield better results when looking purely at clicks. Another notable finding is that there were differences between the media brands. For instance, ‘personalisation’ worked well for only two of the three regional titles included in the study.

There are also other clearly measurable positive effects, which news publishers can use to their advantage by following a few key recommendations: try to include as much detail as possible in a headline, mention the main characters from the article, and use a narrative headline – one that tells a story. Take, for example, the winning headline in the headline test below (red circle).

Aggregating the results from all four titles, the most significant factor influencing click rates was the presence of negative words in a headline.

The role of negative headlines

“No news is good news” is a well-known saying in journalism. However, data analyst Roy Wassink from DPG Media emphasises that this does not mean journalists should seek out negativity just to make a story perform better. “We also look at other metrics beyond CTR. For example, because we focus on loyalty, we know that too much negative news can result in fewer returning visitors over time. It is also crucial that the headline aligns with the article’s content to achieve better completion rates.”

During his presentation at the Subscriptions Summit, Wassink showed an example where a negative headline did not match the positive lead. The result? Many readers abandoned the article early. A headline, after all, is about expectation management - the expectation set by the headline must be fulfilled in the article itself.

Next steps in research

DPG Media plans to delve deeper into the data to uncover new insights. In the future, smartocto will introduce a feature on the front end of news sites that visualises the balance of different user needs within headlines. Wassink also advocates for a greater focus on loyalty in headline testing. He believes that smartocto’s existing loyalty click rate metric can be further refined and expanded, to make sure that the conclusions can go beyond clicks.