User needs and AI: a deep dive in the data bunker

Smartocto recently appointed a Chief Artificial Intelligence Officer. As he and the team have been tasked with integrating generative AI with user needs analysis, we thought it would be a good idea to get Goran S. Milovanović to explain what’s been happening in the data bunker, and how he’s integrating developments in AI with the User Needs Model.

Traditionally, newsrooms have faced challenges in aligning vast content outputs with varied user needs. Our integration of Generative AI into this process marks a pivotal advancement. By deploying generative AI models combined with our in-house, in-depth knowledge base on the User Needs 2.0 model, we've created systems capable of predicting and tagging the User Needs for any given article with remarkable accuracy. The User Needs article tags can subsequently enter various analyses such as being combined with exposure and engagement performance indicators.

And finally, massive, automated User Needs tagging across many articles, combined with traditional editorial analytics, begins to provide insights crucial for any editorial strategy adjustment. Before reading through the rest of this post, you might perhaps want to try out our technology demonstrator, a version of our GenAI driven User Needs system that we keep open for the public to experiment with: the User Needs Playground. For any inputted article, this system will identify the User Needs that the article satisfies, provide an explanation and analysis of the article at hand, and even give advice on possible follow ups.

However, the success in solving the problem of automatic User Needs tagging did not come out of nowhere. This is the story of our struggle with it, our learning, and how we see it now after years of research and development.

The problem of user needs classification

In the almost thirty years that I have been active in research in cognitive sciences, statistics, machine learning, and artificial intelligence, and the more than ten years that I have been exclusively focused on the direct development of systems for predictive and exploratory analytics in Data Science, I have not encountered a problem as challenging as the problem of text categorisation in the User Needs Model.

The requirement is this: for any given text, it is necessary to develop a machine learning system that will tag it according to four axes of the User Needs model: fact-driven, context-driven, emotion-driven, or action-driven. The system should be able to select all tags relevant to the specific article. This is very important because an article that attempts to satisfy multiple axes of the User Needs model simultaneously usually does not target the audience effectively and requires revision.

The reason is quite simple: there’s a lot to be read on the internet, so visitors want a clear view on what they can expect from a piece. If the headline is ‘Thunderstorm this evening’, people might think they are about to get an update about the time and place of that storm, not a background article on how climate change causes more thunderstorms. Editorial focus should lead to a more clear overview for the reader.

NLP (Natural Language Processing) explained in brief

If there's not a clear focus on the writer's intentions, this is a problem par excellence for Natural Language Processing (NLP) approaches in Computational Linguistics - and it’s a tough one.

In the computational analysis of any text, we can clearly distinguish three levels:

• The first level, syntax, relates to the grammatical properties of the text in the original language.
• The second level, semantics, pertains to understanding the meaning conveyed by the text.
• The third level, pragmatics, is very complex.

Once syntax ensures that the text is grammatical, and semantics ensures that the information in it is coherent and consistent (and preferably true), at the pragmatic level we analyse how the author uses language.

Pragmatics relates to how human language is utilised in social interactions, as well as the relationship between the interpreter and the interpreted. What this means in practice is that the interpreter needs to understand the intention of the writer, but most of the time that intention is implicit. No one starts a piece by saying: I'm going to inspire you with this story.

It is precisely at the pragmatic level of text analysis where the key to understanding which User Needs a text satisfies is found. Semantic analysis of the text, such as searching for emotionally charged words used in traditional NLP approaches, can help to some extent in distinguishing the emotion-driven axis from the fact-driven or context-driven axes in the text. However, a typical fact-driven news piece can also carry an emotional charge that sentiment analysis will detect, and thus your classification system might incorrectly tag texts as emotion-driven when they are not. Such and similar situations can completely confuse traditional machine learning classifiers. The essence of the User Needs model is that practically any event can be written about from any axis of the model. You can start with a fact-driven article on some event, follow up by a contextual ‘Give me perspective’ type of story and then attempt at even higher audience engagement by publishing an emotion-driven story. That is why the question of how to computationally recognise which user needs an article satisfies cannot be resolved solely by analysing the meaning - the semantics - of the words and phrases in the text; it requires analysing how the author uses the language in the text, how it addresses different audiences, what its goal is, what effect it wants to achieve, and what reaction it aims to provoke.

---------------------------------------

The tool recognises fairy tales

Interestingly enough, the results still surprise us. We don't call the freely available tool the Playground for nothing, because you can play with stories and learn how Generative AI interprets them.

For example, we discovered that it recognises fairy tales and identifies them as emotionally driven stories. See what the tool makes of 'Little Red Riding Hood' (this text was used).

---------------------------------------

Text pragmatics and NLP at smartocto

This brings us to the problem that our LABS team - in charge of Research and Development at smartocto - has spent the last few years trying to solve: the analysis of text pragmatics represents the most challenging and, at the time we started, still far from a resolved problem in computational linguistics overall.

We have always worked with the latest language technologies. When I joined smartocto in 2022, I already had extensive experience in developing similar NLP systems and years of work as a Data Scientist at Wikidata, one of the largest and most complex semantic data systems in operation. The team had already been combining various machine learning models with dictionary and rule-based approaches, making significant efforts to enhance traditional mathematical methods by integrating them with human expert knowledge. We continued this work together. I tried to contribute with everything I knew; on multiple occasions we set the problem aside and returned to it, sometimes achieving small, incremental successes, but nothing that truly satisfied us. Even fine-tuning BERT, a famous early transformer-based deep learning model, would not deliver because of the scarcity of properly human-tagged data to train such models with.

Then the Generative AI boom started, making really Large Language Models like ChatGPT and Anthropic Claude available, and we realised that our idea to somehow combine human expert level understanding of the User Needs model with the capacity of LLMs to mimic human language almost perfectly was the key to the solution that we were looking for.

Solution = Knowledge Engineering + LLMs

If you try to present a system like ChatGPT with a specific news article and ask it to identify which User Needs it satisfies, it won't work very well. Trust us – there is hardly an approach that relies solely on that which we haven't already tried. LLMs are generalists that almost perfectly simulate human language and are the first NLP systems sensitive to the pragmatics of text, but they are not omnipotent. Understanding and using the User Needs model requires an experienced journalist or editor, while User Needs analysis requires at least a solid expert in media studies. The way we at smartocto finally solved the problem of how to automatically tag User Needs in any text is based on accepting LLMs for what they are and nothing more: they are simply linguistic machines that allow us to use almost all the information carried by a text under analysis efficiently, something that their machine learning predecessors could not recognise and utilise.

Knowledge Engineering

This is where the methodology of Knowledge Engineering comes into play, in which we apply procedures for the thorough analysis of expert knowledge – such as that encapsulated in the User Needs model – and its codification into a format suitable for processing with AI. We conducted a thorough synthesis of all scattered sources of knowledge about the User Needs model and experiences in its application, and then developed a system for characterising each axis of the model in order to be able to distinguish them. We logged every result in the development of the User Needs 2.0 model (see our User Needs 2.0 Whitepaper) and all insights collected from working closely with news brands across the world. This resulted in a set of rules that we then used to precisely direct LLM-based analyses toward our goal. LLMs are known as few-shot learners - systems capable of quickly learning to respond even to data not included in their enormous training datasets when prompted with particularly suited examples or guidelines. The key to solving the problem of User Needs classification in articles was to first discover the right knowledge representation scheme, and then find the best way to inform the LLM about exactly which text features it needed to observe in order to tag the text accurately.

The User Needs Tagger and Recommender we now have at smartocto are complex, integrated software systems that use LLMs merely as the "engine" of the machine, whose operation depends on the fine details that only human thought could introduce into the process. The standards of our LABS team (comprising mathematicians, cognitive science experts, computer science and machine learning engineers, and analysts) are high.

We were told that all problems would be solved with the advent of generative AI; that any author could analyse User Needs through ChatGPT or a similar system now; that it was only a matter of time before media companies developed specific LLMs to solve not just this but all other related problems in media analytics. None of this has happened yet, and after investing so many years in research, we were not willing to settle for anything less than the real thing.

The combination of expert knowledge via knowledge engineering with LLMs enabled us to perform way beyond the classification problem that we began with. In the latest iteration, our User Needs system takes into account not only the text as such, but also the characteristics of the writing style of any particular media outlet, their objectives, their expectations on various audiences. The smartocto User Need Recommender and Tagger are now integrated with other AI powered services under the common umbrella of the smartocto.ai approach. The integration of User Needs article tags with our well-known dashboards and analytics will follow immediately.

---------------------------------------

Educate me gone wrong?

Read this story from the BBC to see how it aligns more with news articles. The introduction seems to suggest that the piece primarily explains why the days on Earth are getting longer.

The article itself is then full of facts and mentions various scientific perspectives. As a result, our Playground becomes somewhat confused, to the extent that you can attribute that human characteristic to AI. The piece is 53% context-driven but also 40% fact-driven. The model finds it difficult to recognise whether the narrative of facts as a whole still contributes to the original purpose of the author. This is the explanation from the tool:

“The article is most probably a Give me perspective article. It delves into the scientific analysis of the Moon's impact on Earth's rotation and the lengthening of days over billions of years. It provides a detailed context for understanding the relationship between the Moon and Earth, including historical and current research findings. While it does present factual information, the depth of analysis and the aim to enhance understanding of a complex subject aligns more with providing context rather than just stating facts. The article does not primarily aim to elicit emotions or inspire action, which is why the scores for Emotion-driven and Action-driven are low. There is a slight emotional element in the contemplation of the vastness of time and space, hence the small score for Emotion-driven. The article does not offer practical advice or steps to take, which justifies the low Action-driven score.”

---------------------------------------

Ethical considerations and the smartocto 8 rules

A word or two about what we at smartocto believe should and should not be done.

First and foremost, from the very beginning of applying generative AI technologies, we established an important principle: we will not create content for media instead of their authors. We believe that the capabilities of Generative AI systems do not rival the quality and insight provided by professional journalists. Nonetheless, AI can serve as a powerful auxiliary tool in the newsroom. It can aid in generating alternative headlines, suggesting paragraph rewrites, or inspiring new questions and angles for stories. However, it is crucial that human professionals carefully review any generated content before publication. While AI can augment the content creation process, the ultimate decision to publish should always rest with a knowledgeable human editor, safeguarding the integrity of the journalistic process.

We introduced the maxim "Navigate, do not write." This guided us in formulating the AI-powered services we decided to develop: instead of creating content, we would develop systems that suggest possible alternative developments for the text the author is working on, either by providing direct recommendations about the text or by offering suggestions in the form of tags that the text in its current form satisfies. User Needs tags are just part of a broader system of tagging and analysing news articles that smartocto now offers. That system equally analyses both the pre-publication and post-publication phases of the news life cycle.

This intuitive approach to ethics that we took in the development of smartocto.ai and in the solution to the User Needs tagging problem led us, after thorough inspection of the currently available AI Ethics frameworks and regulations, to formulate a more general AI Ethics framework, authored by our CEO - the 8 smartocto rules for ethics in AI and journalism:

1. smartocto sees and uses AI as a force for good and actively supports measures that minimise the negative effects of AI on individuals and society.
2. smartocto will strive to work with AI vendors and partners that have a clear ethics framework around AI use in place.
3. smartocto will actively propagate for clients - media and marketing organisations - to work according to clear AI ethics rules, and to make them explicit to their employees and audiences.
4. smartocto will advocate - especially with respect to journalistic newsrooms - oversight, meaning that AI projects are monitored and AI output always needs to be manually checked by an editor before being published.
5. smartocto prefers the data set and output that is the least discriminatory based on language, religion, political or other opinion, national or social origin, property, birth or other status.
6. smartocto in developing AI features, will respect copyright and privacy.
7. smartocto will strive for the least harmful AI output for individuals and society, and is the most accurate.
8. smartocto will strive for and actively support research, training and awareness on the possibilities - positive and negative - of AI.

Coda: what remains to be done?

The answer is simple: everything that journalists and analysts did before AI entered our business and culture. The design of our AI powered systems such as the User Need Tagger and Recommender is crafted not to interfere with anyone’s creative or analytical work. In journalism, these systems might promote you, metaphorically, from sailors to captains, someone who now can and has to take into account more nuanced complexities in content production than before, and steer the editorial strategy to whatever waters their media ship might need to sail. From the perspective of an analyst, User Needs tags combined with numerous performance indicators will equip them to reach even deeper insights on the effectiveness and level of adjustment in content production to keep the crew informed and maintain real data-driven decision making. But both are still unique craftsmen and the masters of their own skills. We just build tools for them, relying on Generative AIs in the current iteration, and we believe very much that it is yet just another tech iteration, not some final state in which automation somehow takes over the process and drives it.

So, drop the AI hype and keep on writing awesome stories: we can help you understand better where your production stands and suggest a perspective from which to look at the article or suggest a change here and there, leaving you completely in control. The science of language and artificial intelligence has made progress, providing us with new tools at our disposal. At smartocto, we have put in tremendous effort to develop tools that can fully utilise these new ideas. However, the art of writing a good story remains irreplaceable.