At the end of March, VRT Innovation and VRT NWS experimented for four weeks with VRT MyNWS, a web application that aims to bring readers news articles based on their interests. The experiment was led by VRT as part of the European CPN project. In this project, VRT Innovation helps with building a software that allows media organisations to personalise their news content. After the launch of VRT MyNWS, 949 testers subscribed to give feedback on their experience of personalisation, the usage of the web app and the display of articles. This is what we learned.
VRT MyNWS: a web app customised to readers
In order to measure whether people are better informed when they receive recommendations or not, the project team built the web app VRT MyNWS, resembling to the general news website of VRT NWS. News articles in the new web app could be found under three different tabs:
My news: personalised articles
Headlines: articles selected by the news department
Just in: most recently published articles
During one month, testers could give feedback via a button in the app, participate in surveys and were kept up to date via a weekly mailing. Overall, the team received over 200 emails with suggestions for improving the app. These mostly concerned the user interface and the recommendation algorithm. On a daily basis, a team of developers processed the feedback.
It was extremely informative to gather input from end users, as well as the news department, and link those two into practice. Only by working together and experimenting on the production floor, we are able to innovate.
Personalisation based on artificial intelligence and popularity
The test with VRT MyNWS took place in two phases. In the first phase of the experiment, the tab ‘My news’ was loaded with articles that were selected according to three algorithms: collaborative filtering, content-based recommending and a random selection of recent articles. By collaborative filtering, readers received news articles that other readers with similar interests also found interesting. By content-based recommending, articles were selected and offered on the basis of the content and metadata of the articles. Lastly, by including a random selection of articles, the project team aimed to avoid that readers would receive too many similar articles and end up in a so-called filter bubble. The testers were divided into two groups that each received their own composition of the three algorithms.
In a second phase of the testing period, the project team chose to recommend articles based on their popularity during a certain timeframe. The team wanted to find out what the ideal timeframe would be. Testers were once again divided into three groups, each receiving popular articles from the last hour, the last 12 hours or the last two days. The results showed that testers appreciated it more when they received articles that were popular for an hour, than for a longer period.
Feedback on the user interface and the algorithm
Based on the click behaviour, testers seemed to appreciate the simple algorithm from the second phase more than the complex system from the first phase. As such, the tab ‘My news’ was used 68% as opposed to other tabs in the second phase, while it was used 38% in the first phase. During the second phase, testers also used the 'My news' tab more when they were shown popular articles from only one hour old. Overall, testers answered that they felt better informed in the second phase.
During the first phase, feedback mostly came on the user interface, such as the width of articles, and the recommendation algorithm, such as irrelevant news. In both phases, the publication date of the article appeared to be an important factor in personalisation. Apart from the small amount of relevant, older articles, the majority prefers to read articles that are maximum two days old.
Unique for recommendations in the news area is that the age of the articles is crucial in maintaining the interest of readers.
The next step for news personalisation
The insights and results of the VRT MyNWS test are processed within the European CPN project. There, the project team calls out to other European news organisations to personalise their news stories with the software from CPN. In a new pilot, the Cypriot news organisation Dias and Deutsche Welle will also be experimenting with the personalisation of their news articles. The insights and next steps will be shared on the project website of CPN. VRT will continue to work on personalisation opportunities. Within the project NewsTAPAS, VRT Innovation also explores content adaption within news personalisation.