Back in 1971, Herbert Simon coined the term “Attention Economy.” His simple but brilliant insight was that an explosion of information will inevitably lead to a scarcity of what it consumes: our attention. Simon believed that this concept would eventually dominate our economy.
The internet has brought fame to this almost prophetic vision. As a collective, we spend almost 10 billion hours per day consuming content on social network platforms across the globe.
This fundamental inequality between content and attention has given rise to a powerful new science: recommender systems.
Recommender systems are one of those inventions that most of us interact with every single day, but due to their invisible nature, we hardly ever acknowledge them. Whenever you do a search on Google, click your weekly playlist in Spotify, open Netflix, or scroll Instagram and TikTok, a recommender system is with you every step of the way. The system sources content opportunities for you, hoping to earn your elusive affection and loyalty. It also decides what you choose from by hiding 99.9% under the water, surfacing only the very tip of our huge iceberg of content.
These systems are so pervasive that they may easily decide crucial parts of our lives. Things like who we spend our time with, who we vote for, and where we work.
These systems are also a spectacular story of success and failure. They sit at the heart of every single major content platform, responsible for drastically improving stickiness. But as a recent article from Arvind Narayanan also pointed out: “Even with the best recommendation algorithm, only a fraction of your feed will be content that you truly enjoy. For example, the average ratio of hearts to views on TikTok is roughly 5%. People are just not that predictable.”
This leads me to the final point—recommender systems are inherently indifferent (or at best, unaware) of our inherent human goals, dreams, and values. More often than not, they simply show us the world we prefer to see, in exchange for improving the metrics their owners care about, which is sometimes to the user's detriment.
What is a recommender system?
While we could spend a whole conference discussing this topic alone, it does feel useful to provide a quick, practical framework of different approaches to making recommendations.
Recommender systems, overall, describe a set of algorithms designed for information filtering and ranking. However, there are only a few ways this challenge can be confronted:
Accuracy
The original iterations of Google relied heavily on concepts like authority and accuracy to try to find the best content for all people. This approach is anchored in the notion that objective or knowledge-based data, such as the length of an article, the presence of verifiable facts, and the number of authoritative sites linking to it, is a genuine expression of quality.
This was extraordinarily successful in scaling information filtering, but an accuracy-based approach misses two fundamental ingredients for success: the person and the feedback loop. Google has long since moved on from only relying on accuracy to power search.
Feedback
As soon as we serve content and recommendations to users in a digital environment, we acquire feedback data. Clicks, views, dwell time, shares, and likes are all part of the myriad of other digital data points now gathered which can provide recommender systems with new and powerful ways of spotting the right content needle in the haystack.
While still not necessarily personalized to each user, these algorithms are nevertheless powerful tools when unleashed across a wide body of content and a large user base, floating the most powerful and popular content to the top, thus giving it more exposure and affecting overall user behavior in a positive way.
This approach eventually also gave rise to the first versions of “personalization."
Personalization
There are many approaches to personalization, but most are not even really “personal” at all.
One such example is mass personalization: A website knowing a user's location, age, and gender may offer suggestions tailored to that user's demographics. Each user is classified by some relevant trait and then offered specific content or products aimed at that group. This means that the personalization is not individual to that singular user, it only leverages a specific trait that matches them with a larger group or cohort of people.
But, let’s focus on newer and more sophisticated forms of content personalization. For example, collaborative filtering is the method of making automatic predictions (filtering) about the interests of a user by collecting preference information from users (collaborating). The underlying and well-proven assumption of the collaborative filtering approach is that if person A has the same content interest as person B, A is also more likely to have B's content interest in a new and different piece of content than that of a randomly chosen person.
For example, a collaborative filtering recommendation system could make predictions about which television show a user should like given a partial list of that user's interests and history. These predictions are now specific to that user, but the system leverages information gleaned from many users to make these. In that sense, it is not true personalization, but it tends to perform well when you have less data available about each user.
Because the flip side is that personalization also moves us into more relative territory. Recommendations are not necessarily better just because they are personalized to a person. Let’s say a user has only ever read two articles, and we have little demographic information available—do we trust the system to pretend it “knows” them?
In fact, one of the common issues of working with many publishers is that they have a large percentage of casual viewers with very little user data associated with them. Users are usually not required to be logged in—unlike platforms like Instagram, where users have to be logged in at all times to consume content.
Now that we’ve covered feedback, personalization, and accuracy, it’s time to cover the other important facets of recommender systems. Stay tuned for part two.
Need publisher solutions and video content recommendations? Drop your details below.