Algorithm aversion is "biased assessment of an algorithm which manifests in negative behaviours and attitudes towards the algorithm compared to a human agent."[1] It describes a phenomenon where humans reject advice from an algorithm in a case where they would accept the same advice if they thought it was coming from another human.
Algorithms, such as those employing machine learning methods or various forms of artificial intelligence, are commonly used to provide recommendations or advice to human decisionmakers. For example, recommender systems are used in E-commerce to identify products a customer might like, and artificial intelligence is used in healthcare to assist with diagnosis and treatment decisions. However, humans sometimes appear to resist or reject these algorithmic recommendations more than if the recommendation had come from a human. Notably, algorithms are often capable of outperforming humans, so rejecting algorithmic advice can result in poor performance or suboptimal outcomes.
This is an emerging topic and it is not completely clear why or under what circumstances people will display algorithm aversion. In some cases, people seem to be more likely to take recommendations from an algorithm than from a human, a phenomenon called algorithm appreciation.[2]
Algorithm aversion has been studied in a wide variety of contexts. For example, people seem to prefer recommendations for jokes from a human rather than from an algorithm,[3] and would rather rely on a human to predict the number of airline passengers from each US state instead of an algorithm.[4] People also seem to prefer medical recommendations from human doctors instead of an algorithm.[5]
Various frameworks have been proposed to explain the causes for algorithm aversion and techniques or system features that might help reduce aversion.[6]
Algorithms may either be used in an advisory role (providing advice to a human who will make the final decision) or in an delegatory role (where the algorithm makes a decision without human supervision). A movie recommendation system providing a list of suggestions would be in an advisory role, whereas the human driver delegates the task of steering the car to Tesla's Autopilot. Generally, a lack of decision control tends to increase algorithm aversion.
Overall, people tend to judge machines more critically than they do humans.[7] Several system characteristics or factors have been shown to influence how people evaluate algorithms.
One reason people display resistance to algorithms is a lack of understanding about how the algorithm is arriving at its recommendation. People also seem to have a better intuition for how another human would make recommendations. Whereas people assume that other humans will account for unique differences between situations, they sometimes perceive algorithms as incapable of considering individual differences and resist the algorithms accordingly.[8]
People are generally skeptical that algorithms can make accurate predictions in certain areas, particularly if task involves a seemingly human characteristic like morals or empathy. Algorithm aversion tends to be higher when the task is more subjective and lower on tasks that are objective or quantifiable.
Expertise in a particular field has been shown to increase algorithm aversion and reduce use of algorithmic decision rules.[9] Overconfidence may partially explain this effect; experts might feel that an algorithm is not capable of the types of judgments they make. Compared to non-experts, experts also have more knowledge of the field and therefore may be more critical of a recommendation. Where a non-expert might accept a recommendation ("The algorithm must know something I don't.") the expert might find specific fault with the algorithm's recommendation ("This recommendation does not account for a particular factor").
Decision-making research has shown that experts in a given field tend to think about decisions differently than a non-expert. Experts chunk and group information; for example, chess grandmasters will see opening positions (e.g., the Queen's Gambit or the Bishop's Opening) instead of individual pieces on the board. Experts may see a situation as a functional representation (e.g., a doctor could see a trajectory and predicted outcome for a patient instead of a list of medications and symptoms). These differences may also partly account for the increased algorithm aversion seen in experts.
Different cultural norms and influences may cause people to respond to algorithmic recommendations differently. The way that recommendations are presented (e.g., language, tone, etc.) may cause people to respond differently.
Digital natives are younger and have known technology their whole lives, while digital immigrants have not. Age is a commonly-cited factor hypothesized to affect whether or not people accept algorithmic recommendations. For example, one study found that trust in an algorithmic financial advisor was lower among older people compared with younger study participants.[10] However, other research has found that algorithm aversion does not vary with age.
Algorithms are often capable of outperforming humans or performing tasks much more cost-effectively.
One way to reduce algorithmic aversion is to provide the human decision maker with control over the final decision.
Providing explanations about how algorithms work has been shown to reduce aversion. These explanations can take a variety of forms, including about how the algorithm as a whole works, about why it is making a particular recommendation in a specific case, or how confident it is in its recommendation.
Algorithmic recommendations represent a new type of information in many fields. For example, a medical AI diagnosis of a bacterial infection is different than a lab test indicating the presence of a bacteria. When decision makers are faced with a task for the first time, they may be especially hesitant to use an algorithm. It has been shown that learning effects achieved through repeated tasks, constant feedback and financial incentives can contribute towards reducing algorithm aversion.[11]
Studies do not consistently show people demonstrating bias against algorithms and sometimes show the opposite, preferring advice from an algorithm instead of a human. This effect is called algorithm appreciation. Results are mixed, showing that people sometimes seem to prefer advice that comes from an algorithm instead of a human.
For example, customers are more likely to indicate initial interest to human sales agents compared to automated sales agents but less likely to provide contact information to them. This is due to "lower levels of performance expectancy and effort expectancy associated with human sales agents versus automated sales agents".[12]