The surprisingly popular answer is a wisdom of the crowd technique that taps into the expert minority opinion within a crowd.[1] For a given question, a group is asked two questions:
The answer that maximizes the average difference between the "right" and "popular" answers is the "surprisingly popular" answer.[2] The term "surprisingly popular" was coined in a 2017 paper published in Nature entitled "A solution to the single-question crowd wisdom problem", which outlined the technique.[3]
Suppose we'd like to determine the answer to the question "Is Philadelphia the capital of Pennsylvania?" The two questions asked of the group, and the average responses, are:
Is Philadelphia the capital of Pennsylvania? ("Right" question)
What is the average probability people will assign to "Philadelphia is the capital of Pennsylvania"? ("Popular" question)
The difference between the answers to the right question and the popular question:
Thus, the No answer is surprisingly popular (10% > −10%). (The capital is not Philadelphia, but Harrisburg.)
The technique avoids the "double-counting" of prior probabilities across participants, a major issue for belief aggregation rules under the naive assumption that participants' answers are independent. Say a crowd has two groups:
When asked to answer a question, non-experts will tend to give equal answers to both questions. This is because they have no reason to expect they are wrong in either direction—their answer is just as likely to be an overestimate as it is an underestimate. (If the participants expected to change their probability estimates after learning more information, they already would have.)
However, the experts have access to both the prior probability and the posterior probability, which allows them to make a better estimate of the group's opinion. Because they know the group contains both experts and non-experts, they will expect the average probability to be in between the prior and the posterior. This means that, unlike the non-experts, their answers will not tend to cancel out when the prior probability (as proxied by the "popular answer") is subtracted out.
Looking again at the capital example, say there are two groups, experts and non-experts:
The strength of the method is that it causes the two non-expert groups to cancel out, thus identifying the opinions of the expert group. (It is assumed that most people who think they have "inside" knowledge are correct and knowledgeable, rather than misled.)
For m>2 candidates, the Surprisingly Popular Algorithm requires votes from an infinite number of voters on all possible ranked permutations (m!) of the alternatives to recover the ground-truth ranking with complete certainty, as discussed in the Nature article. However, the algorithm can be extended to recover rankings using various elicitation formats.[4]