Dr. Weeks’ Comment: Dr. Sui Huang is a wise and insightful scientist and teacher. Here he gently reminds us of the merits of humility and a blazing curiosity.
When peers are not peers and don’t
know it: The Dunning-Kruger effect and
self-fulfilling prophecy in peer-review
by Sui Huang 1)2)
1) Institute for Systems Biology, Seattle, WA, USA
2) Institute for Biocomplexity and Informatics, University of Calgary, Alberta, Canada
Corresponding author: Sui Huang E-mail: shuang@systemsbiology.org
Bioessays 00: 000-000, 2013 WILEY Periodicals, Inc. www.bioessays-journal.com 1 Ex laboratorio
The fateful combination of (i) the Dunning-Kruger effect (ignorance of one’s own ignorance) with (ii) the nonlinear dynamics of the echo-chamber between reviewers and editors fuels a self-reinforcing collective delusion system that sometimes spirals uncontrollably away from objectivity and truth. Escape from this subconscious meta-ignorance is a formidable challenge
but if achieved will help correct a central deficit of the peer-review process that stifles innovation and paradigm shifts.
”˜”˜Real Knowledge is to know the extent of one’s ignorance”- Confucius
”˜”˜I know that I know nothing”- Socrates
Most of us have once felt the urge to blame unfavorable comments in the critique of a manuscript on the reviewers’ lack of relevant expertise. But such complaints often remain fruitless because both reviewers and editors, without malevolence, don’t know what they don’t know in the first place. Such ignorance of one’s own ignorance, as psychologists bluntly call it, can be excused on simple logical grounds, for how should one doubt one’s own knowledge about subject matters whose very existence one does not know about? Yet the recognition that there exist unknowns beyond the realms of what one knows is possible and is referred to as ”˜”˜meta-cognition” – an extraordinary
intellectual accomplishment of human mind due to its capacity of self-reflection. In a rare demonstration of metacognition among politicians, the former defense secretary of the United States, D. Rumsfeld, warned of the ”˜”˜unknown unknowns” as opposed to the ”˜”˜known unknowns” in the war against global terrorism. Thus, while it appears first as an epistemological paradox, metacognition can be achieved if one possesses
a particular intellectual ability to think in more encompassing categories. Conversely, not being aware of one’s own relative ignorance, or ”˜”˜meta-ignorance”, is pronounced among the less knowledgeable and the unlearned. This phenomenon has become known as the Dunning-Kruger effect [1, 2], especially when paired with over-confidence. Why is this relevant for the peerreview process? Do peer reviewers not know when their expertise does not suffice for the task entrusted to them, thus suffering from the Dunning-Kruger effect? Does their display of unmerited confidence mislead editors? The Dunning-Kruger effect and the peer-review process First, attaching the label ”˜”˜Dunning- Kruger effect” to peer-reviewers makes a triple claim: reviewers (i) can be ignorant of the subject matter concerned; (ii) are not aware of it; and (iii) act as if they are experts when in fact they often are not, thereby misleading editorial boards. One job of editors is to find the most suited reviewers. Nevertheless, we increasingly encounter reviewers who evidently lack sufficient expertise to evaluate a submission. The idea of peer review is that one need not stand above whom one judges but that it suffices to be at the same level. However, in reality reviewers are often not ”˜”˜peers” with regard to the pertinent expertise; rather, they are less than equals. One reason is the fragmentation of modern science into subspecialties
with their own terminological dialect. For instance, the term ”˜”˜gene network” can refer to entirely disjoint concepts, depending on who uses it [3]. Failure to consider the ”˜”˜other” meanings of a term prevents the recognition of one’s own ignorance of concepts used in other fields. Second, because of the parceling of science into small kingdoms, authors province with no equal. Finally, the increasingly interdisciplinary nature
of research creates an asymmetry of knowledge: the reviewer as a single person faces the daunting combined knowledge of an entire team of coauthors. Thus, statistically, we can safely accept our first claim and assume that on average, reviewers nowadays
are with high probability less knowledgeable about the subject matter of a manuscript than its authors.
In a perfect world reviewers are wise and will not succumb to the Dunning- Kruger effect. They may exert restraint and, with the awareness that their expertise lies in a different area, rather offer an outside perspective that improves the manuscript. However, such constructive criticism by the nonexpert peer will require the rare gift of meta-recognition. Conversely, authors themselves are not immune to the Dunning-Kruger effect. Research is unpredictable and may veer them into domains outside their expertise, in which case the rare wise reviewer with non-matching expertise who is at home in the relevant neighboring territory may catch the flaws. On the second claim: Are people really not aware of the limits of their knowledge – and why? Possible reasons invoked range from pride and self-deception to the aforementioned epistemological paradox of meta-cognition. The anonymity of peer-review may contribute to a subliminal relaxation of the ethical norms that otherwise commit us to concede obvious shortcomings. In any case, data obtained in a series of controlled experiments performed by Dunning, Kruger, and others over more than a decade firmly establish the human tendency to ignore one’s own ignorance. Intriguingly, meta-ignorance is associated with over-confidence, that is, a tendency to rank one’s own expertise higher than its effective level. Charles Darwin complained that ”˜”˜Ignorance
more frequently begets confidence than does knowledge” [4]. Bertrand Russell lamented about its consequences for society: ”˜”˜. . . in the modern world the stupid are cocksure while the intelligent are full of doubt” [5]. Quantitative analysis confirms that meta-ignorance scales with primary ignorance: the least competent people display the most overestimation of their own knowledge [1]. The latter symptom may explain the authoritative tone of unsuited reviewers that may captivate the editors. This substantiates our third claim. One mechanism that Dunning and
Kruger propose to explain how ignorant people suppress awareness of their ignorance is the so-called ”˜”˜reach-around knowledge” [2]: The uninformed take cues from the label (name) of a domainspecific concept, the existence of which they are not aware of, and relate that label, based on the word used or sound it produces, to a familiar concept from their own field of specialization or even, from everyday life. This is all too frequent in peer-review – for which author has not encountered a situation similar to the following: The term ”˜”˜chaos” has a specific meaning in the theory of nonlinear dynamical systems. But the nonexperts, not knowing about the very existence of this theory, may, when asked
for an opinion, simply fill the term ”˜”˜chaos” that they encounter in a manuscript with their familiar picture of ”˜”˜chaos” of inner city traffic and evaluate the manuscript based on such projected understanding. Similarly, too many reviewers see in the term ”˜”˜epigenetic” not the profound concepts that Conrad Waddington had inmind when he coined the term [3, 6] but resort to seeing only the concrete, little molecular modifications of DNA and chromatin that their lab is studying. Such compensatory use of ”˜”˜reacharound knowledge” prevents any recognition
of paradigm-shifting novelty that emerges in unfamiliar territories because of the failure to understand the novel usage of an existing term
and concept. But isn’t it the duty of the editor to intercept such pathological psychodynamics in the critiques? Editors naturally may not have as much technical expertise in the area in question as the peer reviewers. While perhaps equippedwith amore developed capacity of meta-cognition than the practicing peer-researcher, they have the same humanly reasons to fall victim to the Dunning-Kruger effect. This leads to the second psychosocial phenomenon that I would like to discuss. Combined with the first it will have quite unpleasant consequences for the peer review
process. The self-fulfilling prophecy If editors also succumb to meta-ignorance and if they blindly rely on the reviewers they believe to be experts, then one can envisage a ”˜”˜perfect storm” developing on the horizon: the collision of two systems of meta-ignorance would generate a higher-order self-perpetuating dynamics: a self-fulfilling prophecy of mutual confirmation of falsehoods – a ”˜”˜folie a` deux”. No explanation better
illustrates such fateful dynamics than the following anecdote: In the prairies of Alberta, Canada, winters are cold, wood is scarce, and native Indians are plenty -and well educated they are nowadays. One summer, a young Indian Chief, college-educated and therefore, incapable of reading the signs of Mother Nature, was asked by his people how cold the next winter will be. Embarrassed of not mastering the traditional skills for predicting the weather, and to be on the safe side, he just said to his people: ”˜”˜Well, I think this will be a pretty cold winter this year.” He then
sought help from his college friend, a meteorologist at the local Weather Channel station. ”˜”˜Tell me, Michael, don’t you think we are facing a cold winter this year?” Equally unable to predict the weather so far ahead, and also to be on the safe side, Michael the meteorologist confirmed the Chief’s opinion: ”˜”˜Yes, you may be right, Chief. This winter could turn out to be pretty cold.” Happy about this confirmation, the chief went back
to his people and confidently declared: ”˜”˜I see signs that the next winter will be pretty cold – let’s start collecting wood now.” And so the Indians started to collect wood along the few rivers. As autumn neared, the young Chief returned to the meteorologist and asked: ”˜”˜Michael, now that winter is getting closer, how cold do you think will it be this year?” – ”˜”˜Oh, I think this will be a really cold winter” was his answer. So the
Chief went back to his people and announced: ”˜”˜Folks, this year, I know, the winter will be particularly cold – let’s all join forces to
collect as much wood as we can.” A few weeks later the Chief asked his meteorologist friend for a more accurate prediction of the winter.
The meteorologist answered: ”˜”˜I am certain this will be an extreme cold winter!” Back with his people, the Chief announced: ”˜”˜People – I
have signs that this winter will be so cold that none of our fathers, as long as our memory reaches, have encountered. Let’s collect all the wood we can find!” Just before the winter the Chief consulted his meteorologist friend again, and the meteorologist told him: ”˜”˜This is going to be a record-breaking winter!” Curious about his certainty, the Indian Chief asked: ”˜”˜Michael, tell me, how do you know this?” To which the meteorologist replied: ”˜”˜You know, my friend, I have never seen this before: all the Indians have been collecting wood like crazy this year”. The Chief and the meteorologist form a positive feedback loop of mutual confirmation – reassuring each other in an idea that has little to do with objective
truth but instead will automatically fulfill their initially wrong assertion (prophecy). One cannot deny that sometimes editors and reviewers also establish such a self-fulfilling prophecy that amplifies their Dunning-Kruger syndromes, shields them from outside knowledge and prevents self-critical questioning that would have brought meta-cognition. Conclusion The anecdote of course is a caricature of the peer-review process but it illustratesa simple, important point. In the grander scheme of our scientific endeavor the Dunning-Kruger effect and the self-fulfilling prophecy of the echo-chamber create a tacit culture of collective self-deception that can dramatically narrow the diversity of scientific publications. One manifestation of such self-inflicted limitation is the monolithic dominance of standard views while entire domains of alternative ideas are suppressed. The keen observer will have realized that journals often solicit review articles on important subjects from a rather small pool of authors that editors consider leading experts. The dynamics of self-fulfilling prophecy propels these regular contributors
to authoritative positions above equally knowledgeable peers, which further stifles the diversity of ideas. Such ”˜”˜the rich-get-richer” dynamics
are consistent with the power-law distribution of citations of journals [7]. The psychodynamics of collective ignorance, driven by a positive feedback loop and the Dunning-Kruger effect, seems unstoppable like a perfect storm. What can we do? Meta-ignorance will always be there as long as humans do not know everything. One pragmatic solution lies in the hands of the editor: If the author appeals the rejection motivated by a reviewer who exhibits meta-ignorance, the editor should honor the rebuttal. Indeed, experienced editors, perhaps with tacit recognition
of the Dunning-Kruger effect and echochamber, seem to more generously exert this practice. This may pay off because, as recently shown [8], papers with a rejection/resubmission history have on average a higher citation rate which may well also apply to resubmissions to the
same journal. But the rest of us, when acting as reviewer, can also contribute to suppressing the reinforcement loop of meta-ignorance because it takes two to tango. The human mind has the wonderful, underappreciated freedom of conscious self-reflection to overcome meta-ignorance and to achieve meta-cognition. This will require exceptional intellectual honesty and modesty, and an acute, sensible mind that is open to one’s own limitations even in the sphere of anonymity. Acknowledgments The author gratefully acknowledges support from the Institute of Systems
Biology and from Alberta Innovates. The author has declared no conflict of interest. References 1. Ehrlinger J, Johnson K, Banner M, Dunning
D, et al. 2008. Why the unskilled are unaware: Further explorations of (absent) self-insight among the incompetent. Organ Behav Hum Decis Processes 105: 98-121. 2. Dunning D. 2011. The Dunning-Kruger effect: On being ignorant of one’s own ignorance. Adv Exp Soc Psychol 44: 249-94. 3. Huang S. 2012. The molecular and mathematical basis of Waddington’s epigenetic landscape: A framework for post-Darwinian biology. BioEssays 34: 149-55. 4. Darwin C. 1871. The Descent of Man, and Selection in Relation to Sex, 1st edition. London: John Murray.
5. RussellB. 2009. The Triumph of Stupidity. Mortals and Others: Bertrand Russell’s American Essays 1931-1935. New York: Routledge.
6. Waddington CH. 1942. The epigenotype. Endeavour 1: 18-20. 7. Baum JAC. 2011. Free-riding on power laws: questioning the validity of the Impact Factor as a measure of research quality in organization studies. Organization 18: 449-66. 8. Calcagno V, Demoinet E, Gollner K, Guidi L,
et al. 2012. Flows of research manuscripts among scientific journals reveal hidden submission patterns. Science 338: 1065-9. …..Insights & Perspectives S. Huang Bioessays 00: 000-000, 2013 WILEY Periodicals, Inc. 3 Ex laboratorio
Source of Article: Huang2013_DunningKrugerPeerReview_Bioessays