Brash or brilliant?

How do we select people for career advancement, and are these processes really accomplishing what they set out to achieve? In these posts I am mainly concerned about my own field, science and academia. My worry is that the best scientists are not always selected, and that some groups, such as women, are systematically disadvantaged by our current processes. I think this is a problem, not just because of the unfairness it entails, but also because it means that the quality of the whole enterprise is being undermined. Although I am thinking principally about science and especially about women in science, I think the issues are much more general and so where possible I am going to try to write about them in general terms.


Ampersand Cartoon by Barry Deutsch. Thanks to Rachel Jones for sharing it on Twitter

As I plan to write a few posts on this topic, I want to begin by dividing the processes I refer to into two, potentially separable, parts: criteria and evaluation. By criteria I mean the objective characteristics we use to determine whether someone is doing well at their job. By evaluation I mean the (often subjective) way we determine the extent to which someone meets these criteria. Flaws in the criteria and evaluation would imply that the people best able to do a job are not always selected for advancement. Biases in the criteria and their evaluation would mean that these errors will work systematically in favour of some groups and against others. Neither flaws nor biases need result from a conscious intent on the part of those involved. Flaws might be due to unavoidable gaps or noise in objective information, leading to uncertain measures of objective criteria, and in the case of bias there is ample evidence to demonstrate that unconscious factors play an important role in human judgements under uncertainty. If we agree that our procedures might be flawed or biased, we should identify and fix these problems regardless of whether they are intended or unintended. In further posts I want to return to the issue of bias and how bias plays out in selection procedures and over time through career advancement. But for now I want to raise a problem that I think gets too little attention, and that is the question of whether our criteria are flawed or biased. I think they often are. One case identified in a blog post by Jenny Rohn looks at the effect of eligibility criteria relating to age and experience for a particular fellowship scheme. In this example, she explains how a small and apparently innocuous change in the wording can exclude candidates with atypical but promising paths involving career breaks, childcare responsibilities or industrial experience. The apparently reasonable aim of the criteria is to target those whose rapid progress demonstrates greatest potential. The unreasonable effect is to exclude those whose less rapid progress is unrelated to their potential. The general point is that, whenever we use criteria that do not directly correspond to candidate’s relevant abilities and achievements, we run the risk of introducing flaws that could lead to unintended bias.

It is clearly the case that when we select among candidates for a job or promotion we will want to choose people who are better than others or, in absolute terms, excellent by some yardstick. It goes, frankly, without saying. So why do job adverts so often specify that applicants will be “excellent”, “outstanding”, “world-leading”, “exceptional”, and so on? This can only have the effect of encouraging applications from people who think they’re fantastic and presumably deters more modest people, regardless of their objective abilities. In my view adverts, job specifications, funding eligibility and promotion criteria ought to focus on objective characteristics and scrupulously avoid terms that are open to subjective interpretation.

Wording of a current ad in This is purely illustrative of a widespread trend and no particular criticism is attached to the employer. After all, who wouldn't want to employ an extraordinarily gifted scientist.
Wording of a current ad in This is purely illustrative of a widespread trend and no particular criticism is attached to the employer. After all, who wouldn’t want to employ an extraordinarily gifted scientist.

Flawed criteria are particularly problematic because we tend to equate the outcome of a selective process with the desired result. For example, people will say, “as a modern scientist you do need to be competitive and confident – these are important characteristics” but this is rather circular. The reason a scientist needs to be competitive and confident is that only then will he or she win grant funding, publish numerous papers in glamour journals, gain promotion and win prizes, but this in turn is only because in each case evaluation criteria place undue emphasis on confidence and competitiveness. You do not need to be confident to do excellent science that will make an enduring contribution to knowledge. It seems very likely to me that in science the brash may at least occasionally outcompete the brilliant. The central point of this post is that if we want the best person for a job, we must ensure that our criteria for determining what’s “best” are aligned with what we actually want the person to do. Assuming we have such criteria, it’s still possible that subjective evaluations based on them will be affected by bias, and in future posts I want to show how the mechanisms of bias can be surprisingly subtle and long-lasting.