It’s probably not apparent to the outside world, but scientific research is a cut-throat business. Most research funds provided by government and by charities are allocated through a competitive process where different scientists’ plans and track records are compared, with a few winners getting all the money and everyone else getting none. A good application takes months to prepare. Each rejected application represents a significant waste of resources.
image kirstyhall Flickr.com
This process is repeated over and over again. Those who are consistently successful will go on to long and productive careers in research, but many scientists who don’t win their own funding will eventually have to give up. No doubt in many cases they will go on to fulfilling careers in other sectors. But this is a great pity because by this stage they will have often have gained highly specialised skills and extensive experience which will be lost to the economy.
There are a wide variety of different funding schemes, but to even qualify for the entry one must have a PhD and typically several years of postdoctoral experience. Because most applicants are so highly trained and experienced there are a large number of excellent applications, and the funders are obliged to choose a small proportion (10-20% for projects, as little as 3% for highly competitive personal fellowships) and reject all the rest. Inevitably some of these decisions can seem rather arbitrary even to those who make them.
This post was prompted by a debate between Jenny Rohn and Athene Donald on the criteria used for competitive fellowships. Jenny had argued that some of the eligibilty criteria that applied to fellowships needlessly and unfairly excluded certain potential applicants, the example she gave was a fellowship which excluded applicants with more than eight years postdoctoral experience. Athene Donald responded with the funder’s perspective. this was a detailed and technical argument which related to specific schemes and criteria but the gist of Prof Donald’s position was quite general and summed up in the title and opening sentences:
“Whoever Said Life Is Fair?
“Any parent will be familiar with a child’s endless whine that ‘it isn’t fair’ – that their sibling got the larger slice of cake or that their classmate’s bedtime is half an hour later than theirs. And the parent’s logical repsonse is to say something along the lines of ‘whoever said life is fair?’”
Later in outlining her case with respect to specific programmes she had overseen and in responding to comments on the article Prof Donald acknowledges that the system involves some arbitrary element – chance, luck or unfairness. Of one scheme she writes:
“It was all but impossible to winnow that down ‘fairly’, based on the evidence we had in front of us, to the single figure number of fellowships we had to give out (applications to fellowships ratio I recall as being something of the order of 30:1). Did the ‘best’ win? Impossible to tell. The one consolation one always has – as an organisation, not as an individual – is that although the outcome has an element of luck in it, one can be sure that all those appointed were thoroughly deserving. The fact that another, let’s say 10%, would have been equally deserving is just depressing – for the organisation as well as the individual of course – but at least one knows the money has been well spent.”
“The competition rules are set to encourage a well-defined cohort who have the potential to be the scientific leaders of tomorrow… funders have to be pragmatic and set criteria which lead to manageable pools of applications. They know all individuals who make it through the process are worthy winners. They also know there will have been equally worthy individuals who lose out. No it isn’t fair, but it may be doing the best that can be done with the money available.”
Related issues are dealt with in an insightful post by Megan Cully which again wrestles with the difficulty in determining who is best:
“One in ten biologists has a professor/assistant professor position 10 years after completing her PhD. Admittedly, some of those have left science of their own volition, but many more have been driven out by a lack of opportunity. Theoretically, if everyone wants a to become an academic, a 10% success rate should mean that the best 10% of scientists get positions while the rest do something else, which isn’t that different from a lot of other careers. Surely we want the best scientists to lead their own research programs. That’s the problem. I’ve seen people in that top 10% get academic jobs, and I’ve seen people in that top 10% leave science altogether. Same for the other 90%.”
Dr Cully argues that luck plays a big part in these huge life-changing decisions:
“Some of this comes down to outstanding experimental design and skillful execution, but in equal measures it comes down to luck. Even outstanding scientists don’t publish exclusively in the Holy Trinity. Some great ideas simply don’t pan out, or the answer to a key question was “no” rather than “yes”. Biology can’t be bent to the experimenter’s desires. The answer doesn’t change the quality of the work, but it changes the interest factor and therefore the impact factor of the resulting paper.”
Of course, this luck applies equally to all scientists. So is Athene Donald’s “whoever said life was fair” a rational response? Are funder’s really “doing the best that can be done with the money available”?
I don’t think so, or at least, I don’t think we can be at all confident about it.Funding competitions are won, not on the basis of an individual’s capacity to become an effective leader or to deliver great science but based on a combination of skill and luck. In a highly competitive system, only those who are both excellent and lucky will gain funding. Those that are excellent and unlucky may not gain funding. This much is acknowledged in Prof. Donald’s post. Skill cannot be assessed directly, but is evaluated on the basis of scientist’s track record and the quality of the proposal itself. The track record plays an important part in the judgement.
If funding leads to increased output then it follows that those who have been funded in the past will have a stronger and more competitive track records when it comes to future competitions. The lucky and excellent are likely to outcompete those who are merely excellent. It is even conceivable that a fairly good but very lucky applicant might beat an outstanding but unlucky one. Over time, the small initial impact of luck could easily be amplified.
In other games of skill and chance, we take steps to avoid the arbitrary advantages that can accrue from good fortune. For example, in poker it is advantageous to sit immediately to the right of the dealer. The dealer is initially chosen at random, but the position rotates with every hand, so that no one player gains an overwhelming advantage from where they happened to sit down. Similarly in football it is more difficult to play against a headwind. A toss of a coin decides the initial direction of play, but it is reversed at half time to reduce the effects of chance on the outcome of the match; there is an advantage to playing at home, so important games are played out over two legs or at a neutral venue. In the league each team plays each opponent at home and away.
Scientists are similarly careful to avoid bias in the design of their experiments. For example, psychologists are trained to counterbalance the order of conditions in an experiment, because learning or fatigue that builds up when performing one task might affect the performance on a subsequent one.
Without detailed evidence or a mathematical model (coming soon), it is difficult to tell whether an accumulation of random factors leads to suboptimal funding decisions, but it certainly seems possible. I am not sure how far funders consider this issue. To what extent are current funding structures reinforcing luck at the expense of skill? Are there any further steps they could take to reduce any cumulative inequity in their decision-making and improve the overall quality of science they deliver? I think the jury is out.
I regularly commute to work on my bike. I always notice a headwind but I am rarely aware of a tailwind, it just seems easy sometimes. But the wind is still there. No one ever said that life was fair, but where it is unfair, we can and should try to make it better.