This week psychologists, neuroscientists and statisticians have been again prompted to reassess our methods by a widely-circulated preprint (pdf) accusing some in the field of acting like “methodological terrorists”. Thankfully, the initial urge to identify with one or other “side” (and to hit back at “opponents”) has largely given way to more thoughtful, nuanced and long overdue discussions about how research should be conducted. I am not arguing for or against one of the “sides” in this debate, but I am just sharing my perspective.
Scientists have learned that we need to continually question our beliefs about the natural world, and test them against the evidence. As Feynman put it: “The first principle is that you must not fool yourself – and you are the easiest person to fool.” We have to question our beliefs. This applies to our beliefs about the scientific method as well as to our beliefs about the natural world. It applies to ourselves as well as the other people who we think are wrong.
Arguments and debates are good – they are an expression of this questioning approach. But we know that people are fallible. The experience of other people being wrong is commonplace, but no-one thinks themselves wrong, and we only grudgingly accept that we may have been wrong in the past. Our own confidence may be misplaced, so we should approach arguments about natural phenomena with humility and generosity. If we are right, this approach also has the advantage of being more persuasive.
This is doubly true of questions about the scientific method, epistemology and statistics, where evidence may be lacking and which may go beyond our own expertise and training. As far as I can see, even those who do regard themselves as experts in these areas are prone to contradict one another and change their views over time. I really don’t believe you can “overthink” this stuff, underthinking (on all sides) is a far greater problem in my view. I don’t claim my thinking is especially coherent or complete, but neither do I accept that others have all the answers.
I’ve been a full-time practicing scientist for over 20 years and more or less every day I think about how to conduct research so as to maximise my contribution to knowledge. I think my contribution will depend on my specific skills, knowledge, experience. It will depend on my prior beliefs and hunches, the models and hypotheses I generate, and which I favour, taking into account, for example, their compatibility with other phenomena/explanations, their complexity, their flexibility, their testability. It is not, for me, as simple as specifying an optimal or satisfactory method for assessing the truth or falsehood of an individual hypothesis that someone else has already come up with. Indeed some scientists, like myself, may feel that they can make a bigger contribution in the generation of new models and methods than in the elimination of existing alternatives.
A new model or theory develops through a long and iterative process. It draws on a wide range of theoretical constraints and multiple empirical observations associated with varying degrees of evidence. To the best of my knowledge there is no recipe for this type of work. Little is purely observational, some is exploratory, but much is systematic and hypothesis-driven, only without the aim of producing a definitive binary result in a single study. Clearly such studies should never be presented or interpreted as definitive, although a collection of results can be very persuasive. Should the results be suppressed in the file drawer, or reported? Should all results be reported, even if in retrospect the researcher realises the design or method is fundamentally flawed? How will the authors gain credit if an initially flakey idea proves transformational? Should the ideas be shared before they’ve been triaged against experiment?
Although reproducibility is an absolutely critical part of science, it is not on its own sufficient. There are other types of research, including experimental research, that are valuable in forming and selecting among hypotheses. At least some of this research is worthy of publication, not least so that it can help others, in parallel, form and select between viable hypotheses and models and to identify effects and predictions worthy of more thorough testing. In my view, experiments with small sample sizes that each yield weak evidence can form part of a cogent, perhaps even optimal, strategy. I don’t doubt the integrity of fellow scientists just because they publish a result that is not definitive and proves to be unreplicable. I think observation, exploration and low-powered experimentation have important places in science. In the absence of evidence to the contrary I think these preferences are a matter for individual scientists and are unrelated to the prevalence of questionable research practices.
None of this is an argument against preregistration or registered replications which, I think, can be very valuable tools in the process of eliminating hypotheses once they are fully formed and selected. I do think that the earlier stages of the process are undervalued and mismatched to our current peer-review and publication processes which sometimes assume that the only result worth publishing is a definitive rejection of the null hypothesis. Respectfully, I disagree.