And the science dunce cap goes to...
The Male Brain sent over this link a few days ago, and it's just too good not to turn into a full post. It's all about how scientists can reach two completely different results from the exact same data set:
In a test of scientific reproducibility, multiple teams of neuroimaging experts from across the globe were asked to independently analyze and interpret the same functional magnetic resonance imaging dataset. The results of the test, published in Nature today (May 20), show that each team performed the analysis in a subtly different manner and that their conclusions varied as a result. While highlighting the cause of the irreproducibility—human methodological decisions—the paper also reveals ways to safeguard future studies against it.
“This is a landmark study that demonstrates clearly what many scientists suspected: the conclusions reached in neuroimaging analyses are highly susceptible to the choices that investigators make on how to analyze the data,” writes John Ioannidis, an epidemiologist at Stanford University, in an email to The Scientist. Ioannidis, a prominent advocate for improving scientific rigor and reproducibility, was not involved in the study (his own work has recently been accused of poor methodology in a study on the seroprevalence of SARS-CoV-2 antibodies in Santa Clara County, California).
Problems with reproducibility plague all areas of science, and have been particularly highlighted in the fields of psychology and cancer through projects run in part by the Center for Open Science. Now, neuroimaging has come under the spotlight thanks to a collaborative project by neuroimaging experts around the world called the Neuroimaging Analysis Replication and Prediction Study (NARPS).
Neuroimaging, specifically functional magnetic resonance imaging (fMRI), which produces pictures of blood flow patterns in the brain that are thought to relate to neuronal activity, has been criticized in the past for problems such as poor study design and statistical methods, and specifying hypotheses after results are known (SHARKing), says neurologist Alain Dagher of McGill University who was not involved in the study. A particularly memorable criticism of the technique was a paper demonstrating that, without needed statistical corrections, it could identify apparent brain activity in a dead fish.
Perhaps because of such criticisms, nowadays fMRI “is a field that is known to have a lot of cautiousness about statistics and . . . about the sample sizes,” says neuroscientist Tom Schonberg of Tel Aviv University, an author of the paper and co-coordinator of NARPS. Also, unlike in many areas of biology, he adds, the image analysis is computational, not manual, so fewer biases might be expected to creep in.
That problem has been well known for years by now. Even basic findings from seminal papers cannot be easily replicated, using the best available scientific methods. And even the best and most experienced scientific teams around the world cannot replicate each other's experiments using the exact same data sets. This is unquestionably a huge problem because it destroys the credibility and validity of scientific research in all aspects.
However, once you understand that science is not a single monolithic field of human knowledge, but is in fact at least three, if not four, different things in one, you begin to realise that it is not the scientific method that is the problem. It is the scientific PROFESSION.
As I have stated many times before, "science" is in fact three different pieces - these days, more like four:
- Scientody, the scientific method of hypothesis, experimentation, observation, refutation or acceptance;
- Scientage, the body of available and testable scientific knowledge;
- Scientistry, the profession of science - that is to say, what scientists do;
- Scientism, the religion of science fetishism that has taken over much of the world today;
Furthermore, if you read Thomas Kuhn's book, The Structure of Scientific Revolutions (I haven't, so if I get something wrong about it, someone please correct me), the philosopher who wrote it made it clear that science does not proceed in a uniform straight line of any kind. Instead, it goes through much the same process that most accumulations of knowledge go through. That is to say, things accumulate over time and "settle" in one accepted paradigm, until something comes along and seriously challenges that view.
That revolution is almost always met with universal disdain and derision, until enough evidence accumulates around it to make it clear that it cannot be reasonably refuted. It then becomes the new paradigm, until it is challenged in its turn.
The root of the replication crisis lies in the fact that it relies primarily on what can be seen, observed, and measured. That is an extremely powerful and useful method of understanding the world, make no mistake. But the core problem with it lies in the fact that it relies on observation.
This would not be an issue if the observer were an unbiased, unfeeling, total mechanical and mechanistic robot.
Without belabouring the obvious, I think we can all accept that human beings are anything but robotic. (Not even me - and I've been called robotic and machine-like to my face many times, especially recently.)
Humans always interject our own biases and beliefs and systems of thought into our observations of the world. There is no getting around this fact. To think otherwise is to deny that scientists are human beings. It is to give scientists a degree of reverence and separation from Mankind that they do not deserve.
In the end, problems like the one that The Male Brain highlighted are endemic to science because science is a HUMAN field. Can it ever be corrected? Well, probably not, because scientists have grown far too confident in the supposed infallibility of the scientific method. Far too many scientists believe that they, and they alone, have the keys to unlock the secrets of the Universe - when, in fact, they struggle mightily to explain their own findings and ideas.
Looking ahead to the future, I think that we are going to come to a time when scientists will have to admit, with great sadness and frustration, that they have struggled for hundreds or even thousands of years to climb to the top of a mountain of human knowledge, only to find that philosophers and religious scholars have been sitting there waiting for them to catch up the entire time.
That isn't my original idea either, by the way. I paraphrased the late NASA astronomer Robert Jastrow, who said the following:
At this moment it seems as though science will never be able to raise the curtain on the mystery of creation. For the scientist who has lived by his faith in the power of reason, the story ends like a bad dream. He has scaled the mountains of ignorance; he is about to conquer the highest peak; as he pulls himself over the final rock, he is greeted by a band of theologians who have been sitting there for centuries.