Why Are Scientists So Obsessed With the Peer Review?

I recently got into an argument about the safety of the measles vaccine.

I’m quite fond of the person I was debating with so I tried to gently persuade them that the benefits of most vaccines outweigh the risks. I was also curious as to where they were getting their information from, particularly the so called studies that link autism to childhood vaccinations. I was happy that they were able to produce the reports but my heart sank when I realized the document she sent me was a work of pseudo-science.

So, I started dissecting the report against vaccinating children in a point-by-point critique. About half-way through fact-checking the document I realized my efforts were futile. Even if I had convinced my friend that the document was wildly inaccurate, it wouldn't solve the underlying issue; that there are many people spewing fake and pseudo-science all over the internet and it can be very difficult to tell the difference.

This is a huge problem for both scientists, who are fond of their jobs, and society as a whole, who are fond of the medical and technological advances produced by scientists. If the goal of science is to improve the world around us, it is imperative that we trust the work that scientists do. Otherwise we might as well say, "That's it, we've learned enough and we don’t need any more solutions to our problems".

So, rather than discussing the lack of merits of one particular report I thought I would take a few days to write an overly long post that nobody will ever read about the scientific method, the peer review and the relative merits of faith vs skepticism. So, voilà, my next way-overdue blog post.

The Prevailing Scientific Opinion (changes)

Sometimes it seems like scientists just can't make up their minds. Nutrition debates (eggs, are they healthy or not?) drug recalls (DES anyone?) and filling thermite-covered balloons with hydrogen  (anyone want to go for a ride?) are good examples of how scientific opinion changes over time. Given all these goofs it would be easy to think that, by enlarge, "expert" scientists are crap at their jobs.

That would be incorrect. Scientists are a lot of things, awkward, nepotistic, sometimes squinty-eyed but they (we) generally are insanely hard workers with good intentions-not unlike most everyone else out there. But the fact is that because science is an incremental process every so often, scientists are necessarily going to get things wrong.

Let's take a wee detour and travel back in time to 350BC. At the time Aristotle was one of the most brilliant visionaries the world had ever seen. He was transforming math, physics, arts, zoology even politics. Yet he was firmly of the opinion that the universe revolved around the earth. And why shouldn't he have been?  Telescopes hadn't been invented yet and when he looked at the night sky he would have been dazzled by the stars moving across the sky in a nearly perfect orbit. But there were questions that remained unanswered, like what caused the retrograde motion of some planets and why did their brightness change in time?

Copernicus, bit of an awkward turtle. Thankfully in the time before twitter trolls.

It would take another 800 years before someone else would challenge that view.  It wasn't until 1543 that a one Mr. Copernicus (and several more of his obscure predecessors) came up with a new theory about how the solar system worked. Had he been alive when De revolutionibus orbium coelestium was published he would have spent the majority of his life being ridiculed.

At the time there wasn't overwhelming evidence for Copernicus' theory. However, more and more evidence started to pile up in support of his model of the solar system until finally everybody agreed; Humans are not the center of the universe (despite the objections of many a two year old).

This is a good example of how scientists and the general public should think about new scientific ideas. It's good to be skeptical when something new comes out, be it a theory of the origin of the universe or a most excellent new diet pill. However, more people look into a field the evidence almost always becomes clear. Through careful observation theories will either be supported (yes! we revolve around the sun!) or refuted (DES is a bad thing to give pregnant women!).

The evolving of ideas is a critical aspect of scientific thinking. From climate change to treating complex diseases like ALS; science is an incremental process. Even Stephen Hawking gets it wrong sometimes (and has to cough up $100 to a colleague in the States). Does that mean he's a crap scientist? I highly doubt it.

The "Peer-Review"

A popular mantra in academia is publish or perish. And publish means publishing in “peer-reviewed” scientific journals. The peer review process may be new to anyone who stopped taking science courses in high school. Basically, the peer-review is a quality control for scientists. Once a scientist, or anybody really, does a research project they can submit it to scientific journals for peer-review. A research paper submitted to peer-review is often called a manuscript.

The basic premise of the peer review is that all work submitted to the journal must be clear, logical, unbiased and repeatable. The manuscript first goes to journal editors who decide whether the research fits the aims of the journal. If it does, they send the manuscript on to three other people who are familiar with the subject. The reviewers then recommend to the editor that the work be accepted, accepted with revisions or outright rejected. The benefit of this is that people who know the field intimately get a chance to anonymously critique the work. If a manuscript is rejected that doesn't mean it can’t or won’t get published. There are hundreds, maybe thousands of journals out there and it is not uncommon to submit something to three or four journals until the manuscript is accepted.

This is not to say that good work can't be done outside of the peer review system, it can and does. Many pivotal works have been published as books. However, the vast majority of work that has resulted from and supports their ideas has been peer reviewed. This includes pivotal works by Copernicus, Galileo, Darwin and even Einstein. Want a recent example? Go watch the Dallas Buyer's Club

The peer review system is far from perfect. Crap can get through and editors and reviewers can be biased, sexist, and nepotistic. But let's not throw the baby out with the bathwater. The peer-review system is the best system available for quality control of reasonable science. I like to think of it as a group of highly-competitive vacuum cleaner salesmen.

Bear with me now.

Imagine a vacuum cleaner salesman showed up at your door and said, "The Suck-O-Tron is the best vacuum cleaner ever made!” He then proceeds to spill all sorts of unsavory things on your carpet and clean them up with the device. You are intrigued but would rather not spend the money at the moment-thank you very much. Later on, you meet three vacuum cleaner repairmen. They confirm that the Suck-O-Tron is great and they stand by the product. Would you feel more confident that you were making a good investment? I hope so or this silly analogy is worse than I thought.

The peer-review system works in a similar way. Knowledgeable people get to critique the work of another scientist and weigh in on whether the design, analysis, and interpretation of the results are reasonable or whether there is a critical mistake that needs to be addressed prior to publication. This largely prevents bad science from getting published and also prevents the need for everyone to be an expert in everything. While it’s FAR from perfect the peer-review system is like a stamp of approval. Again, good science can be done by anyone but if something seems too good (or bad) to be true and it hasn't been peer-reviewed-buyer beware. You may be in for a shit vacuum cleaner.

Scientific Literature is F*****g Incomprehensible

 In order to remain impartial, scientists are trained to divorce themselves from their work. An unfortunate consequences of this is that they often end up writing using a passive voice. E.g... methods were optimized based on previous research (Snargelenfurf et al. 1633) and improved upon until the reader fell asleep. On top of that, scientists have a very bad habit of using jargon that nobody outside our working group understands. The combined result is that the scientific literature can be incomprehensible. I have yet to meet anyone who is delighted by the process of reading a scientific manuscript.

Too much jargon face

This is a problem that scientists desperately need to fix. With the plethora (large numbers) of people producing fake or pseudo-science articles it is imperative that we clean up our act and make our work understandable.

On Science and Faith

They are not the same, and that's great! Faith, is a belief that is not based on proof. How nice that is, faith transcends and gives us comfort. The faith in god, homeopathy, my dog not peeing on the floor when I leave, are all good things. Faith allows people to feel connected to something greater than themselves. Faith can even heal in a real, statistically significant way.

Science is not based on faith, and that’s also great! Where faith starts out with the answer and looks for evidence to support that idea, science says, “I have no idea what’s causing this phenomena! Let’s try to find out.” Every single discovery in science has been made by people asking questions, digging, and following the evidence. If god has been directing it, then great, but it’s been the hard work, sweat and blood of scientist that have brought us modern medicine, alternative energy, aviation, smart-phones, conservation drones and so much more. Both faith and science are beautiful beautiful things that are invaluable to us as a society.

Where we get into trouble is when the two become entangled. It's surprisingly easy to get the wool pulled over our eyes if a reputable, trusted source is telling us something that is not true. Both the History and Discovery channels have been doing that lately by producing fake documentaries on everything from doomsday to extinct and mythical creatures. When the people we trust to inform us tell us lies it is fundamentally abhorrent.

Back to the MMR Thing

As everyone knows by now, the original scientist who published a link between the MMR vaccine and autism MADE THE WHOLE THING UP. This was a case of bad science sneaking through the peer review system. What's great about science is that it must be replicable. As soon as the purported link was discovered, other scientists started looking for it too. They ALL came up with nothing. Nada. Zero. Zilch.

Do I believe that all of the subsequent studies are perfect? Of course not! Do I trust the publically owned vaccine companies? Not as far as I can throw them. What I do have faith in is that there are hundreds of skilled dedicated people out there looking for the causes of childhood cancer, autism, chronic headaches etc. etc. etc. and that with enough time, resources, and help from the wider community they will find the answer.  

And they will publish it in peer-reviewed journals. 

---

Snarky things I refrained from saying but reeeeeeeeeeally wanted to:

1. You don't have to vaccinate but all unvaccinated people must stay together and away from the vaccinated ones and the ones who medically can't be vaccinated

2

.

3.