Why “peer reviewed” studies aren’t reliable (especially for beauty science)

There are a lot of myths in beauty, and over the years, I’ve noticed that a lot of them come from misunderstanding how science and scientific evidence works.

Here’s one big misconception: the idea that if a source is “peer reviewed”, it’s automatically higher quality than a source that isn’t. Unfortunately, it isn’t any sort of guarantee, especially for a field like cosmetic science.

This article is adapted from this video, which explains the extremely dodgy “peer reviewed” paper that convinced a lot of people to promote rosemary oil for hair loss.

1 How science works

2 The peer review process

3 Problems with peer review

4 The bottom line

5 References

How science works

To understand peer review, we need to talk about how science works.

Back in the day, when scientists looked like this…

…to do science, you’d:

Poke around some random stuff
Find something interesting – science was pretty new back then, so everyone was making discoveries everywhere

Tell your scientist pals about it at your science club meetings
They’d clap or argue with you, and some would build on your discoveries in their research

You might write a book about it or get an article published in a magazine (which in science is called a journal),
scientists would clap or argue with you, some would build on your discoveries in their research
If they really didn’t like it, they might burn down your house

But after a few hundred years, people had discovered a lot of stuff, and science was far more complex. A single journal editor didn’t know everything about the subject anymore. So to decide which articles should be published, they would ask other scientists who knew more about that topic to weigh in.

By the 1960s, with super advanced technology like photocopiers, this process of peer review – getting other scientists to decide if your articles were good or not – became a lot more structured.

The peer review process

These days, the (simplified) general process for publishing peer reviewed academic research is:

Do some research
Write an article
Send it to a journal
Editor review: The editor checks that it mostly looks OK, if the research is important enough (that’s going to come up again later), if it matches the journal topic
Peer review: If it passes this initial review, they’ll send it to maybe 2 or 3 scientists who specialise in that area. These reviewers will check the paper, including whether your experimental procedures are appropriate, and whether your conclusions make scientific sense with established knowledge, from previous studies.
Acceptance: The reviewers send feedback to the editor, then your article will be rejected or accepted (usually you’ll have to make a few changes before publication)
Publication: Your article gets published – hurray!
Post-publication review: After publication, other scientists comment on your work or try to repeat your experiments. Papers might get corrected or retracted, it might be compared to other studies in reviews and textbooks, and add to the general body of scientific evidence (the less complicated “established science” you get taught at school and university).

Problems with peer review

This peer review system is generally pretty good – or at least it’s probably one of the better systems for building scientific knowledge. But it’s far from perfect.

Some big issues with peer review (this is a very incomplete list):

Less popular topics aren’t scrutinised as much

Plenty of dodgy papers get through, and it’s especially the case with an area like beauty, where a lot of the most rigorous research is never published (I discussed this in my post on whether retinol is a scam). So there isn’t a lot of replication, and not many scientists are discussing and critiquing the studies that come out.

Related post: Is Retinol a Scam? The Science

Peer review isn’t rewarded much

Peer reviewers aren’t really paid or recognised when they review papers well. Peer review is voluntary and mostly anonymous, so the quality is pretty variable, and it tends to take a long time. If a reviewer is checking a paper that probably won’t get a lot of attention, they might not take it as seriously and won’t read it as carefully.

Peer reviewers can be biased

Reviewers might also not read a paper as critically if one of the researchers is well known in the field, or comes from a prestigious institution – because you’d expect them to do good research. Sticking a prominent author on your paper is actually a well known hack for getting papers published quickly with minimal revisions.

This is one reason why there’s been so many high profile cases of retractions lately, for blatant issues like photoshopping.

Some journals have instituted double anonymised peer review, where the peer reviewers don’t know who the authors are, to reduce bias. This can help, but doesn’t entirely remove bias – since the reviewers are experts in the same area as the authors, it’s not that difficult to guess who they are based on the topic or methods they use. Papers also often reference previous work the authors have done, which makes it even clearer.

Dodgy journals might not actually do peer review

There are also dodgy journals that don’t actually send papers out for peer review – they’ll just let it sit for a bit, then wave it through. Most journals charge authors to publish in them, so there’s a financial incentive for dodgy journals to publish as many papers as possible with minimal effort.

(Note: A dodgy journal doesn’t mean the study is automatically bad – it just means you don’t know if it’s been peer-reviewed properly. And along the same lines, a good journal doesn’t mean the study is automatically reliable – but the obviously dodgy stuff is more likely to have been weeded out.)

The bottom line

What all this means: you have to be very careful when looking at any scientific paper. Don’t just take what it says at face value – you need to read the whole paper slowly and carefully to work out if the claims it makes is reliable.

While we can get a lot of useful information from cosmetic science studies, it’s also easy to fall for bad science, or bad study interpretations. I’ve debunked many questionable “scientific” claims before:

This leads us into another huge issue with evaluating scientific papers – a lot of the time, you can only read the abstract. This is a huge issue because abstracts often aren’t accurate summaries of papers! Here’s a deep dive into how abstracts should be used.

Related post: Why you can’t trust study abstracts

References

Kincaid E. Former Stanford president retracts Nature paper as another gets expression of concern. Retraction Watch. Published December 18, 2023. Accessed February 19, 2024.

Jones N. How journals are fighting back against a wave of questionable images. Nature. Published online February 12, 2024. doi:10.1038/d41586-024-00372-6

Bik E. Opinion | Science Has a Nasty Photoshopping Problem. The New York Times. Published 2022. Accessed February 19, 2024.

Types of Peer Review. Wiley. Published 2024. Accessed February 19, 2024.

Lagut A, Wong M. How to read an academic science paper. Beauty SciComm Group. August 28, 2023. Accessed February 19, 2024.