Why you can’t trust study abstracts

One of the biggest sources of myths in beauty is misunderstanding how scientific research works, and how to interpret scientific evidence.

A huge problem I’ve seen is people relying on abstracts – the short summary of the paper – instead of reading the full text. Here’s why this isn’t a good idea.

This article is based on part of this video on why rosemary oil isn’t actually a science-backed treatment for hair loss.

1 What are abstracts?

2 Why abstracts aren’t accurate

3 Conference abstracts

4 Example

5 Further reading

What are abstracts?

As I explained here, “peer review” doesn’t guarantee that a scientific paper is high quality. Peer review has many problems, so you always need to read the full text of a paper critically. From my years of reading peer reviewed cosmetic science papers, I’d estimate that about a quarter to a half of them have statements that can’t be taken at face value.

But there’s a huge problem: a lot of the time you might not actually be able to access the full text.

Many papers are behind paywalls. Academic publishing is a pretty crappy system: journals don’t pay for research, reviewers review for free, and the authors usually have to pay the journal to publish the paper… but the journal gets to charge people to read it. There are more regulations to make government-funded research publicly available, but it’s an ongoing process.

The part that’s always free to read is the abstract, and is meant to be a summary of the paper. Here’s an example of what they look like when indexed in PubMed:

However, the abstract isn’t necessarily an accurate summary of the paper, and basing your opinion on the abstract without actually reading the full paper is probably the biggest trap when it comes to interpreting studies.

Never trust the abstract!

Why abstracts aren’t accurate

In academic research, abstracts are only really used to work out if a paper is relevant to your research topic, so you can decide whether it’s worth reading the rest of the paper. It’s only the starting point for interpreting the paper, not a substitute.

Abstracts are missing details

Abstracts are really short, so by necessity it’ll be missing a lot of information. You won’t be able to critically assess if the methods they used are dodgy, for example.

Importantly, it’ll be missing many of the limitations of the study, and the authors probably only included the results they wanted to highlight. These are often the most subjectively interesting results – a lot of the time, null results (where the researchers didn’t find an interesting link) won’t be included.

Abstracts can exaggerate the paper’s importance

There are also less benign reasons that abstracts also usually aren’t 100% accurate.

Abstracts are often treated like advertising for the rest of the paper. It’s the first thing journal editors and reviewers see, after the title and authors (outside of double anonymised peer review, at least).

There are more papers being submitted than can be published, so research that seems more important is more likely to get published – authors often only include results and implications they want to highlight. So after you read a lot of papers, you’ll notice that abstracts usually hype up the paper a bit too much. (This is also a big problem with the “future research” sections of papers.)

Think of the abstract as like the two sentence synopsis of a movie. Green Lantern sounds promising, The Boy and the Heron seems reasonably straightforward to understand, and The Room doesn’t come across as an incredible work of genius.

Conference abstracts

Some abstracts are even less reliable!

Peer-reviewed journals often publish conference presentation and poster abstracts before the conference. These are meant to let potential attendees plan which presentations and posters they’d like to check out, and to serve as a record of the event.

But while these abstracts get published in peer-reviewed journals, the actual content and research aren’t really peer reviewed. Conference abstracts are usually accepted and published even before the presentation or poster is finalised! All it really means is:

the authors paid the fee to present at the conference
the conference organisers felt that the topic was suitable for the conference
the research exists (probably)

Example

Here’s one good example where solely relying on the abstract led a lot of doctors, dermatologists and scientists to recommend a likely ineffective treatment.

This study found that rosemary oil worked as well as 2% minoxidil for hair loss. The abstract actually sounds quite impressive, claiming there was a “significant increase in hair count at the 6-month endpoint”.

Unfortunately, the full text of the paper is full of obvious errors – a big red flag for questionable research practices and unrigorous peer review.

Additionally, the “significant increase” turns out to only be an extra 1-6% in hair count on average, which could be accounted for by seasonal hair count variation. (Note: “Significant” doesn’t mean “big” in scientific studies – it means statistically significant, which essentially means there’s a low (usually less than 5%) probability of getting that result if there was no real effect.)

You can read my full analysis of the evidence for rosemary oil here.

If you want to get better at critically analysing papers, check out this article I cowrote on how to read an academic science paper. And practice!