Science is never “done”. We will never have all the answers tied up in a bow with no chance of there being changes later. There are theories that are pretty settled: we aren’t all going to start falling away from the ground unless something big happens (you know, like a huge, Jupiter-sized planet colliding with us). The fundamentals of evolution aren’t going to be contradicted, only refined. But to discover new knowledge a lot of modern scientific work is going to be on the edges of what we already know. Thus, it’s definitionally pretty tentative. A science paper is thus just one team’s (or author’s) view of some question. A science paper is not gospel. Just because it’s published “in a journal” doesn’t mean it’s good research or even that it’s remotely true.
But people (rightly or wrongly) put a lot of stock in claims when they are backed up with a journal citation. Unfortunately, it’s pretty easy to get anything published in some journal, somewhere. When we consider claims that are not consistent with broad scientific consensus on the topic, a good article on those claims would have to be very impressive. Scientists who successfully challenge consensus do not solely get published in journals no one has heard of (or journals that many other scientists think have low standards). Good challenges of consensus on a topic are well-written, well-supported and acknowledge the contradicting research. An article I was passed recently is a good example of an article that is not good quality and shows traits common to low quality articles. This example is related to biotech agriculture (aka “GMO crops”) but the signs can be applied to many types of journal articles.
The Example Article
The article I was sent is “A Review on Impacts of Genetically Modified Food on Human Health” (PDF) by Verma, et. al. I discussed this article on the Biofortified forum so some of this was discovered by others there. The article is ostensibly about transgenic crops and their uses and harms, but it shows strong signs of being written to support a pre-conceived position that isn’t consistent with the body of scientific research. It thus is a good example of what to look out for when someone sends you an article that “proves” something is true.
Broad, Strong Claims Using Slanted Language
First, credible science articles do not make over-the-top claims — at least not usually and generally not in strong language. When I skimmed the article, I noted a section heading with the title “GMOs ARE INHERENTLY UNSAFE”. First, researchers in biotech crops don’t often call them “GMOs” in their articles because the term has different connotations to many since nearly all human foods are “genetically modified” from wild sources. It’s also a somewhat loaded term due to activists over the last decades and some science articles I’ve read only mention the term in the keyword list. But more importantly the phrase “inherently unsafe” is just not scientific language. There are very few substances that could be labeled that way but even chemicals that strongly cause cancer — or even nuclear materials — would probably not be called “inherently unsafe” in a good article because there are still legitimate uses for them. Few things are “inherently unsafe” — or inherently safe for that manner.
Detecting Cherry-Picking
You don’t need to necessarily know a field well to spot that an article is cherry-picking the other research it is using to bolster its position. In this case, this is ostensibly a review or commentary on biotech agriculture in general. If you’re aware of the topic at all you know that there are a lot of biotech crops planted worldwide. Thus, in a section titled “GM DIETS CAUSE LIVER DAMAGE”, if it is going to fairly demonstrate the claim is true, it would need to cite many research papers that did not show this as well as the few that do and argue why the latter are more convincing than the former. Why? If GM food actually caused liver damage, we would have strong epidemiological evidence and many worried primary care doctors. There would be studies on it. And, in fact, there are numerous studies on GM foods looking for organ damage (primarily on non-human animals since they are easier to do controlled studies with) and very, very few show any problems. A balanced review of this question thus needs to cite those articles. This section doesn’t. But you can spot that the article is cherry-picking because it’s just not credible that there are only a handful of studies on this question and none of them contradict the claim. There’s almost always some paper contradicting a health or medical claim.
But sometimes when looking at an article you don’t have background cues like this to judge the likelihood of cherry-picking. I firmly believe that non-scientists should be able to look at the scientific literature themselves. You have to be careful to not fall for the trap of thinking that one article is gospel but you can go read the literature. If you don’t know a lot about a topic, a good way to get a handle on it is to find a review article in a high profile journal. They are usually written in a less specialist way and explain more terms. I would look for one in a PLOS journal (they have several open access ones that anyone can freely read) or in Science or Nature (the “prestige” general subject journals of the world) or a subset of subject-specific journals. But you’ll run into some trouble. How do you decide which journals to trust a review in? A Google Scholar search for the topic plus “review” will probably turn up a well-cited review. But unfortunately, you might not have access to it. But let’s assume you get access, this should give you a good idea of the field but more importantly it will have lots of citations. If you think an article is cherry picking and it misses a seemingly relevant citation in this list that contradicts the article’s claim, then you might ask why.
Major Typos, Editing Mistakes or Ungrammatical Language
This one is admittedly a bit hard. Even a good quality journal might publish something with typos or other editing mistakes. But one that has many of them or incredibly blatant ones was probably not carefully edited by the authors or the journal staff. This example has one so blatant it says the journal is of very low quality. The abstract (the introduction of a scientific article and often the only thing many people read) contains this: “?not sure what is being said here?”. Those question marks are not a typo on my part and are actually in the PDF I downloaded. This is clearly an editor’s insertion during the editing process that was left in the published document. The article itself is dated from 2011 so there’s been ample time to correct it on the journal’s website. It hasn’t. I’m left with the impression that it wasn’t carefully reviewed before publication and moreover that hardly anyone has read it after publication.
Now, this particular sign is obviously somewhat problematic. Many of us don’t really know English well enough to distinguish stilted, poor writing from actually ungrammatical writing — and a lot of scientific writing is stilted and poor even in good journals. Moreover, many scientists by necessity publish articles in English even though that isn’t their first language or even one they are fully fluent in. However, we’re talking about articles that challenge more consensus views in a field. A good one will most likely get support to improve any language issues before publication because it will be so note-worthy. In any case, it’s only a sign and I would err on the side of giving an article the benefit of the doubt for spelling and grammatical errors.
Predatory Open Access Journals
You’ll have noticed if you’ve ever tried to read published scientific research that often you just can’t get to the paper. Most journals still require subscriptions to access articles. A scientist publishing in a particular field most likely has access to the more important journals in his field (often thru their university having an institutional subscription) but you probably don’t. But journal subscriptions are expensive, even for a university, and many researchers don’t even have access to all the papers relevant to their research. Some scientists have been pushing for a model called “open access” where any article is freely available to anyone (on the internet) once published. The premier example is the PLOS journals mentioned above, but even the high-profile closed access journals Science and Nature provide a way for authors to make their paper open access. The main benefit of open access is anyone can get to the research but many of the open access journals also make an effort to publish more rapidly than some other journals (though they are still peer reviewed).
But this leads to a darker side. Open access journals currently primarily depend on publication fees paid by the authors of an accepted paper to pay for the costs of maintaining the journal. Accepted articles are still reviewed by other scientists for free but editors and staff to maintain a website does cost money. Good open access journals are transparent about their funding structure and also waive publication fees for scientists that can’t afford it (e.g. from developing nations). But the benefits of open access have been sufficiently hyped — and researchers sufficiently desperate to get published — that there are numerous predatory open access journals published by companies that aren’t necessarily prioritizing good quality science but rather accepting papers regardless of quality so they can collect publication fees.
This particular article was published in a Bentham Science journal which is part of a group that look pretty scammy to me. One problem being noted with this group is that scientists who don’t even work in the field a journal is supposed to cover are being asked to be editor. Editors at journals are responsible for filtering and curating appropriate work to be peer-reviewed, usually set policy on peer review and quality and often even have the authority to publish a paper despite peer reviewer objection. Choice of editor for a journal is a big deal and extremely important towards setting or maintaining quality. A publisher that asks inappropriate people to be editor is not a good publisher, especially if they do it repeatedly. This publisher also spams scientists repeatedly to publish in their journals. I don’t think I can trust anything coming from this group of journals (though see the addendum below for more on why this paper is particularly poor.)
These are just a few signs that an article someone sent you isn’t very good. It’s not fool proof and you would probably want to ask someone you trust for their thoughts as well. Modern social media (and heck even email!) means you can actually find someone who is well-respected to ask. You can even send a tweet to some scientists and many will answer. There are also a lot of science journalists on twitter as well: they may not be scientists but they can spot bullshit pretty well. The point here is not to nit-pick a paper to shreds, but rather to let you figure out if a strong claim someone is making based on some paper is a reasonable claim and supportable by the balance of scientific evidence. Big claims require strong evidence and a poorly written paper in a crappy journal that doesn’t acknowledge contradicting research just isn’t good evidence.
Addendum: And even more wrong ….
As a small (hopefully) addendum, I want to note that no one should cite this paper ever as evidence that GMOs are harmful. Beyond the above flaws, we discovered that:
- The journal editor and one of the authors of this paper appear to be the same RB Singh. Editors do sometimes get their own work published in their own journals but it can be a sign that they are misusing their position to increase their own publication record.
- The RB Singh in question appears to be a researcher who is fairly convincingly believed to have problems with their work. While it’s possible that the Ram Singh on this paper is not the same one as in this noted case in the BMJ, they seem to share the same affiliation in the same city in Utter Pradesh so I think it likely. I’m not sure I would trust a researcher with this publication record (and resistance to providing data).
- Karl Haro von Mogel on Biofortified discovered the “GMOs ARE INHERENTLY UNSAFE” section was plagiarized from a document published by an anti-GMO activist Jeffrey Smith. I hope this is obvious, but a credible scientific article is not going plagiarize.
With all of that, I’m sad to say that this paper is cited by a publication from the Center for Food Safety. I originally learned about it from a local Democrat who replied to questions I had sent to the local group trying to get a GMO labeling law on the ballot. I’m sad to see it apparently must be commonly cited in activist groups. It’s frankly not a credible document.
Updated 2012-12-02 12:15: I fixed a few missing words and cases of unclear grammar.
Fancy Beans 