ZOMG! Mothra! Mutant Fukushima Butterflies and Reading Science News Critically
The headlines are scary: “Mutant butterflies found near Fukushima plant”, “New casualties from the Fukushima nuclear disaster in Japan: mutated butterflies”1, “Fukushima disaster spawns mutant butterflies”, “Mutant butterflies linked to radioactive fallout from Fukushima nuclear disaster”. Semi-serious references to Mothra and bad nuclear sci-fi abound. But what does it really mean?
The first story (or at least the first heavily linked story) about new research on abnormal butterflies caused by radioactive exposure from the Fukushima accident was probably the BBC story. It’s a splashy result: “severe abnormalities” in butterflies collected near the site of a nuclear disaster. The abnormalities get worse as time goes on and new generations are born. An independent researcher highly praises the study. Anyone wary of nuclear power is expecting exactly these kinds of results and will find the story compelling. But that’s when we should be a bit more skeptical, possibly even trying to understand the paper ourselves and not just accept the media explanation. In this case, media reporting has been too uncritical, preferring scary images and stories about abnormalities in place of examining the results.
This post ended up being a lot longer than I expected. I would apologize but everyone is saying we need to read more longer pieces, right? :)
The BBC Article and the Study
First, let’s read the BBC article critically. It starts:
Exposure to radioactive material released into the environment has caused mutations in butterflies found in Japan, a study suggests.
That’s a pretty strong introduction so we’d expect a very strong study. We should be looking for information about significant results: large numbers of butterflies, significant differences between populations with more or less radiation exposure, etc. Let’s read on! After explaining that the researchers collected butterflies in regions near and more distant to the Fukushima Daiichi nuclear power plant two months after the accident, the news article says:
By comparing mutations found on the butterflies collected from the different sites, the team found that areas with greater amounts of radiation in the environment were home to butterflies with much smaller wings and irregularly developed eyes.
Deformed eyes are mentioned in several of the news articles. However, if you read the data table describing which abnormalities were found in this collection (May 2011), you find that only wing, color pattern and appendage abnormalities are called out separately. In fact, the rates of eye abnormalities are not actually called out anywhere, for any sample2 nor does it appear that eye abnormalities specifically were significally correlated with distance from the power plant (or radiation levels). Some classes of abnormalities such as wing, appendage, and color pattern are grouped together but there’s a bucket of “other” abnormalities as well. Eyes are in the “other” group so we don’t know how many butterflies had abnormal eyes or if eye problems by themselves were significantly correlated with exposure. The mention of eye abnormalities is thus not giving a realistic picture of the results. So, why call out abnormal eyes at all if not significant? Well, gruesomely deformed eyes are more eye-catching (pun intended). The study also includes pictures of various abnormalities and, to this layman’s eyes, only the eyes look particularly weird. It’s not surprising eyes are used as an example, but they just don’t appear to be representative of the results.
Continuing on, the BBC article states:
It was by breeding these butterflies that they began noticing a suite of abnormalities that hadn't been seen in the previous generation - that collected from Fukushima - such as malformed antennae, which the insects use to explore their environment and seek out mates.
A striking mutation that might make it hard for the butterflies to breed sounds bad. However, the paper itself does not give any indication that deformed antennae specifically appeared more frequently in populations bred from those collected closer to the accident. Antennae are grouped with other appendage deformities which the paper does say is significantly correlated with radiation measurements and distance from Fukushima. But we just don’t know how many were specifically antennae abnormalities. The authors also do not demonstrate that the malformed antennae are leading to breeding trouble, so at best this claim is speculative. It’s just one of many possibilities — but it’s scary and understandable.
The next paragraph talks about a second collection of butterflies in September and October:
Six months later, they again collected adults from the 10 sites and found that butterflies from the Fukushima area showed a mutation rate more than double that of those found sooner after the accident.
The team concluded that this higher rate of mutation came from eating contaminated food, but also from mutations of the parents' genetic material that was passed on to the next generation, even though these mutations were not evident in the previous generations' adult butterflies.
The idea in the study is that small amounts of damage in the parents are being passed on and resulting in abnormal features in succeeding generations. The explanation of how it could happen is actually a pretty good summary of a section in the paper though it is stated in a much stronger way. However interesting the mechanism is, it still has to be reliably demonstrated for it to make sense and that’s where the BBC report falls down. The BBC basically ignores sampling questions. It gives overall numbers collected in in the field (144 in May and 238 in September) which makes the result sound solid, but the numbers per location matter statistically. For the first collection in May, these are some locations with as few as three and no more than twenty-four (actually collected from multiple sites in one region). They just aren’t very big samples.
To calculate an abnormality rate change (“mutation rate of more than double”3) between the first and second collection, you can only compare similar data. In this case, you have to exclude butterflies where we don’t have collections in both time periods. Only seven of the ten locations are in common which limits the total numbers collected to 121 in May and 192 in September. Curiously, the samples in May for Shiroishi and Koriyama were not used to calculate abnormality rates and significance between all the May collections because the sample sizes were too small — totals of 6 and 3. But later, when calculating this change in abnormality rates, the samples for Takahagi and Mito — also only 6 butterflies collected — are included without explanation.
Sample sizes are actually a pretty big problem in this study. It’s a common problem for sampling to not be described in a news article, but it’s critical to deciding whether or not a result is likely to be true. Small samples or biased samples usually indicate inconclusive or unlikely results. Considering some of the problems in the samples behind the “double” the abnormality rate result, it should have been described with more skepticism.
However, easily the best sign of uncritical reporting in this BBC article is the quote from a scientist not involved in the study:
"This study is important and overwhelming in its implications for both the human and biological communities living in Fukushima," explained University of South Carolina biologist Tim Mousseau, who studies the impacts of radiation on animals and plants in Chernobyl and Fukushima, but was not involved in this research.
"These observations of mutations and morphological abnormalities can only be explained as having resulted from exposure to radioactive contaminants," Dr Mousseau told BBC News.
Why do I find these quotes so outrageous? The researcher’s statement makes a reader believe there must be few problems with the study and the methods and results very convincing. “Overwhelming” even. However, the results described in the news article are some of the most problematic ones in the paper. For a scientist to describe that as “overwhelming” should have made the reporter consider the credibility of the statement or wonder which part of the paper was being referred to. At the least, I would expect the reporter to seek out other opinions (who might agree even!). But very few complex science questions are answered “overwhelmingly” in a single paper. The world is just too complicated for a single paper to tell us much with high confidence.
Unanswered Questions
Further questions the reporter should have been asking about are related to the breeding experiment. The researchers took the collected butterflies and had them produce eggs. This sounds like a great idea and possibly could detect damage affecting reproductive cells — child generations would have developmental abnormalities if their parents’ eggs or sperm cells were damaged. However, in this study, the children only come from a handful of butterflies per site — only 3-5 females laid eggs. There are a lot of possibilities for why this might have resulted in increasing abnormalities but due to the small sample size we don’t need an interesting explanation: it could all just be chance. Even assuming this experiment provided a strong result, there are other questions to answer (or at least ask). The lab bred butterflies showed higher overall rates of abnormalities than the second field collection. The second field collection is actually three or four generations after some of the butterflies used in the breeding experiment so if the damage is persisting in the butterfly population, I would expect similar abnormality rates or a huge decline in butterfly populations due to the abnormalities. Alternately, the butterfly population is recovering from any damage. Or there’s another explanation I haven’t considered. But the paper itself doesn’t discuss this and the media reporting ignores the question.
Sadly, the part of the study that I think is most convincing is not widely reported on. Another experiment in this paper involved raising butterflies on leaves collected from various sites in the Fukushima region. The larvae used were from eggs laid by Okinawan butterflies that live far from Fukushima. The control group was fed on leaves from Okinawa as well. The sample sizes in this part of the study are still not huge, varying from 16 to 77 males and 21 to 59 females raised per region4, but much better than the other parts of the study. The results seem to show developmental problems — excess deaths at various stages of the butterfly life and increased abnormalities in those fed on leaves collected closer to Fukushima. This is pretty convincing to me that there’s something to this. But this part of the research is not even mentioned in the BBC article and I haven’t seen very many media reports that do. In any case, I highly recommend reading this paper for yourself.
Other Media Reports
So now we have a decent grasp on this study and what it can say and how important it might be. How are other news organizations reporting this? How are people sharing it on social media? The answer is: poorly. The following stories I found using Google News on Monday and Tuesday morning or saw on twitter. Obviously there have been more published since and this is not fully representative. Sadly almost all repeat the same basic results that the BBC article does and many even quote the same independent researcher.
The Australian ABC reported the study as being published in Nature itself rather than the very new Scientific Reports which is published by the same company. But it was otherwise a very brief and context-less article.
The CBS News story barely mentions the actual paper results and instead chooses to mention that the region around the plant has reopened and is considered safe and that radioactive tuna5 were found off the coast of the United States — though thankfully they mention the levels are not believed harmful. But there’s basically nothing concrete about the results given.
Grist went all out for journalistic excellence and led their story with: “If this news surprises you, you haven’t watched a lot of B-movies.” That’s certainly informative. The post then goes on to imply the exclusion zone isn’t really populated with many people. This may be true, but the zone is actually very small — and most of the samples in this study came from regions outside the current zone (Supplement Table 1 puts only the Hirono sites within the current exclusion zone which is less than a thirty-two kilometers radius). The Grist article gives the impression that these results are for places people are not allowed to live in.
ABC News (on Yahoo for some reason) has a longer article than most and actually mentions some of the results, but it’s entirely unskeptical. ABC also mentions the tuna found to have Fukushima-origin radiocative nucleotides in them (thankfully also noting that the amounts aren’t considered harmful.)
The Examiner (which was so good on the GMO grass story) runs as their lead image a protest sign with “Don’t Nuke The Future” and a skull at the top. Then it can’t help but mention Godzilla as a comparison to this “more modest scale” of change. No really. The article is otherwise about as good as the BBC article (though it lacks any quotes from other scientists) and follows the BBC’s structure fairly closely.
In the joke category, we have Pets Lady which titled their post “Is Mothra Real? Mutant Butterflies Spawned by Radiation Fallout?” I guess we won’t need to read much of that one. I only mention it because for some reason the site comes up when searching Google News for “fukushima butterfly”.
Gizmodo went right for tabloid and titled their piece “Mutant Butterflies Link Fukushima Meltdown to Human Disease”. Obviously nothing in the research says anything about human impact. The piece even uses panic while pretending not to: “While this finding isn’t cause enough for panic, there’s no way it could be considered good news.” The piece otherwise doesn’t really discuss the results or significance and depends primarily on quotes and comparisons to Chernobyl and the nuclear attacks on Hiroshima and Nagasaki.
Discovery News (which should know better) chose an inflammatory title: “Nuke Disaster Spawns Mutant Butterflies”. Like the joke entry, they lead with a Mothra reference: “Japan may have a real-life Mothra on its hands.” Further lines from the piece play up the significance and danger with emotional language. “The fluttering freaks were found by the Japanese entomologists”6 is nearly content-free but reads pretty. The sentence “Only near the site of the damaged Fukushima Dai-ichi nuclear power plant did the scientists find abnormal butterflies” is blatantly untrue: abnormality rates as high as 35% were found 130 kilometers away (in Mito). Even Tsukuba (170 kilometers away) had rates around 6-7% in both May and September samples and was clearly seen as a “control” sample7. But otherwise, the Discovery article is pretty derivative of the BBC and even uses the same quote from an independent researcher.
There are obviously a lot more news articles out there of varying quality. Salon even decided to declare these results evidence of a new species of butterfly. The reporting on this story is at best derivative, generally uncritical, frequently invoking fear and speculating on risks for humans.
Social Media Impact
Because of social media plugins and search tools I can estimate the reach of these stories. As I’m typing this8 I see 4700 shares on the BBC story. The Australian ABC story doesn’t seem to have social media plugins that count shares, but Backtweet gives over one hundred links to the article. CBS has around 300 between the major networks. Grist has around 175 shares. The ABC news story (again on Yahoo) has 1300 Facebook recommends and 102 tweets (and a sad 22 on G+). Weirdly though this story seems to have the most comments on it (almost 1500!) The Discovery article had around 275 social media shares.
Searching twitter, I’m seeing a lot of irresponsible text in linking to the story. The word “mutant” appears pretty commonly which is a loaded way to describe these results, evoking bad movies and Cold War era nuclear scare stories. As an example, the Nature Conservancy chose the text: “Fukushima caused mutant butterflies. What does this mean for humans?” and links to a WSJ blog that asks “This begs another question: Does that mean there will be an increase in deformities and mortality rates among humans also?” Never mind that nothing in this research demonstrates issues for humans. Research on the effects of low level radiation exposure is very contentious and inherited damage hasn’t been found in offspring of survivors of the Hiroshima and Nagasaki bombings (which surprises me and doesn’t directly relate to Fukushima anyway).
Tweets linking to some news story with the text “mutant Fukushima butterflies” are pretty typical of how this story is being shared. Mothra jokes are common. I’m not seeing much that is critical or even a bit skeptical though as I got ready to post this I saw a critical blog post on it (though it possibly goes too far to the other side, entirely dismissing the study and ignores the convincing parts).
What’s the Problem?
Honestly, it probably won’t be much of a story in a few weeks. Likely it will be forgotten9. But I think it’s a good example of mediocre science reporting. The study itself is interesting and not fatally flawed10. Some of the results don’t seem incredibly significant, but good science doesn’t need to be (in fact lack of significance is interesting too). However, the news reporting gives the impression the results are much stronger than they are: strong headlines, commentary only from the study authors or those praising the result, no exploration of the limitations of the results (aside from cautioning that the results can’t be “directly” applied to humans), etc. But science just doesn’t work this way. First results are rarely correct in all respects. Peer review doesn’t guarantee correctness or significance. But news reporting prefers a tight, simple story. Emphasizing the ambiguity of these results just wouldn’t get as many page views.
Thanks especially to PlanktonMath and mem_somerville for lots of links and discussion on this story. PlanktonMath first brought it to my attention.
For reference, the study and supplement.
At this time, no one has died specifically from nuclear-related harm due to the Fukushima accident though some workers have exceeded regulatory lifetime dose levels and may have future health issues. Many people tragically died but amazingly not from radiation.
Technically, in the supplement figures, it is mentioned that one of the F2 generations was created from F1 parents with eye abnormalities, but there were only 23 eggs laid and none survived. This data doesn’t really tell us how many existed in the F1 or original parent generations.
Here’s another example of bad reporting: the researchers weren’t measuring “mutation rate” which generally means measurements of changed genes. A correct phrase would be “abnormality rates”. For all we know, most or all of the abnormalities are caused by a very small set of genes that aren’t changing much, just spreading thru the population.
Technically these are the sample sizes in Figure 5f which describes forewing size differences. I couldn’t find the sample sizes overall.
“Radioactive tuna” is in fact the phrase used.
An earlier version of the story used “etymologist” rather than “entomologists”.
When breeding a second generation in the lab, the researchers chose male butterflies from Tsukuba to breed with abnormal female offspring of butterflies collected at the other sites. Several of the significant results call out that all sites differed from the result at Tsukuba.
I recorded these numbers on the morning of Tuesday, August 14th but I let the post simmer before posting it and chose not to get current numbers. They’ll be out of date immediately anyway.
Except by anti-nuclear activists who live in a strange world where it’s better to decommission nuclear power plants and burn more coal.
I might harshly criticize parts of the abstract but somewhat unsupportable abstracts are common. I admit I went into reading the paper in an “attacking” mode because of the quality of the BBC report. It seemed perfect for the “poor science feeds incorrect public belief” problem. But it’s got some good stuff. Pity we still have the bad science reporting driving incorrect public belief.