Randomness and Samples: Why that GM corn study isn’t as scary as you think

This past week a new study came out finding a connection between glyphosate-tolerant corn and cancer (and also Roundup and cancer). You can read more about the study at the BBC or this Discover post. But here I want to expand on the main reason I was immediately skeptical: the small number of control animals compared to experimental animals. I struggled with how to express what seemed obvious to me (my statistics training is scattered) and a person I follow on twitter put up a post which mostly got there. I later made a comment on a Mother Jones story about the study in a similar fashion. I want to elaborate on that here to hopefully make it clear that (perhaps surprisingly) interesting (or scary in this case) data can actually just be random.

Study Background

First, a quick background on the study (PDF). In this study, 200 rats were raised in various experimental groups. Half the rats were male and half were female and they divided both sexes up into similar treatment groups. Ten rats of each sex were controls: they ate normal diets (rat chow which is usually a combination of corn, soy, and supplements) and drank normal water. In the control cases, they actually ate 33% conventional (non-GM) corn grown by the research team in their rat chow. Three groups of ten rats for each sex ate different amounts of a particular variety of glyphosate-tolerant corn. Three groups of ten rats for each sex ate different amounts of glyphosate-tolerant corn that had been grown in a field actually treated with Roundup¹ . Finally, another three groups of ten rats ate the regular (control) rat chow but were only given water spiked with different amounts of Roundup. To make this clearer, here’s a table.

Remember, for each sex there are ten rats per group. They then raised the various groups for around two years, keeping track of which rats seemed to get tumors. As this particular breed of rats very commonly gets tumors as it gets older the control groups showed tumors. The researchers recorded when animals died (and in some case euthanizing them²). Their graphs sometimes suggest greater effects for treated groups than control but one reason many scientists look at this study with skepticism is that many treatment groups did better than the controls³. Basically, it’s not clear that the results are consistently showing worse effects from treatments — this might just be a case where tumor-prone rats get tumors and a non-rigorous analysis of the data would suggest a problem.

Simulating Rats Prone to Tumors

Andrew Kniss in his post noted that this rat variety is extremely prone to tumors. He used a study that showed 72% of female rats having tumors by two years old. Others have seen figures around 50% or 60%. He then used that probability of developing tumors to generate simulated groups of ten rats — control groups and experimental ones as in the study. In my comment, I used a figure of 20% because the current study saw anywhere from 20% (males) to 60% (females) of the control rats developing tumors. Using some R code (a statistics-oriented software development language), I then generated several sets of ten groups of ten rats to simulate a control group and the nine experimental groups. The R code is this:

colSums(matrix(sample(c(0, 1), 90, replace = TRUE, prob = c(0.80, 0.20)), nrow = 10, ncol = 10)) 

That code may look complicated, but it’s not. What is does is first generate ten groups of ten random numbers either a 0 or 1. A 0 means no tumor and a 1 means a tumor. The probability for no tumor is 80% and for a tumor 20%. Each group of ten is then added up to give the number of rats with tumors in each experimental group. Remember this is all simulated. I’m just having a computer randomly generate how many rats in a group of ten will get tumors based on an expected probability. I ran this a few times and here are five runs I got. In some sense I am simulating the study experiment five times using random numbers.

I’ve labeled the columns in the above table as with the experimental groups in the original study. So pretend the first column in each row is a control group. Then if you treat the rest of the entries in that row which represent our experiments, it certainly looks like our experimental groups may have done “worse” than the control. For example, for run #2, only one control rat got cancer but the experimental groups 1, 2, 3, 6, 7, and 8 all had more rats with cancer. But this is just randomly generated data. Now for a moment, let’s rethink this and treat more of our groups as controls and not experiments. Pretend that E2, E4, E6, and E8 columns are also control groups and leave E1, E3, E5, E7 and E9 as experimental groups. All of a sudden the data look a lot more random, as it should. Statistical tests let us decide how many control and experimental subjects we need to have enough “power” to determine whether our results are due to actual effects and not chance (and even then, usually we can only say with 95% or 99% confidence the results aren’t due to chance).

Low Tech Rat Study Simulation

But not everyone is a programmer and wants to run programs that will generate random numbers. But you probably have dice or coins. Being nerds, we have a lot of dice. You can roll some six-sided dice (D6s) or some ten-sided dice (D10s):

Actually, you can roll any kind of dice. I rolled D10s (my ranger in my first dungeons and dragon game used D10s for damage rolls). Ten-sided dice have sides labeled with 0, 1, 2, up to 9. For our simulation, we only need 0 or 1 values so I’m going to treat 0, 2, 4, 6, and 8 as a zero and 1, 3, 5, 7 and 9 as a 1. Odd is a tumor, even is no tumor. This changes the probability to 50% for a tumor but is a lot easier to count quickly (and is actually closer to known tumor rates for this breed of rat). Here’s my first roll which is going to be the number of tumors in a group of ten rats for a control group:

And here’s a few full runs. You can see why I wanted to roll ten dice and just wanted to add up the number of odd numbers.

At this point I got pretty tired of rolling and counting.

But I hope you got the idea: with only one group of ten control rats (per sex), it’s really easy to possibly see harmful effects in the treatment groups that we just don’t know are there. That’s why we have statistics. They let us know with some likelihood how much data we need to collect to know whether we’re seeing true effects. In this study, the authors did not show any standard statistics that would demonstrate one control group of male (or female) rats is enough for nine other groups of experimental rats — and most independent experts are saying this setup just isn’t likely to allow us to rule out random effects. So while this study’s results are admittedly scary, it’s just as likely that the results are random.