Significance, Magnitude, and Base Rates
Do statins increase the risk of diabetes? That’s the title of a recent article in The New York Times that was undoubtedly read by hundreds of thousands of people with high cholesterol. (Statins are the class of drug that is the most common medication used to treat high “bad” cholesterol.)
The answer, apparently, is “yes.” Multiple studies have, in aggregate, shown that people who take statins develop Type 2 diabetes at higher rates than similar people who take placebos.
But what does that mean? One possible interpretation is that taking statins will cause you to develop diabetes, so you have to decide which you would rather have: blocked arteries or diabetes. That is pretty clearly incorrect.
As people reading this post probably realize, the relationship between statins and diabetes has been shown to be statistically significant. This means, more or less, that the aggregate difference in outcomes between the treatment group (those who took statins) and the control group (those who got a placebo) is unlikely to have occurred by chance. The conventional threshold is that the probability of the difference occurring by chance is less than 5 percent (on which see this famous XKCD cartoon), although researchers always like to point out when the probability is less than 1 percent or 0.1 percent.
We have been trained by a generation of well-meaning data journalists to look for statistical significance. The simplified version of the lesson goes like this: if it isn’t significant, ignore it; if it is significant, it’s real. But that’s not the whole story.
If you have high cholesterol you may be hesitant about taking statins if you read that they are significantly linked to diabetes. But let’s look at the data.
In the initial 2008 study cited by the Times, 270 people in the treatment group developed diabetes, as opposed to only 216 in the control group (p = 0.01, for those who care). That’s an increase of 25 percent (54 / 216), which seems pretty, well, significant.
But there were 8,901 people in each group. That means that only 2.4 percent of the control group developed diabetes, as opposed to 3.0 percent of the treatment group — a difference of only 0.6 percentage points. A 2016 meta-analysis of studies including a total of 91,140 participants found that approximately 4.5 percent of the aggregate control group and 4.8 percent of the aggregate treatment group developed diabetes. (Different studies have different populations, so you wouldn’t expect the baseline percentages to be the same across studies.)
How should you interpret this? If 4.5 percent of the aggregate control group developed diabetes, and you are similar to the average person in the studies, that means your risk of getting diabetes is 4.5 percent — even before you take a single pill. Taking statins increases that risk to 4.8 percent. (The estimated risks are slightly different because of weighting, but that doesn’t affect the overall point.) In other words, the chance that statins will cause you to develop diabetes are 0.3 percent, or less than one in three hundred.
What’s going on here? Large sample sizes give you a lot of explanatory power: you can identify effects that are small but real. So it’s pretty much certain that statins increase the risk of diabetes, at least for some people — but, for most people, the magnitude of that increase is pretty small. And when you’re considering risks, that’s what you should care about. The fact that something is virtually certain to increase a risk (p < 0.0001 or whatever), on its own, doesn’t matter very much. What matters is how much it increases that risk.
In this case, the base rate — the frequency of diabetes in the sample as a whole — is just pretty low. So even if you increase that frequency by 25 percent, as in the initial study, which sounds pretty scary, you’re still only increasing the risk to an individual by about half a percentage point — which most people would probably take in order to lower their cholesterol.
No one is average. You may have reason to believe you are at a particularly high risk of Type 2 diabetes, in which case this may be a risk you want to consider. In general, though, magnitude is what matters, not just significance.
