The Problem with Statistics

By Donovan Makus

Did you know that 83.6% of statistics are made up on the spot? Naturally, this false statistic is included in the 83.6%, but the bigger question is, would you believe that statement? So long as the statistical statement passes the “glance test” of being possible, and doesn’t look like it was generated on the spot by being a convenient round number such as 80%, many people will believe the speaker. Studies have shown how statistics that seem authoritative can easily sway readers or listeners. Afterall, “facts are stubborn things, but statistics are pliable.” These words by Mark Twain capture one of the great issues with statistics, not only that their credibility may not stand up to scrutiny, but also that they can be manipulated. This isn’t surprising given that few topics seem to inspire as much fear and be equally as misunderstood as statistics. However, statistics is a key part of our world, and something we should interpret carefully.  

Before examining how statistics can be manipulated, it’s useful to go through a quick primer on the two major types of relationships statistics show. One is the gold-standard, causation. In a causative relationship, you can show that less sleep causes more car accidents in a closed track experiment, or some similar relationship. There could always be another confounding variable–perhaps the people sleeping less are also younger, more inexperienced drivers–but through the power of statistics and good experimental design, you can show something causes another thing. Correlation is the other major type of statistical relationship; a weaker relationship than a causation, this one cannot show anything beyond the fact that two variables are related to varying degrees. An example would be a country-wide survey asking about sleeping time and car accidents. Here, without the experimental design, you cannot conclusively say that less sleep causes more car accidents. Understandably, these kind of studies are easier to complete, and are still useful for statistical relationships you can’t experiment with; you can’t randomly assign kids at birth to different parents for an experiment, but you can survey them and their parents through the years. To show that there is a correlation or causation, scientists often use “p-values,” which reflect the probability of their results occurring by chance. P-results below certain thresholds indicate significant results; typically, a probability of 5% (0.05) is used, although cutoffs vary by field. Through all these tools, scientists can state that their findings are not merely random. 

Causations and correlations statistics form the basis of the natural and social sciences. Does a drug adversely affect the population size of sea shrimp, or is the slightly smaller population a result of perfectly normal random fluctuations? Through the power of statistics, the results can be shown to be not significant, or significant at 90%, 95%, or even 99% levels. While this by itself is incredibly useful, the power of modern computers has led to a new era in statistical analysis. While previously, statistical analysis was difficult to perform on a mass scale, and, as a result, focused on the main question of the research, scientists can now use the power of mass statistical analysis computer programs to search large datasets for statistically significant relationships between thousands of variables, finding only significant ones, which they then include in their final reports. This can lead to some truly hilarious “correlations.” For instance, did you know that from 2000 to 2009 in the United States, the number of people who died after becoming entangled in their bedsheets and the per capita cheese consumption is 94.71% correlated? Or that there is a nearly perfect correlation (99.26%) between the divorce rate in Maine and the per capita consumption of margarine between 2000 and 2009? These are statistically significant correlations, courtesy of Tyler Vigen’s spurious correlations website, which includes many more examples, but these statements hardly show anything. I would challenge anyone to explain how divorces and margarine are interrelated.  

While these examples seem almost funny, scientific studies have fallen prey to this same sort of issue. In one study, New Zealander scientists found that the owners of dark-haired cats suffered more allergies than owners of light-haired cats, a finding that doesn’t make sense since the functional difference between the 2 palettes isn’t related to allergies. The use of “p-hacking” has further clouded the scientific publishing world. By experimenting with different statistical tests, after collecting data, scientists may be able to create statistically significant findings by changing their experimental measurement endpoints. Do particular American political parties’ power affect the economy? Not if you use stock prices, but if you switch to inflation instead, you may discover a statistically significant relationship. Another approach is to modify the significance level; if your cutoff is the common 0.05 and your p-value is 0.07, why not raise your cutoff to the 0.1 (90%) level if you can, or at least report the statistically significant results at the 90% level? This problem is broad, from the natural sciences to the social sciences, and it is also serious, since some of these studies are used to make broad-reaching policy decisions.   

While scientists may engage in some questionable statistical manipulation, this issue isn’t a reason to give up on scientific publishing or methods. Publishers and leading scientists are aware of the problem and taking steps to address the issue, ranging from requiring authors to report how they analyzed their data, to requiring pre-study outlines of the study design, questions to be statistically tested, and cutoffs for a significant/not significant result. These are good changes, but the real lesson here is to be perpetually sceptical of anything you read, particularly if it tries to use scientific tools to sway your opinion. Everyone has their biases and agendas that can potentially appear in their articles and statements. Simply asking the “why” question, be it “why is this true?” or “why did the author include this information?” goes a long way in helping find the weak links in false information. The amount of data available in today’s day and age is overwhelming, and it’s easy to see a headline saying “Scientist says…” and take it at face value, without confirming the actual methods used, particularly if it seems to confirm our preexisting beliefs. While no one has the time to comb through the research behind every news article, being aware that sometimes, studies are not as sound as they first appear in a carefully-written press release or news article, is an important part of being an engaged, aware, citizen.   

Leave a Reply