Interpreting the Validity of Statistical Arguments

Interpreting the Validity of Statistical Arguments

Interpreting the Validity of Statistical Arguments 

“There are three kinds of lies: lies, damned lies, and statistics” (a remark often attributed to Mark Twain)

Numerical data and statistics are generally regarded as reliable information.

Unfortunately, vested interests often make deliberate efforts to distort statistical information in order to support their own causes. Here are a few examples.

Emphasize and omit

The simplest technique for distorting statistical results is to emphasize particular aspects of a wider statistical study, while omitting other aspects. For example, on 17 February 2006, President Bush (USA) announced, “About twenty-five million seniors have signed up for this new plan since January 1st”, referring to a new Medicare prescription drug plan. The intent was to emphasize that large numbers of seniors had signed up for the plan at that time. However, the Boston Globe reported that the “actual number of volunteer enrollees is much smaller, about five million” (http://thinkprogress.org ). The other twenty million seniors had been automatically enrolled, or already had drug coverage from their former company’s health plan. The figure of twenty-five million was not incorrect, but was misleading in the way it was used.

Select a biased sample

Another technique is to deliberately select biased samples. For example, in 2004 the Motion Picture Association of America (MPAA) conducted a survey of Internet users and published the result that “24% of Internet users download pirated movies”. A criticism of this MPAA finding was later published at a number of websites. The MPAA implied that their finding applied to all Internet users while their survey sample included only users with broadband access (less than 50% of all users at that time).

The MPAA also characterized all downloads of video material as pirated, whether it was or not. The sample was deliberately biased so that the result would lead to a perceived higher percentage of pirated downloads.

Create your own data

In some cases outright fraud is used to produce faulty statistical results. For example, on 11 January 2006, the scientific journal Nature reported a story from South Korea.

The renowned scientist Woo-Suk Hwang had been charged with fabricating data to support his work involving the cloning of embryos and stem cells. Apparently, in two major studies Hwang’s data had been deliberately fraudulent.

There is some comfort to be derived from the fact that all of the above examples were detected because of the inherent nature of statistical information. If anyone carries out a statistical investigation or quotes statistical results of any significance, that information can be checked and verified.

The bottom line is that statistics provides the best-known set of tools for garnering evidence about the state of the human condition and the state of the universe.

However, we still need to use caution when we are presented with statistical results. 

There are some who would misuse statistics to distort information. Those same people attempt to hide their distortions behind the prestige generally associated with statistical methods.