OUR BLOG

Thinking About Statistics

Written by Amy Gyoba on February 2, 2012

Statistics are everywhere, whether we’re talking about crime rates going down or high school graduation rates going up. It’s no different when it comes to websites, where we track metrics like page views, conversions, and bounce rates. Although we’d like to believe that we’re savvy about the numbers, we may not be as adept at interpreting them as we think. In his book Thinking, Fast and Slow, Daniel Kahneman explains why.

One Mind, Two Systems

Kahneman characterizes our two thinking systems as fictional figures called System 1 and System 2. System 1 operates automatically to quickly generate complex patterns of thought. We use this system when we perceive that one object is farther than another one or when we detect an angry tone of voice. Its capabilities can be innate, such as recognizing objects, while others are learned, such as reading and understanding social situations. Whether they’re inborn or acquired, System 1 thinking requires little or no effort on our part.

In contrast, System 2 thinking is deliberate. This system handles operations that require our attention, and we don’t perform them well if we’re distracted. We use this system when we focus on one person’s voice in a crowded and noisy room or when we search our memory to identify a surprising sound. System 2 thinking involves making choices and deciding what to think or do.

The Division of Labour

Although they function in different ways, they work together to make sense of the world. Both systems are active when we’re awake; System 1 runs automatically while System 2 operates in low-effort mode, using a fraction of its capacity. System 1 generates suggestions based on impressions, intuitions, and feelings. System 2 will usually adopt these suggestions with little or no modification. Most of what we think originates with System 1, but System 2 has the final say. However, when System 1 runs into a situation that doesn’t fit into the model of the world it constructed, System 2 is activated to help by searching our memory to find a story that explains the anomaly.

This division of labour works well most of the time because System 1’s models of familiar situations and short-term predictions tend to be accurate. It’s good at automatically identifying causal connections and events—even when it’s not a solid connection. System 1 suppresses ambiguity and spontaneously constructs stories that are as coherent as possible, and System 2 will generally accept that answer without questioning it. This story construction can work to our disadvantage in considering statistics.

The Wrong Focus

We tend to focus on the story rather than the reliability of statistics. In one example, Kahneman cited a study of kidney cancer rates in 3,141 counties in the US. The counties that had the lowest rates were mostly rural, sparsely populated, traditionally Republican, and located in the Midwest, the South, and the West. If we try to explain this pattern, System 1 brings up facts and associations that support an explanation for System 2. We would probably examine the rural characteristic because we associate it with clean living (access to fresh food and no water or air pollution). This explanation makes more sense than trying to explain the lower rate in terms of political views.

However, the counties that had the highest kidney cancer rates also had the same characteristics (rural, sparsely populated, Republican, same locations). Again, we might focus on the rural lifestyle, which might mean poor access to medical care, a high-fat diet, and too much tobacco. For both situations, we look for a story that explains the cancer rate. We’re so focused on explaining how a rural lifestyle contributes to a lower or higher cancer rate that we miss the fact that these areas are sparsely populated; as Kahneman points out, smaller sample sizes tend to yield extreme results more often. Rather than questioning the validity of the statistics, we skip over that to try to explain the results.

Objective Results

So what does this mean for you? When it comes to your website metrics, do some preparation before you gather data. Here are a few tips to get you started:

1. Define your goals. The information you’re looking for will determine which metrics you’ll use. Are you trying to increase conversions? Do you want to make sure that your site is user-friendly? Are you looking for more information about your site visitors?
2. Decide which metrics to use. Before you begin measuring, make sure the metrics will answer your questions. For example, if you want to test the usability of your site, you should consider metrics such as the time taken for a task, the error rate, and the user’s subjective satisfaction.
3. Pick a tool that meets your needs. There are plenty of options out there, so do your research to find the best fit for your goals.
4. Make sure your statistics are reliable. Before you try to explain the statistics, make sure they have a solid basis. Are you looking at the right metrics? Is your sample size big enough? Is it composed of unique site visitors?

Remember that website statistics aren’t necessarily objective because you can interpret them subjectively based on your prior experiences and associations. If it’s your own site, you may be too close to what you’re measuring to analyze it. For example, you may be too invested in a design to objectively evaluate the metrics for the calls-to-action. Bringing in outside help to assess your site’s analytics can help ensure that your metrics are being measured and interpreted properly.

Image Credit: TEXample.net