Too many anecdotes, not enough data

I love this quote I read recently:

The plural of anecdote is not data.

Suw Charman-Anderson wrote this in reference to a story published last week about a 15-year-old intern at Morgan Stanley who wrote a report on teen’s use of technology. The report got tons of press from the likes of Bloomberg, the Financial Times, and the Guardian, in part for statements like this:

“Every teenager has some access to the internet, be it at school or home. Home use is mainly used for fun (such as social networking) whilst school (or library) use is for work. Most teenagers are heavily active on a combination of social networking sites. Facebook is the most common, with nearly everyone with an internet connection registered and visiting >4 times a week. Facebook is popular as one can interact with friends on a wide scale. On the other hand, teenagers do not use twitter. Most have signed up to the service, but then just leave it as they realise that they are not going to update it (mostly because texting twitter uses up credit, and they would rather text friends with that credit). In addition, they realise that no one is viewing their profile, so their ‘tweets’ are pointless.”

My emphasis added. You can read the full report here.

As Suw points out, while this report is interesting, it’s completely anecdotal. The problem with anecdotes is that they’re stories, they’re generalizations that trick us into thinking they’re data. At worse, we use them to replace data. They might be based on a single data point, in the beginning, but by the time they’re told again and again to a growing network of people, their influence grows way beyond that initial data point (assuming there was a data point to begin with). They lead to statements like “Teens do not use Twitter”…which might be based on some initial data that this kid and his friends don’t use twitter (we don’t know) but as a statement it is simply not true. There are teens who use Twitter. What often happens is that an anecdote like this gets passed around and around (we’re all guilty) until people start accepting as fact that all teens don’t use twitter.

(as Chris Fahey says in the comments: “you cannot take an anecdote, pluralize it, and act as if you have tons of data”)

And as designers who need to make decisions based on data, not anecdotes, we need to watch out for this sort of thing. If we’re not careful we’ll start making design decisions based on a single person’s opinion, not fact. While anecdotes can help guide us toward the right questions, the answers we ultimately use should come from data.

Sometimes it seems like we are swimming in anecdotes. Anecdotes are the currency of design, the primary exchange that we pass back and forth with each other in order to sway hearts and win minds.

I recently watched a movie called He’s just not that into you, based on a book of the same name. There was a scene in which the main character Gigi, a woman who was trying to find Conor, a guy she had had one date with, was told by Conor’s friend Alex that he just wasn’t interested. If Conor was interested, Alex says, he would have found a way to call her. Since he didn’t call her he probably didn’t want to.

Gigi responds with a story she heard of how a friend of a friend started off in a similar way, but that they ended up back together again somehow, and lived happily ever after. She saw the circumstances similar to hers…it could just be a case of Conor losing her number or not having a way to get in touch with her. In Gigi’s view she could realistically replicate the success of the story of the friend of a friend.

Alex tells Gigi that she’s living her life around exceptions, not rules. The rule is that if a guy doesn’t call, he’s not interested. End of story. And the data in this case is that Conor didn’t call. The exceptions are fantastic stories of lost phone numbers and such. His recommendation is to forget about Conor because he is the rule, not the exception.

In other words, Gigi had an arsenal of anecdotes that she used to get her through life, while plainly ignoring the data she should have been looking at. If she only remembered all the times before that guys didn’t call, and it didn’t work out, then she would have known what the result was going to be. Instead, she focused on the exception and not the rule, the anecdote and not the data, the story and not the reality, and she ended up in a fanciful world.

Is this not true in many parts of life? We take shortcuts, we tell anecdotes, instead of looking at the data. In the design world this happens all the time. People extrapolate from their own experience or stories they’ve heard about using the web, generalize it, and share it with others. That’s how anecdotes happen. But very rarely can we make concrete decisions based on them.

So the next time someone tells a wonderful story that sounds a tad too good to be true, ask yourself: “is this an anecdote, or is it real data?”.