But let’s not confuse “anecdote” for “narrative” or summary. “data” can be just as easily manipulated as “anecdotes”. It is the play between our observations & our human response to those observations that make us human, and allows us to bring our humanity through to execute on constructed visions into designs.

I’m not saying you aren’t just cautioning us to be more analytical and rational (which is not a bad thing) but this piece taken alone does seem to be suggesting that the story is less important than the data or experience behind its creation.

Being an anthropologist by study/background, it is the experiencing not the data collection that makes observational research so powerful and important for decision making.

I totally disagree. THe plural of anecdotes absolutely is data. Even one anecdote is data.

In your example, Alex is only slightly less guilty than Gigi of using anecdotal data to form a conclusion. Her knowledge of the one-call rule can only be anedotal.

In many ways, Alex is actually worse because she has used her idiosyncratic anecdotes to form a hard-and-fast rule, whereas Gigi is keeping an open mind because she knows that exceptions are possible. Gigi might even have other anecdotal data to work with (i.e., she felt sparks between her and Conor that would transcend any rule).

“The plural of anecdote is not data” is itself a kind of folksy anecdote, one not supported by the mountains of amazing qualitative research whose entire foundation is essentially nothing but piles of anecdotes.

Okay, let me offer you a way out of this, Josh! I think what you wanted to say is “You cannot take a single anecdote and pluralize it to pretend you have a ton of data.”

In other words ONE anecdote is only ONE data point.

Thanks for the pushback, guys. I’ve updated the post to reflect your concerns…making it clear that I’m saying that in practice I think we lean on anecdotes much more than we should, even if they are based on a single data point.

The other troublesome thing about anecdotes (in addition to our penchant for treating them as data) is that over time they compress a rich, embedded, textural high-resolution experience into a simple, decontextualized snippet, stripped of all its rich, overdetermined and complex aspects.

One of my favorite movies, Six Degrees of Separation, has a great scene toward the end where Ouisa Kittredge (Stockard Channing’s character) says of their recent experience with the class-jumping Will Smith: “I will not turn him into an anecdote, it was an experience. How do we hold onto the experience?”

Indeed – an anecdote is typically used to illustrate an already held conviction, and simplify or abstract away from the raw data which might be more difficult to understand – and more human.

I think the reason these studies get so much press is just a combination of laziness by blog authors and a desire to write catchier, more provocative headlines.

“Teenagers don’t use twitter” will get more reads in your RSS feed than “Many suggest that adoption of twitter by those under 25 significantly lags their adoption of other social computing technology.”

Just doesn’t have the same ring, does it?

I love this quote too. Could we make t-shirts of it, you think?

@john…well put, and great quote! The reality is often much, much richer than the resulting anecdote.

@john you are spot on. At the end of the day too many bloggers, who are otherwise great writers and very independent & unbiased, let SEO decide the title of their posts.

It’s no different than writing a book and letting the marketing/media outlets determine how it will be perceived.

Putting “visibility” over “viability” it seems.

If something does not get read does that mean it’s not a useful work? Maybe. But “tidying” up your titles just to beef up your SEO so you’ll get read, or get comments isn’t the way either.

When we reduce everything to a crisp little sound bite like @john’s example above not only is the soul of the experience stripped away, but a sentiment, story, comment may go “viral” (such as “Teenagers don’t use twitter”), thus destroying any chance of capturing any viable data.

The problem, as I see it, is that Suw’s quote — although it is a clever statement — is already based on the value judgment that data is “valid” and anecdotes are not. I think this confuses the issue. An anecdote is a data point – multiple anecdotes may agree with each other in terms of their conclusion, but this obviously doesn’t equate to statistical validity. But since when is statistical validity the standard for design research? I agree with @chrisfahey, that you need to be careful not to extend your logic to the point of requiring designers to base their work only on quantitative data. What’s wrong with this: by definition, quant can never deliver the richness of a person’s opinions and experiences that a good designer needs to connect with. Quant is structurally incapable (in any cost-efficient manner, at least) of providing this input.

So we are left with anecdotes, opinions, experiences, conversations… Is that so bad?

Interesting post with a good deal of relevance to not only design but research as well. As I began to say in my tweets, it’s important to remember that quantitative data is interpretive. I fear when we start to think of it as agnostically factual that it distorts our appreciation of it. All response is based on stimulus, whether internal or external. And unless we’re talking about purely chemical responses, it is variable. Change the location or the lighting and the data changes, change the weather or the room temperature and the data changes, change the researcher or testing time of day and the data changes. How often is this variability accounted for or noted in your research?

This is a basic tenet of psychological research: intervening variables are legion. So while we as members of the Western world prefer to reduce our data into bite sized chunks, let’s keep in mind that our tests might have returned different results if it hadn’t been raining that day, or if our subjects hadn’t had a bit of a headache, or if we weren’t worried about our child being sick, or if the tester hadn’t just gotten that good bit of news. Psychological studies show that moods affect performance, and as a race, we’re a pretty moody bunch. So while we’re busy ticking off boxes about what was easy or hard, keep in mind that there’s a whole other level that is usually not accounted for. And that level is qualitative variability.

Your last point regarding focusing on an exception rather than a rule reflects a recognized psychological phenomenon. People do tend to rely on anecdote in their everyday lives and they also tend to find the anecdote that suits their personal preference. Dan Ariely covers this quite nicely in his book, Predictably Irrational, Chapter 9, “The Effect of Expectations: Why the Mind Gets What It Expects.” I suggest that this is why Apple doesn’t do focus group testing. (But actually, focus group testing is problematic for a variety of reasons, which I’m happy to discuss in greater detail elsewhere.) Testing tends to reduce information and design to a low common denominator; this is not the locale from which dramatic and influential change occurs. This is the most difficult aspect of testing. In an old notion but a true one: humans are resistant to change, even when it’s for the better (covered in Ariely’s book mentioned above; Drew Westen’s book, The Political Brain; and George Lakoff’s book, The Political Mind).

So ultimately, while an anecdote does not constitute a complete data set, it is vital from a human perspective, primarily because we respond and remember things from a narrative standpoint and not a data-driven one. What all of this means is that quantitative data is almost useless without interpretation (performed by moody humans) along with some explanatory qualitative data that serves to frame it.

The anecdote trap is well-known in social psychology. It’s called the availability heuristic. Loosely put: human beings tend to look for readily-available evidence that supports what we already believe.

This is closely linked to another concept, the representativeness heuristic. Certainly, a 15-year-old who is interning at Morgan Stanley is not representative of 15-year-olds, let alone “teenagers” at large.

This is why Chris is wrong on this one. True, anecdotes are a kind of qualitative “data” — but only insofar as they provide information about the person telling it.

As a designer, whenever I hear an anecdote about use, I immediately think of alternate scenarios and consider who might represent them. Ideally, then you go out looking for those people and hear their stories.