One of pieces Anne references is Andrew McAfee’s The Ties that Find, a nice overview of the idea of weak ties, which originated with the fascinating work of Mark Granovetter, who wrote the original work The Strength of Weak Ties(PDF) in 1973. Weak ties are relationships we have with people outside our own social networks. We don’t utilize them often, but we utilize them in certain situations to help us with things our social networks can’t. Most importantly, weak ties gives us a perspective outside of the normal groups of which we are a part, whose perspectives tend to become homogenized over time as we learn and become familiar with the people we spend the most time with.

What struck me about Anne and Andrew’s pieces was the implicit idea of the value of diversity. Neither mentioned this explicitly, but for those familiar with James Surowiecki’s work The Wisdom of Crowds, diversity is crucial to wisdom, and thus problem solving. Weak ties helps explain how we continually introduce diversity within our social groups, by periodically leveraging those relationships with people outside our close-knit social networks.

To help understand what was going on, I made a sketch of strong and weak ties:

Click for full-size version

McAfee then makes an interesting point: weak ties are valuable for enterprises who suffer from groupthink. So, while there is lots of pushback on the time wasted by employees using social networking software (many companies prohibit MySpace and/or Facebook behind the firewall), McAfee suggests that under the right conditions social networking software can increase innovation:

“The implication for SNS is obvious: Facebook and its peers should be highly valuable for businesses because they’re tools for increasing the density of weak ties within a company, as well as outside it. My Facebook friends are a large group of people from diverse backgrounds who have very little in common with each other. Furthermore, their profiles give me a decent way to evaluate their expertise. These online friends, in other words, are a large group of bridges to other networks. Facebook already provides me a few good ways to activate these bridges for my own purposes. I anticipate that enterprise SNS (whatever that turns out to be) will have many more.”

The design challenge, then, is to build software that leverages weak ties. The “network” feature of Facebook is an obvious example of a way to organize groups, but does it improve communication between the members within?

One of the most important results of people interacting socially online is that we can measure the effect of social influence. A ground-breaking study by Columbia professor Duncan Watts showed how this could be done. (I wrote it up in How Aggregate Displays Change User Behavior) This is one of the most important studies I’ve seen…it clearly shows a relationship between people’s actions and the aggregate information that’s shown to them in the interface.

For those not familiar with Watts’ study, it showed that when faced with an interface showing what other people did we are definitely influenced by that behavior. If we are shown a list of the most downloaded songs, as in the study, we cannot help but give more weight to those songs downloaded more. We’ll be more likely to download those songs ourselves. This echoes countless studies from social psychology that show how we are affected by our environment.

Dangerous Territory

But one dangerous effect of aggregate displays might not be apparent at first. After we realize that our displays are affecting people, the next question becomes: which aggregate displays do we show and when? This is a question a lot of design teams are grappling with as they build out their social software.

But taking it even further we get into ethical territory. This was made plain to me by a question that someone asked me the other day after we were discussing Watts’ study. They asked: “If people respond to aggregate displays, and change their behavior accordingly as they’ve done in Watts’ study, aren’t those people also in a position to be manipulated?”

Well my first thought was “You rascal. You clever rascal. We’re just beginning to get a handle on how aggregate displays affect behavior and you want to exploit it already!”. But my answer had to be “Yes, people can be manipulated if the data being aggregated isn’t accurate or valid”.

In subsequent talks I’ve had with people, it has become clear to me that many industries not only manipulate aggregate numbers, but they rely on them to drive business. In the music industry, for example, this is common practice. Studios choose which artists to promote, and suddenly they’re everywhere, their pseudo-popularity created in order to generate actual popularity. This is how we have the Britney Spears’ of the world. She’s a talented singer, of course, but she’s not that talented. While she might be in the top 1% of the population when it comes to singers, her talent is not proportional to the marketing and advertising budgets and revenue that she generates, which is in the top .0001%. (FYI: I made these figures up) Britney is a popular phenomenon, but its impossible to tell how much of her popularity is actual fan support and how much is artificially generated.

So there’s a distinction we can make between actual popularity (which has social influence) and artificial popularity (which also has social influence). The first is driven by actually being a fan of someone. The second is driven by bordering-on-unethical advertising.

It is easy to imagine how this distinction might get unethical in a hurry. Imagine, for example, a list of “Most Popular” music wasn’t really the most popular. Maybe someone paid someone else to show their music on the list. Instead of the most popular, it was the most paid for. This is where real danger comes in. We’re starting to see ways of influencing people’s behavior online…but where do we draw the line?

Some people don’t even like actual popularity, as they are aware of its influence and don’t like the rich-get-richer effect, so to speak. But to me actual popularity is fine, for the most part as long as it’s real. If people are passionate about Britney Spears then it’s OK they influence others…they should be able to spread their passion via word-of-mouth. But when that excitement is artificial, then, well, it’s just not telling the truth…

This is especially true in social interfaces that model complex social interactions. In some cases there is just no way around it: human relationships are complex and so whatever view we offer into them must have some complexity as well. That doesn’t mean they should be hard-to-use, it just means that they communicate sophisticated information.

Take the reviews on Amazon.com. For years Amazon’s interface showed the average review, so viewers could tell the general mood surrounding a book. If it was a 5 star or a 1 star book, then that would be instantly recognizable.

But over time it became clear that the rating system had a fault: if the average rating was somewhere in the middle, say 3.5 stars, it was unclear whether it was just a dull book that most people rated as mediocre or if it was a polarizing book that half the people rated 5 and half the people rated 1. A political book, for example, usually polarizes.

So the review interface could be made more sophisticated, showing more information about how the reviews for a particular book were distributed. Amazon came up with a nice interface for this:

I’ve talked to many folks who have made positive comments about this interface. They like seeing more information, and it doesn’t confuse them. Instead, they get a more accurate picture of the reviews than they had before, and that helps them make a more informed decision. More is more, in this case.

If you aren’t familiar with the details, here is a quick recap. ReadWriteWeb also had a nice timeline of events.

There are two ways you can look at this incident whereby Diggers overwhelmed the site by repeatedly (up to two per second) digging stories containing an HD-DVD crack code.

One is as described by Mike Arrington of Techcrunch: Digg Surrenders to Mob. Simply using the word “Mob” makes for great press. We gravitate to mobs because we know they’re messing with the Man. They’re anti-authority, they’re doing what they’re not supposed to, they’re pissed and fighting for their rights. We think of the French or Russian or American Revolution, and we like it.

But maybe, just maybe, mobs aren’t that bad. Terry Heaton had an insightful observation: “What I find most fascinating here is the automatic assumption that chaos is evil. This is a purely modernist perspective, but life itself proves it to be false.” He argues that the so-called Mob was more like the site at its finest…that a Mob is nothing more than democracy at high speed. I tend to agree with this.

The other way to look at the situation is as I described it: Digg Surrenders to Community. The difference is in those two words: Mob and Community. Now, I wasn’t being as calculated as Mike was being, I’m sure, but when realizing the stark contrast afterward it occurred to me that you either acknowledge the voice of the people on Digg as a group, or you do not. You either view them as a passionate Community, or you view them as a anarchic Mob.

In the subsequent days we have learned more about what happened. Digg CEO Jay Adelson was interviewed by Jason Calacanis (well worth the listen) and explained the incident from their view. He says: “”what we saw on Tuesday night was how far you can go…what the limitations of a democratic site can be and in this case it was pretty clear that no technology I could come up with and no amount of people I could hire could solve the problem”.

This story is amazing on several levels. It can teach us a lot about designing for community in an age when that community is completely dispersed around the Web.

A Community Protecting Itself

The digg community practiced what only relatively mature groups can: they acted to protect the group. The group they acted to protect was the one whose members have votes that are not censored, but counted. Clay Shirky, in his classic piece “A Group is its own worst enemy”, explains the phenomenon:

“here’s this very complicated moment of a group coming together, where enough individuals, for whatever reason, sort of agree that something worthwhile is happening, and the decision they make at that moment is: This is good and must be protected. And at that moment, even if it’s subconscious, you start getting group effects. And the effects that we’ve seen come up over and over and over again in online communities.”

The group effects in this case, of course, was overwhelming the site with diggs. This hadn’t happened before and was remarkable because all of the members were acting together…separately.

Setting Legal Precedent

Despite its incredible growth and fame, Digg is a very young company in an undefined space. We do not yet know how user submitted content will fare in the legal system, as there are few precedents to work from. The biggest question in the Digg case is this: Is Digg responsible for the content submitted by its users? Can Digg be held legally responsible for the crack code on their servers?

The easiest way for Digg to escape their situation would have been to kill it, shutting down the site until the wave of protest died down. Adelson explains their thinking…(we) “would remove it to protect ourselves, taking the safe way out”. Undoubtedly, this is what the vast majority of site owners will first think or be recommended by their legal team. They don’t want any trouble if they can help it.

But on second thought Digg realized that their adherence to the cease-and-desist was lame given the nature of their site. They run a so-called “democratic” site whose content is dictated by the will of the people. If the people want a certain crack code on the home page, who is the site to change that? In their case censorship is as bad as, if not worse than, blindly following a cease-and-desist, especially one concerning DRM.

But if there is legal action in this case, it will set an early precedent for all future decisions concerning user-generated content.

Overcoming Software Limitations

99% of the time we accept the limitations imposed on us by the software we use. We accept that we can only have so many programs running, that we can only share iTunes songs on 5 machines, that we can’t talk between IM services easily. For the most part, we’re just glad to be able to do what is offered because it is better than what we had before. In other words, the design of a web site usually dictates the behavior that happens there.

But there is a small population of people, entrepreneurs and hackers, who routinely fight these limitations. These people are incredibly important because despite their small numbers they lead a much larger community to water.

In the case of Digg, however, the community as a larger whole challenged their limitations. That’s what revolutions are all about…they are magnificent because they don’t happen all the time and they usually happen for a very good reason. In this case the Digg community needed to remind Kevin Rose and Co. that they were the valuable editor, not whomever was taking down the code posts.

For the first time, Digg users fought through the technical and editorial restrictions to take full control of the site, if only for a short time, overriding the site owner’s ability to do anything other than shut down the service or let it go on unchecked. There was no way to slow down or stop the submissions without shutting down the site…there were just too many of them. For a short time on May 1, 2007, Digg users fought for and won the Digg.com domain.

What are the Design Implications?

One thing became very clear in the Digg incident: very few people who posted the HD-DVD code knows how to use it to crack a DVD. And that’s an important point because this wasn’t about the necessity of posting a particular code…this was about the community’s insistence that they not be told what they could or could not post. Once you give democratic power, once the community has had a taste of voting, they will not give it back. If you live by the digg, you die by the digg.

Now, there are two things we can potentially learn from this event. One is that you need tighter controls over the content people submit to a site, increasing the limitations imposed by software. We could design our social software with throttling, for example, so that in times of crisis we can slow down submissions in order to filter them easier. This would make it easy for developers to write tools that allow editors to simply remove any submission with a certain string, for example, and it would have time to work.

The other thing we could learn is that this behavior was appropriate and desirable in a healthy community. Imagine if, despite the attempts of thousands of people to submit the hacker code, Digg was able to squash the revolution, so to speak. What would have happened? Well, most of those folks would then have felt like their voice wasn’t being heard, that Digg was in complete control, and that their so-called democratic site wasn’t very democratic at all.

This is the definition that got me interested in Web 2.0 in the first place. It makes sense, easily contrasts with “desktop as platform”, and is accurate: we are seeing a tremendous platform move to the Web.

Unfortunately, sometime after Tim used this definition way back when, it went haywire and eventually ended up meaning nothing more than the Web itself. And really, that’s all it is…just a trend on the Web. In addition, O’Reilly went the VC route, focusing on business people while alienating technologists.

I don’t harbor negativity for someone who has a meme that helps people understand what’s going on. Ajax, web standards, SAAS, P2P, and other things are all figments of the imagination…they’re just other words for technologies that do certain things. And holding events is fine, too. People make the choice to come, let them come. Everybody has a flag to fly.

So, I applaud Tim returning to the original definition, after all this time. Don’t try to be everything to everyone.

However…it’s still not nearly as compact as it *could* be, and it’s not really a business revolution…it’s a technological trend.

In addition to “leveraging” this or that, how about focusing on building stuff that people love? Could that be part of all this? Or does it have to be about “network effects applied to user contribution”?

My guess is that if you focused on one and not the other, you’ll be much more successful than vice-versa.

Cnet’s Elinor Mills, in a piece describing Digg rigging on a wide scale, writes:

“dubious Internet marketers are planting stories, paying people to promote items, and otherwise trying to manipulate rankings on Digg and other so-called social-media sites like Reddit and Delicious to drum up more links to their Web sites and thus more business, experts say.”

This shouldn’t come as a surprise, at this point. (see Brian Clarke’s writeup) Digg’s gaming issues have been widely known for many months now. In Digg’s Design Dilemma, I pointed out that the design of the site had a lot to do with the gaming going on there. But, I should add, it also had a lot to do with the growth of the site…so there’s no clear answer about what to do.

It’s becoming clear that any successful site sees its share of gaming. Even Del.icio.us, who refused to comment on the Mills story, has seen gaming. And this is the way that media has worked for a long time. Who controls the media controls the story. That’s why we’re seeing so much SPAM/gaming…we should expect it in any successful social site.

Real money changing hands

It’s getting to the point where real money is changing hands to game Digg. Mills writes:

“Companies charge as much as $15,000 to get content up on Digg, said Neil Patel, chief technology officer at the Internet marketing firm ACS. If a story becomes popular on Digg and generates links back to a marketer’s Web site, that site may rise in search engine results and will not have to spend money on search advertising, he said.”

I think the observation is interesting, the inference shaky. There are people who think that all they need to do is to game Digg for their advertising? Is there any example of a company who survives on Digg gaming alone? I doubt it.

(Update…Neil Patel, quoted in the story, blogs about how Mills has inaccurate info)

In addition, Mills quotes Barry Parr, who makes a controversial comment, to say the least:

“Digg and others are working hard to deal with this kind of abuse,” Jupiter Research analyst Barry Parr wrote on his blog this week. “But until it is eliminated, the credibility of social-news sites will be in question.”

Credibility in Question?

Hmmm…let’s compare this to another abusive environment where we might have questions about credibility. I wonder if the lobbying in Washington is hurting the credibility of the U.S. government. Until lobbying by Big Oil and Big Pharma and Big Insurance goes away, the credibility of our government will be in question. Ha! It may be true, but that doesn’t mean it will be going away any time soon, or that over time it lessens…at this point most people simply accept evil interests as part of the deal. I think maybe we should accept that SPAM will be an ongoing problem for social sites…after all, they’re social sites.

So, let’s agree right now. The credibility of social news sites will always be in question, OK? And, for that matter, so it should be for non-social news sites!

Nobody said that putting the “social” in play on web sites would mean an end to corrupt behavior. In fact, the opposite may be true. As we model real-life behavior better and better online, as we open up communication channels and people increasingly live their social lives online, we’re going to model every part of those lives…the good and the bad.

What to do about it?

But what can we do about it? I think the answer comes from the community of the site. At some point you cannot rely on algorithms to do your SPAM harvesting for you, you have to rely on the wisdom of real people, because SPAMmers are really smart. Hopefully, crowds are smarter.

To this end both Digg and Reddit claim to have strong communities that self-police. They help drive out SPAM when they recognize it as such, burying stories that don’t seem right. Digg CEO Jay Adelson says:

“There is technical information that only we could know that flags us when someone is attempting to manipulate (stories and rankings)…By merging the algorithms and the people I believe we have a foolproof system.”

“Foolproof”?…I wonder what he means by that…

James Surowiecki, author of The Wisdom of Crowds, has written a great piece: In Praise of Third Place, which details Nintendo’s innovation while being in third place in the gaming industry…a position they fell to after their initial system was superseded by the Playstation (Nintendo…that system we all know and love…Contra anyone?)

This piece brings two thoughts to mind.

First, it could almost be written about Apple…innovating to make the best-of-breed hardware and software with market share a lower-level priority…not trying to be everything to everyone. The moment market share becomes top priority, you slowly erode the magic that is attention to detail.

Second, it shows how great a writer Surowiecki is. He’s more than relevant, he’s leading the discussion, not following awkwardly, a criticism some folks like to levy against traditional publishing outlets. As blogging matures, the criticism is growing, led by Mike Arrington, and for many publications Mike’s right, they are slow and awkward. But Surowiecki is a true exception…he’s a better writer than any blogger I know of, and he writes current, interesting pieces on a regular basis. And, his book, as well as any other, has become a critical work in figuring out and explaining the networked world we live in.

Ok, so I got into hot water for my Thoughts on the Impending Death of Information Architecture post…

But I’m completely fascinated by this subject. In that piece I referenced a work by Elaine Petersen entitled Beneath the Metadata: Some Philosophical Problems with Folksonomy. Elaine eloquently argues that since tagging systems can contain incorrect information (non-Aristotelian, she calls it brilliantly), they will eventually fail to serve our needs. She says:

“Although folksonomy advocates are beginning to correct some linguistic and cultural variations when applying tags, inconsistencies within the folksonomic classification scheme will always persist. There are no right or wrong classification terms in a folksonomic world, and the system can break down when applied to databases of journal articles or dissertations.”

This argument, as I’ve mentioned before, is one about relativism. Is it OK to have systems which contain misinformation, even if it happens to be the way someone thinks and tags?

Let me put it more bluntly: Do people have the right to think how they want?

If we re-ask the question in this way, the answer is clear. (And no, I don’t think it’s ridiculous to equate this argument with allowing people to think what they want. At some level it *is* about that, in a weird science-fiction way)

So, of course we have the right to think what we want, at least most people think so. (insert analogous religious argument here about actions and beliefs)

Anyway, if you’ve read Bokardo for any period of time (go here to win prizes) you know that I believe our systems should model our behaviors and thoughts, not the other way around. We shouldn’t have to map what’s in our head to some other idea set every time we use software if we don’t have to.

If I want to tag the New York Yankees as “the best team money can buy”, and someone else thinks that’s just plain wrong, then tough for them. That’s how I want to tag it, that’s how I want to re-find it, and that’s how I think about the Bronx Bombers (or was it the Yankees?). In folksonomies the view of the system is *my* view…warts and all.

Moreover, other folks in Red Sox Nation might tag it similarly, thus propagating the potential falsity in the system for Yankees fans to find (except, of course, the Yankees are the best team money can buy). Note, though, that their version of the system will have their version of tags for the Yankees…we still have a problem, according to Elaine…there is information in the system that doesn’t agree with other information in the system.

Geez…sometimes I don’t even agree with myself.

Scale is the Great Equalizer

But the thing is, and this is where Elaine underestimates folksonomies, scale matters. Even if a few people tag things incorrectly, most people won’t. This doesn’t have to do with the fact that most people are Good, it’s just that if we ask enough people the same question or have them observe the same phenomenon, where their experiences overlap will tend to be the reality of the situation.

At this point, we could go many ways with this topic. One way would be to tie in James Surowiecki’s brilliant book The Wisdom of Crowds, which makes a lengthy dissertation on the subject of aggregating individual viewpoints. If, under certain conditions, we aggregate the individual decisions of many people, the result tends to be equal to or better than an expert’s view. Here’s the Wikipedia entry for the Wisdom of Crowds, which gives a quick but good overview, and is no doubt a great irony in and of itself…(the crowd writing about the Wisdom of…itself…in a relativistic system with no authoritative voice except the accumulated voice of all its members)

Another way we could go with this topic is where Dan Stewart went. Dan, commenting on Dave Weinberger’s lengthy reply to Elaine, points to another, relatively important document Bokardoans should all be familiar with by now (I’ve talked about it enough):

“Elaine makes the argument that if an item on the web is tagged with words that do not describe it, then the system breaks down. In The Anatomy of a Large-Scale Hypertextual Web Search Engine by Sergey Brin and Lawrence Page the authors state, “Also, it is interesting to note that metadata efforts have largely failed with web search engines, because any text on the page which is not directly represented to the user is abused to manipulate search engines. There are even numerous companies which specialize in manipulating search engines for profit.”

So Dan ties in the Google PageRank algorithm to the folksonomy argument. Cool! However, at this point you may be thinking that Dan is a proponent of tagging systems. Alas, no, he is not. He goes on to say:

“Metadata is data about data, and tagging a page on the internet is essentially adding metadata. For the same reason that search engines no longer rely on metadata, social bookmarking could be abused and eventually become worthless.”

I think Dan has this second bit all wrong because he fails to distinguish where the metadata comes from and who is using it. If it comes from the expert, it’s expert-supplied metadata. This is exactly the type of metadata that Brin and Page were talking about, and in particular the <meta> tags of HTML. Those are defined by the author of the page (the expert) in the head portion of the HTML document.

As the Brin/Page quote points out, meta tags weren’t shown to the user of the page. This meant that document authors weren’t writing them for their users and thus had little incentive to make them accurate. Instead, their primary use was to tell user agents (search engines) what the page is about.

Because there is no personal use, meta tags get abused. If it doesn’t make a difference to the author what the meta tags say, then they’ll manipulate them away from what best describes their page to what best gets search engines to return them high in the results. This is the inflection point: at this point they become, essentially, SPAM.

However, tags are not defined by authors. They’re supplied by users. They’re user-supplied metadata. As a result, they’re used by the very people who created them. And, it is in that person’s best interest to keep them useful. Even though they can be incorrect like SPAM, they are not like SPAM in that someone actually has incentive to keep them valuable for human use.

BTW: this all seems to follow The Del.icio.us Lesson.

Further, what is the best example of user-supplied metadata on the Web? Links, of course. Links are essentially references to other documents. Links are created by authors but differ from meta tags because people actually use the links, following them and learning from them. Whereas manipulated meta tags didn’t hurt the user experience, manipulated links seriously kills it. If you are putting up bad links on your pages, people respond negatively…and swiftly. They just won’t come back. It’s definitely in the author’s interest to keep links valuable to users.

…and what does Google use to model how we value content? Links!

And we know why we can aggregate links in this way…because we have a large enough set of them to weed out the inconsistencies even as they continue to exist. We’ve got scale, baby!

This isn’t to say that SPAM isn’t a huge problem…it is. I certainly don’t envy the SPAM harvesters at Google. But if we look at all the people making links…the vast majority are creating valuable, non-spammy ones.

So where Dan sees a divergence and a route away from tagging, I see a convergence and a route toward tagging. Not only are tags user-supplied, personal-use metadata (and that will be their primary reason for being), but they also scale really well on a social level because they’re like links…if we have enough of them the incorrect ones (created by spammers and non-spammers alike) actually get lost in the Crowd…

And what does that leave?

Wisdom, I hope.

The most recent controversy happened on September 5th, when someone named jesusphreak posted Digg the Rigged?, an in-depth article exposing some of the curious details of recently-popular stories on digg. Many of the stories, jp pointed out, were dugg by members of the Digg Top 30, or the 30 most popular digg members (popular being measured by number of stories submitted that were promoted to the frontpage). The Top 30 includes Digg founder Kevin Rose.

This was not the first time that someone has pointed out this phenomenon. On April 18 of this year Macgyver at ForeverGeek posted Digg Army, which included screenshots of who dugg two recent articles on the site. Each article had the exact same 16 people digging it in the exact same order. Of the first 19, 18 were the same. Included in that list of people was, again, Kevin Rose. ( for an in-depth history see Tony Hung’s excellent: A Brief History of the Digg Controversy)

These incidents, taken together, are more than coincidence. They strongly suggest that Digg is being gamed by a small number of users, artificially inflating the value of stories that might not deserve such attention. This flies in the face of the democratic ideal of the site. And so far, nobody has claimed that the two articles I mentioned are false: Digg exposes most of the voting activity for all to see. A small group of users is definitely having a large effect on popular stories.

But before we get outraged at the corruption of it all, we should give everyone involved the benefit of the doubt and consider how this might have happened without evil influences.

Don’t blame the users

The users of Digg aren’t to blame. They’re simply playing by the rules as outlined by what they can and cannot do on the site. They’ve figured out how to play in the environment they find themselves in, and that’s OK. Jason Calacanis, creator of digg-clone Netscape.com, said in his post One User, One Vote: “The top users earned their spot and they should be reward for their contributions–not penalized.”. I agree with that…it’s not like there were any rules to follow.

Blame the design

Instead of blaming the users, blame the design of the site. From the ranking system, to the friends feature, to the display of content, to the ease with which users vote, the design of Digg.com conspires to make it haven for gaming. Not only is the pile-on digging activity possible on the site, it is actually enhanced and made easier by the very design of it!

Here are the features in question:

• Rankings list
  If you want people to compete, rank them. This is a big part of the reason why there is gaming on Digg. Getting a higher ranking becomes an incentive to game because if you do then others will notice and you’ll get recognition. (that’s important to people, even in social software) In addition, with the recent offer by afformentioned Calacanis to pay people for this type of work, high rankings may also be a source of income.
• Friends feature
  The Digg friends feature is the means by which the top users promote stories so quickly and with such synchronicity. In particular, the friend’s history page shows the stories that friends have dugg in reverse-chronological order, so that the newest diggs are at the top of the page. By refreshing this page often, top Digg users (who are all friends in the system) can stay up-to-the-moment with each other. During waking hours, a quick 30 diggs will draw further attention to any story, making a pile-on more likely.
• Exposing who diggs what
  At the bottom of each dugg entry is a list of people who have dugg it, and serves as the evidence that the two articles above used to expose the issue going on at Digg. This is a perfect example of what in psychology is called “social proof”. Social proof is something that is “proved by society”. In other words, the mere fact that others are doing makes it seem like it is what should be done. We learn that way, by mimicking the actions of others. So, when we see someone else digging something, we would be much more likely to digg it ourselves. Or, to put it another way, we let others influence our decisions and help make them for us.
• Stories at a distance
  It is very possible to interact on Digg, digging stories and burying others, without actually reading a story. That’s because Digg only shows summaries of posts. If you want to read a post, you actually have to click on them and go to the external site to do so. Many people will make this extra effort. But many people won’t.

  In It’s all a Farce Anyway Tara Hunt recounts an interesting (and scary) conversation with people who game digg. They submit stories and ask their friends to digg them. After a post reaches a certain digg count they say: “people just hit digg if they are remotely interested in the topic.”. This, again, is the effect of social proof, exacerbated because the stories are at a distance and it is extra effort to read them.

• Ease of voting
  While it takes extra effort to read posts, it takes almost no effort to digg them. This might be backward…digg is essentially making it possible to vote without knowing what you’re voting on. Although the digg feature is amazing, an excellent example of technology that makes our lives easier, it is also in danger of trivializing them.

  This reminds me of a story by Derek Powazek in his book Design for Community, where he makes the point that the harder it is for someone to comment on something, the better the comments are. In other words, people who jump through hoops (or pay attention long enough) to comment are the ones who really care about the subject matter, they’re invested in the story and see value in taking the time to respond.

Other factors

The Digg community is protective. Stories that are about digg get a lot of positive attention there. If you want to get noticed, for example, simply write a post entitled “10 Ways to Get Dugg”. That will get them interested. However, it has also been noted that many Digg community members react strongly to anti-Digg content, often burying it when it reaches the front page of the site, effectively censoring it. This has the unintended effect of making it seem like Digg the service is censoring all non-Digg content (which isn’t necessarily false, either).

Also, people use Digg in many different contexts. I’ve dugg stories myself that I just want to read later…stories that I don’t have time to read right now but that seem valuable to me and I want to be able to find them later. It could be that others do this activity as well, causing votes where none should happen. When you give people tools, they don’t always use them as prescribed.

The result: no independence in voting

The result of all these factors is that Digg breaks the cardinal rule of voting: independence. As outlined in James Surowiecki’s book The Wisdom of Crowds, independence arises when a person makes a decision (votes, diggs) without the direct influence of others, on their own, by making up their own mind. Of course, there will always be influences on that decision…what others have said, where their political party is leaning, their current situation, but in the end they need to have the privacy of their vote. On Digg, no votes are private, and when you make them you can’t help but notice the way others are voting.

If we compare this to how people vote in Presidential elections, we see how different it is. In those, we anonymously vote. The anonymity of the vote is key…once we start exposing who voted on what we’re gaming the system because we are inevitably influenced by others votes. And the ranking of voters just solidifies this…imagine if we could see how others voted over time in Presidential elections…

Digg vs. Del.icio.us

The voting on Digg is in contrast to a site like Del.icio.us, where voting (saving a bookmark) is done more independently, often without having any idea whether or not someone else even viewed it, let alone voted on it. Del.icio.us isn’t immune to gaming, however, as there is a popular list, and it’s very easy to simply copy those bookmarks into your own, driving up the numbers just like on Digg.

So far, however, Del.icio.us seems to be more immune to the outcries of gaming. This may result from a smaller user population, as it is nowhere near the size of Digg. But I think it has more to do with the nature of the tool. On Del.icio.us, the main value is personal, as people use it to store bookmarks that are valuable to them. On Digg, the bookmarking utility is secondary to the voting, in both the interface and the wording used on the site.

Digg’s response

Later this past week, Digg responded to the controversy by changing its algorithm:

“This algorithm update will look at the unique digging diversity of the individuals digging the story. Users that follow a gaming pattern will have less promotion weight. This doesn’t mean that the story won’t be promoted, it just means that a more diverse pool of individuals will be need to deem the story homepage-worthy.”

I think this is the wrong approach. By keeping the above features the same…Digg is asking for gaming. As gaming occurs, they’ll have to change their promotion algorithm. Then more gaming will occur to override the new algorithm, which they’ll then have to change. In any social system gaming will occur, but I think Digg’s problems are much more basic: their features simply beg to be gamed. Better to focus on the independence of voting, not on the algorithm. By making much of the ranking and voting hidden, the diversity of the site would increase. It’s exposing information that leads to sameness.

What would change mean?

Even if Digg were to change, however, to alter some of the features above to make voting more independent, we still couldn’t be sure that they would work. People test the boundaries of all social tools, finding every which way to bend them to do something useful. Sometimes it’s fine, sometimes it really does hurt the quality of the site.

Digg couldn’t just say “let’s move the digg voting widget somewhere else” and be done with it. That would introduce a new set of problems, based on the new context. However, they did add a new feature lately whereby the Digg widget shows up right on the posts themselves. That could potentially solve a lot of these problems, getting the voting mechanism much closer to the content people should be reading before voting on. Though it isn’t clear whether or not this is part of the solution, it seems like a step in the right direction. (I’m trying it out below – you may not see it if you’re in an aggregator that strips scripts)

The main conflict of the episode is that George knows what will happen when his two worlds collide: “Relationship George will kill Independent George”. Obviously, however, this difference is only in George’s mind, where there is a clear separation between his love life and his life among friends. It appeals to us because we somehow feel awkward when in the same situation: I certainly remember when I introduced my wife to my family and friends…I was pretty anxious. (thankfully, it turned out great)

The Seinfeld episode is analogous to the current non-struggle we’re having between our “digital life” and our “real life”. Our “digital life” is made up of blogs, email, subscription feeds, and aggregators. Our “real life” is, apparently, everything else.

Nicholas Carr, whose writing I enjoy because he can argue the spots off a leopard, recently described the phenomena of “self-commoditization”, or “producing marketable digital versions of ourselves”. He describes self-commoditization as people creating their own private reality shows, a form of narcissism, and little more than self-consumption. And he’s optimistic about the huge opportunity this brings to those who would make money from it because “there’s little constraint on the supply of digital selves”.

Carr writes in another piece (Selling Ourselves):

“When we communicate to promote ourselves, to gain attention, all we are doing is turning ourselves into goods and our communications into advertising. We become salesmen of ourselves, hucksters of the “I.” In peddling our interests, moreover, we also peddle the commodities that give those interests form: songs, videos, and other saleable products. And in tying our interests to our identities, we give marketers the information they need to control those interests and, in the end, those identities.”

Carr’s argument, in addition to its searing doomsdayishness, comes (as most of them do) with the notion that somebody ought to make money from this. That’s how he makes the jump from our interactions with others to some sort of selling/buying relationship and, ultimately, to the intimation that our “digital life” will kill our “real life”, much like George was afraid of.

If you do happen to be in the media (whose goal is to make money) the question is an interesting one. But for the rest of us who see the difference between our “digital life” and our “real life” shrinking every second (or even non-existent), the question of making money is of little importance, and comparing social interactions to money changing hands completely obscures the situation. Much more intricate than money exchange are social standing, peer opinion, having friends, and being liked. You know, the human things. I continually bristle at the notion that our interactions online can be explained in economic terms.

I think the dichotomy of a “digital life” being somehow different from our “real life” is becoming more false every day. Not only do people understand how web technologies work, but they’re leveraging them to improve all parts of their lives. And the evils that Carr is so quick to point out (gang mentality, self-commoditization, and my personal favorite: blogospheric lynch mob!) are simply online representations of people’s behavior…behavior that hasn’t changed for millenia. When Carr gets excited about the latest emotional upswell online and compares it to selling our souls to the Devil, it is more interesting to watch him construct an argument for discussion from it than to actually go watch the event. Sure, there are issues with identity, but for the most part people are honest and are who they say they are.

Am I being too optimistic? Maybe. We could spend all of our time focusing on the tiny fraction of evil folks, hoping that by calling them trolls they would simply go away. But we’ve always had those and probably always will. Instead, I would like to see more positive stories about how people are improving their lives with technology from writers like Carr. Unfortunately, I don’t think that will happen anytime soon.

The main difference in the last 15 years of human living isn’t that somehow being online has created an alternate universe for us. It’s not that the Internet has made us into lynch mobs. The main difference is that instead of our hazy memory of what happened we have a digital record.

Does being online change our behavior? Yes, certainly, but most of it is in terms of how we do something, not why we do it. We don’t suddenly become narcissists, any more than we used to be. If we can quell the notion that social software is ruining society then we can recognize and repair those tears in the social fabric that do exist.

My wife, just yesterday, was in an email discussion made up of a group of mothers across town. They planned to get together at one of their houses, and when they met in the afternoon not a single one of them remarked how their two worlds had collided, or how bad they felt about their communications becoming advertising. They even referred to their email conversation as “talking”.

Update: I recommend reading two thoughtful follow-ups to this post that went into more depth and nuance than I did:

• Our Fluid Selves, Online and Off by Anne Zelenka
• The web is the real world by Jen Spadafora

I’m not one to believe that we’re in a Bubble 2.0 or anything like that (aren’t we always bubbular?), but here are a few ideas about why some of the web apps out there fail.

1. Focus on social instead of personal.
   Following up on my Del.icio.us Lesson post, this is a critical reason why web apps fail. Many apps focus on being the new social killer-app when, in general, people don’t have time to worry about what other people are doing, and will only use software that benefits them personally at every step. You could call this selfishness or laziness, but I would call it optimization. For example, we simply don’t have time to tag things for tagging sake. Instead, we might tag things if we think that it will help us in the future, but adding tags to an app does not a solution make.
2. They solve too many problems, or try to.
   This is when the buzzwords rear their ugly head. If you’ve got a list of problems you’re solving with an application, it stands to reason that you can’t solve any one of them fully. Instead of trying to solve more than one, focus like gangbusters on one problem and really nail it. If you think about the successful web apps out there right now that have garnered impressive mindshare, it should be easy to line up the one problem (or activity) they really get right. Flickr: photos. Del.icio.us: bookmarks. Facebook: college. Myspace: identity. Gmail: email. Plaxo: contacts. Tailrank: news. Etc…
3. They’re about making someone other than the user happy.
   So much focus is on aggregation right now that it is easy to overlook the happiness of users. Many services, such as Technorati Tags or Google Sitemaps, exist solely to make the aggregators happy, and not the user themselves. They sell themselves on incentives that sound like what a movie agent might say to an aspiring actor: “We’ll make you famous, kid. You’ll get found!”. First of all, this is all talk directed at the developer, who is not the user. That’s a huge tip-off right there. Second of all, if the aggregators had their way everyone would be using these formats, which simply dilutes the value for everyone else and only serves to lock the site into some weird relationship with the aggregator. This is not how it should be. That’s why I stopped using those two services ages ago. Instead, focus on adding features that make the user happy, and when that happens everyone else can be happy, too.
4. They sell it the wrong way.
   Web apps are not about Ajax, tags, Web 2.0, SOA, REST, or any other technology. Why do so many startups and web pundits focus on these terms when talking about a product? To get a better frame of reference, talk about how your app empowers users to improve their life. Think about how the long-term successful companies sell their stuff. They relate it to some bigger idea. So, for example, Nike has always embraced the hero archetype. They might talk about how great their foam arch is, but that’s always secondary to how buying their shoes makes you a hero. Their commercials are often amateur runners out running in the rain. How cool is that? Way cooler than double-density shock foam. A good example of this in web apps is the messaging from 37signals. They’re not selling software, they’re selling rebellion.
5. Not in it for the long haul.
   If you build it, they will not come. There is too much competition right now, so another wiki-type application isn’t going to set the world on fire. I can’t tell you how many stories I’ve heard about web apps that became successful only after they adapted to their user base over time (short periods of time, but over time nonetheless). Their initial effort didn’t work, or was too similar to another one, but they were in it for the long haul and they adapted to what their users wanted. Flickr is a great example of this. Flickr started out as a game called Game Neverending. That didn’t work, but their second attempt did. Many web app makers would never make it to the point of seeing the light (or admitting the failure).
6. They show too much of what’s going on, and get gamed.
   One of the big promises of aggregating the wisdom of crowds is building systems that use the input from huge user populations to come up with value. However, as people get used to how the wisdom is aggregated, they figure out how it all works, and the more public the mechanism for aggregation, the easier it is to figure out. That’s why gaming is such an issue with Digg. The voting on Digg is public, so you can see which items have the most votes before you submit your vote yourself. This goes against one of the principles of the Wisdom of Crowds, which states that in order to successfully harness it, each member of the crowd needs to be making an independent vote.
7. They don’t have an underlying business strategy of improving people’s lives.
   Most business strategy is about making money. However, this is a short term goal. If you focus only on ways to make money, then you’ll make decisions that in the short term seem good for the balance sheet but in the long term actually work against it. Take the case of LLBean. Where everyone else is trying to get away from call centers and move all of their customer interaction to a web site, LLBean actually allows you to talk to a human being almost instantaneously. Their phone number is easily found on their web site/app. This probably does cost them a lot more than if they had some contact forms or an instant chat room, but it sure does make it quick and easy to give them money. My sister worked at LLBean for a time, and I was always impressed by the way that they empowered her to handle customers. It probably cost them money in the short term, but people remember when you make their lives easier, not harder. Many companies, unfortunately, see the Web as a way to reduce direct communication with customers, when in reality it should cause an increase in communication if you’re successful.

Update: 7 More Reasons Why Web Apps Fail

The answer to this question is why the Wisdom of Crowds is a counter-intuitive, dangerous and powerful idea. In reality our democracy does not harness the Wisdom of Crowds effectively, because it does not recreate the three conditions that are needed in order to do so.

What are these three conditions? The requirements are diversity, independence, and decentralization.

• Diversity
  Diversity, or large variation in ideas, is crucial to being able to extract wisdom from the crowd. We need liberal, conservative, crazy, hair-brained, logical, rational, irrational, and all ideas in between. The reason why we need such diversity is probability. The more diversity in the ideas that we include, the higher the probability that one of those ideas is the *right* one. Because sometimes it just happens to be that one hair-brained idea that leads us down the right path.

  In the case of U.S. politics, we rarely have more than two competing ideas, one each from conservatives and liberals. All conservatives marshall themselves around the same ideas, and all liberals do the same. This is exactly the opposite of diversity.

  Interestingly, the best ideas often come from individuals who have a unique view of the world. Think Darwin, Da Vinci, Einstein, Joan of Arc. These individuals changed history by having ideas that nobody else had (or could articulate). By including as many diverse individuals as possible, there is a higher chance that wisdom will prevail. Including another set of the same old ideas won’t lead to anything new, just more of the same.

  In addition, it doesn’t take a genius to have a unique idea. Only to articulate it.

• Independence
  In addition to a diversity of ideas from which to choose, we need the quality of not imitating the first idea that comes along (even if it is diverse). Instead, we need to hold to our own ideas, if possible, to come up with our own conclusions. That’s not to say that we won’t agree with other people, but each of us has our own set of knowledge, and holding to that set of knowledge will keep our thought independent. Independence is about how we react to ideas as they are presented to us.

  Again, the two-party system kills most independent thought. Each side has their “candidate”, and party members tend to follow that candidate. It’s not that voters can’t have their own, nuanced view, but holding to that view and voting independently will rarely get your obscure candidate elected. The game we’re playing is that we would rather vote for the candidate whose views are closer to our own than to vote for the candidate whose views are farther away. That’s the only way to optimize our vote in a two-party system. Unfortunately, it doesn’t represent our independence of mind very well.

• Decentralization
  Most of us tend to generalize ideas so as to make them applicable to other situations. This, however, might not be the best way to tackle problems in order to extract the wisdom of crowds. According to Surowieki, a better way might be to attack problems individually, in a decentralized manner, so that those people closest to the problem are the ones who solve it, not somebody from on high. Decentralization is crucial to tacit knowledge, that sort of knowledge that is hard to communicate to others, and is often vital to solving the problem at hand.

  A lack of decentralization hurts big time when someone in Washington is making decisions that affect people everywhere else. For example, the Bush White House has consistently made decisions that favor big business over the environment. This is probably because all of the problems that the White House sees are the ones in Washington, dutifully lobbied by oil companies and the like. Meanwhile, those people closest to the problems of the environment, the ecologists, biologists, hikers, and outdoor enthusiasts are out where their problems lie…not in Washington. And the same goes for all localized problems…if people in Washington can’t see (or refuse to acknowledge) what’s going on elsewhere, how can they make any reasonable decisions about it? They can’t.

In general, I think we sense that these conditions are correct when we say things like “Don’t be a sheep”, or “Keeping up with the Joneses”. We don’t like follow the leader, and we have more respect for those people who go out on their own and live by their own standard, even if we don’t agree with it. That’s what makes Thoreau so inviting, that he’s saying “to hell with all ideas except those in my head”.

But when we talk politics (at least in the U.S.), it’s always follow the leader, or follow the party. We’ve broken the game, and now the only way to win is to choose one of two sides. And it’s one step away from the opposite of diversity, which would be totalitarian rule.

If there is one idea that encapsulates what Web 2.0 is about, one idea that wasn’t a factor before but is a factor now, it’s the idea of leveraging the network to uncover the Wisdom of Crowds. Forget Ajax, APIs, and other technologies for a second. The big challenge is aggregating whatever tidbits of digitally-recorded behavior we can find, making some sense of it algorithmically, and then uncovering the wisdom of crowds through a clear and easy interface to it.

Some folks like to point to technology as the heart and soul of Web 2.0. I don’t think so. The heart and soul of Web 2.0 is the new ideas that drive technological and social innovation, and the one crucial idea is the one found in Surowiecki’s seminal book. It has forever altered the way that software is written.

And the evidence is mounting. Today, Richard MacManus writes of the new features on Rojo, and in explaining what they are Chris Alden tells Richard that they’re emulating Pagerank:

“How do we do it? (determine relevance) Generally, just like Google used link metadata to determine relevance of search results, there is a fair amount of metadata we can use to infer relevance, including how many people are reading, tagging, and voting for a story, how popular the feed is — both to you personally, to your contacts, and to all readers, as well as things like link data and content analysis. ” (emphasis mine)

The end result is relevance engines, filters, recommendation systems, Web 2.0 software, or whatever you want to call it. And Surowiecki brilliantly sums it all up.