The Long Tail, or the death of the product shelf (where shelf space becomes irrelevant when content is digital) brought on tremendous change in the economics of distribution. Netflix rents most of its movies from the catalog of past movies, not from the current list of blockbusters. Same with Amazon and books, iTunes and music. Christopher Anderson goes into a lot more details in the book he wrote on the subject: The Long Tail.

When content is digital, a public good, it is freely distributable by electronic means. It is infinitely copyable at 100% fidelity. Moreover, as the Long Tail shows, libraries of content can be built cheaply which provide value for the long term. Once Google digitizes all the books in the world they won’t ever have to again.

In other words, all content is available at all times.

What does this lead to? The Paradox of Choice! There are simply too many things to choose from. Which of the thousands of movies on Netflix do I rent? Which of the books on Amazon do I read? Which of the songs on iTunes do I listen to?

In the past, we listened to either the creator or the distributor for help. Since choice was limited, they would steer us to something in their limited selection. You either went to one of the movies at the local theater, or you didn’t watch a movie. You either bought a book from the book store or checked one out of the library, or you didn’t read. If you were lucky enough to be near a creator (like a rock band) you either went to the pub to listen to them or you went without live music.

The creator and the distributor, however, had a problem. They were always and forever biased. You couldn’t ask either the band or the book store for a recommendation because they would only recommend something in their repertoire.

With digital content, the repertoire contains all possible choices. The Long Tail has given us more choice than we could have hoped for! Now the distributor can simply tell us which is the best book, the best movie, and the best music. Right?

Contrary to what we would think, however, most distributors are still biased. They still try to pick products for you, rather than helping you find the best fit for your needs. They know you’re going to buy something because they have everything. So many distributors make deals with manufacturers to see who will pony up the big advertising dollars. Who will buy the most end caps. Who will buy the preferred ads on the web site. Who will pay money for the ability to get separated from the pack.

Instead of the Long Tail solving the choice problem, the customers are still often left with the question: how do I choose the right item for me?

Social design is thus forced upon the marketplace. The Long Tail begets social features that let users help each other (either implicitly or explicitly). The only way for people to find out what’s best for them is to route around the system in the way they’ve always done.

Ask other people. Have conversations. Give and get recommendations. Tell someone what your preferences are, and they’ll give you their best guess.

And that’s what Netflix and Amazon and iTunes have done. They’ve accepted that customers do in fact know a wealth of information about their wares compared to any one source (even themselves). In a sense they were forced to recognize this, for they had no other way to give recommendations to their customers. (any amount of research would show that people still struggle mightily with choosing items online) The old constraint of shelf space, and thus a less-than-everything inventory, is gone.

Which leaves companies who have not done so (competitors to Netflix, Amazon, and iTunes) with a choice: do they help this process and build social tools for their customers, or do they continue to support business as usual?

In his 2005 Les Blogs presentation Doc Searls, in his explanation of what blogs are and what they are not, suggested that:

“We are all authors of each other.”

What exactly does Doc mean by this? Does he mean that we author other people’s lives, and they ours, whether or not we want them to? Or could it mean something more optimistic, that we author each other gladly?

Then there’s the problem of popularity. How does popularity fit into the idea that we all author each other? Don’t popular things help shape us, too? Do the voices that add up to popularity author us in the aggregate?

Popularity is maligned as much as any attribute known to man. If you are popular, you are probably not worth paying attention to. It’s as if we are saying: “You already have too much attention, and I’m not going to give you more.”

But I think there is much more to popularity than unwarranted attention.

Memetrackers!

The current explosion of memetrackers (Memeorandum, Tailrank, Megite, Findory) is a great place to observe these questions play out. Gabe Riviera’s Memeorandum, one of the first of the bunch, started with a list of RSS feeds and grew out organically from there (at least that’s the legend I’ve heard several times). On tech.memeorandum, you can get the latest buzz in the tech blogosphere, and you can track a meme for the 10 or 12 hours that it’s hot. Unfortunately, as many have pointed out, it sometimes feels like an echo-chamber with the same blogs bubbling to the top consistently.

I once heard that the most popular bloggers weren’t necessarily the ones with the best ideas, but they were the ones who could amplify those ideas most effectively. Just like any other writing, I guess.

Today’s hottest blog, of course, is Mike Arrington’s Techcrunch, a clearinghouse for the latest Web 2.0 news and product releases. Mike’s success seems to rely on his ability to quickly get the story and tell the most relevant details before others do. His tireless excellence is really something to watch, making bloggers like me wonder how many hours there are in his days.

Technorati, who likes to throw around big numbers, has the Technorati 100, a list of popular blogs which, according to this study by Tristan Louis, changes rather frequently. Today’s Doc Searls is tomorrow’s … what was that guy’s name again?

Popularity

The energy put forth in figuring out how to get onto Memeorandum, Techcrunch, and Technorati is proof that people look to those services as a light in the fog. People want to be and want to know what’s popular, and the primary utility of these services is to help them do that.

Popularity, it seems, is the goal of many a blogger. We get a kick out of seeing our name on the lists. If I was Boing Boing sitting in the #1 spot in the 100, I would be pretty damn pleased with myself. I might even take a vacation.

This makes sense because outside of acclaim from our readers we really have no idea if what we’re writing is jiving with people. For a quick calculation we instead look to numbers and rankings to suggest how well we’re doing.

So if getting popular is at least part of our goal, why is there always such a negative connotation about it? Seth Finkelstein, in response to a Scott Karp post in which he laments the rise of popularity sites like Digg and Reddit, reiterated a common distinction between popularity and importance:

“Finding what’s *popular* is easier and more profitable than what’s “important”. In order to find the popular, you just poll either the crowd, or the demagogues (people who are experts – at what’s popular). That’s very simple (relatively speaking).

But how do you find what’s important, what you *need*? What do you code for? The first cut is to poll a niche rather than a general audience. But problems there are that there might not be enough of a sample, and the economics are even less supportable.

These questions don’t often get discussed extensively because the hype-machine runs on populism and demogoguery, so that’s what gets amplified and echoed. But also, there’s more to discuss, and moreover, a service which acts to find the popular is, recursively, a popular topic for coverage.

There is that word again: hype. Whenever this word is mentioned, it is in relationship to something that the author wishes would go away. Same with populism. Never mind that millions of people are talking about it. It’s hype. Forget it.

The endless struggle to get at personally-relevant information, to weed out the hype and focus on what is important to us, is the defining problem on the Web at this time.

Importance is relative

The problem, as Seth points out, is that each of us has a different definition of importance. But that’s only part of the problem. The other part is that most of us secretly yearn for popularity. So even as we argue against popularity as an idea, we really don’t mean that popularity isn’t important to us. Things that are popular that we don’t find important don’t make sense, but if we were popular, if our blog was in the top 100…well then that’s OK.

But I don’t think that should stop us from trying to alleviate the problem. I think we’re learning more about this as we go. For example, I think we can distinguish between two types of popularity:

Echo-chamber popularity

This is the popularity that Seth was talking about and that everybody hates. It’s the one that people complain about when they complain about Memeorandum, about Britney Spears, about the wicked pretty cheerleader and her dumbass jock boyfriend.

Interestingly, however, our anger at this type of popularity can only occur after we’ve made the judgement that it isn’t important to us. Most people don’t complain that the Beatles were popular because they’re a pretty good band. Their popularity was justified. And very few people complain about memeorandum if they’re the ones getting consistent traffic from it.

Only after we’ve read the stories on memeorandum can they be part of the echo-chamber. The first reading, the reading at which the information was new to us, was actually valuable in some way, if only to let us know that it wasn’t important to us.

Word-of-mouth popularity

This is the popularity that we all value. When a friend recommends a movie to us, we listen, even if they’re talking about the same damn movie everyone else is. Because it is coming from a personal authority, we don’t consider it a part of the echo-chamber, we consider it valuable information. People have a great way of knowing what will be important to their friends, and so probably won’t give them the same echoes that they hear elsewhere.

We all want people to say nice things about us. If we become popular that way, then all is well.

The problem is that it’s really hard to have one type of popularity without the other. In fact, they’re basically the same thing, except one is popularity that we value and the other isn’t. It all depends on whether or not we find the popular thing valuable. If we don’t, we start saying things like “hype” and “echo-chamber”. These words actually say more about the relationship between the person saying them and the topic than they do about the topic itself.

Most things aren’t valuable to us

Just as the Power Law dictates that in a network some things will be popular and most things won’t, some popular things we’ll find important but most we won’t.

The technologies that we’re using to leverage the network now, most particularly RSS, are giving us more access to more things. With that comes more things that we don’t value. Is this a failing of the network? Or is it a failure of the people on the network? Or is it a failure of the technology? Is it a failure at all?

Probably not. It’s simply that we have more of everything, and are already taking for granted the increase in valuable information because there’s so much useless information as well. Of course, there was a time before cell phones and computers, but who the hell remembers that?

What we need is to model that in our software in a way that allows us to make judgments about which information is valuable to us, and which is merely hype. In other words, we need to be able to pick out those popular things that we still value, despite our fears of popular things. I think this is what Dan Bricklin is getting at in his insightful article: When The Long Tail Wags The Dog.

So I think Doc is right, we are all authors of each other. We value the same things we’ve always valued: the information we receive from family, friends, writers we admire, and other personal authorities. And oh yes, we’ll value a small portion of the popular things too, because we don’t discover everything through the people we know. We’ll probably also continue to deride popularity as a bad idea, because we always have, ever since the cool 6th graders were listening to Guns and Roses and using the F-word.

And all the while we’ll secretly ego-surf to see how popular we would be if we believed in that sort of thing.

Web 2.0 – Leveraging the Network (2.74 MB pdf)

In the talk I spoke about how Web 2.0 companies distinguish themselves by leveraging the network of which they are a part. Brittanica, for example, has had a web site for quite some time and were slow to leverage the network in any particular way. Wikipedia, on the other hand, exists only because they used the available network to improve their contents communally. And Wikipedia, of course, is a much, much more popular site.

As in my last talk: Web 2.0 for the Rest of Us (which includes a podcast), I started down the road toward Web 2.0 from the standpoint of those Web companies who have excelled: Google, Yahoo, Amazon, and eBay. They obviously know more about succeeding online than anybody else, and have become so successful so fast that we often take them for granted, even though they are barely a decade old. So, I find it particularly useful to ask: What makes them so special? What have they done that others haven’t? And I find myself coming back to the same answer over and over: they know how to leverage the network. From Google’s pagerank algorithm to the APIs of eBay and Amazon to the movie ratings on Yahoo, these companies know how to harness the collective activity and intelligence of people to make their services better.

For those who want only the quick and dirty (without the pretty pictures), here are the talking points:

1. The home page is no longer the most important page on your site.
2. The information architecture that people use to find your content is, increasingly, not yours.
3. Each feature added to an application is more to think about – for everyone.
4. Folksonomies are a way for users to map their own, familiar vocabulary to your alien one.
5. Words are the currency of the Web. Spend the most time on your words.
6. Seducible moments are those increasingly rare moments when you can talk to your users in an appropriate context.
7. Recommendation systems are a forced move.
8. Users want control.
9. Users appreciate tools that help them make their own well-informed decisions.
10. The best software models human behavior.
11. Links model how users value content, and are only the start…
12. Sometimes it is easier to design for yourself than others.
13. There is always an opportunity for a better interface to data.
14. All things being equal, faster interfaces allow for more innovation.
15. Most people are willing to trade their personal information for good service.
16. As choices grow, so does the importance of learnability.
17. Redesigns are dead.
18. Network effects are rare, and killer.
19. Network effects work in the opposite way for teams building software.
20. Personal value precedes network value
21. People rarely do things for the “good of the network”
22. Del.icio.us, though providing very cool tagging features, is mostly about a single person remembering items for later.
23. “The accretion of tiny marvels can numb us to the arrival of the stupendous”

I would appreciate any and all feedback, as I’ll be giving this talk in the future and would like to improve upon it in any way that I can.

The reason why he’s had difficulty is because asking for attention is the ass-backwards way to get it. Putting the cart before the horse. Asking for sex on the first date. You might succeed once in a while, but it will probably do more harm than good if you want to be really successful. If that’s his idea of what blogging is, then he’s probably going to be sorely disappointed.

Blogging is like Dating

Blogging is a lot like dating. Like a date, a blog post can be the first step in a meaningful relationship. You’re just getting to know each other, finding out your interests, seeing how the conversation goes. Being cautious at this stage is prudent: it’s hard to know if the person who seems normal at first glance is actually a quack underneath it all.

Similarly, asking for others to link to you can come across like being asked for cash by a beggar. Solicitation is never pretty, as both the solicitor and solicitee usually feel uncomfortable. Here you are trying to get them to do something without any incentive, not even sex! In fact, you’re asking them to give up their most precious resource…a blogger giving a link is one of only a very few ways to give a gesture of attention. Therefore, it is a big deal.

Old Media and Gatekeepers

The lack of attention Karp received from the bloggers causes him concern. He muses on what he calls “gatekeepers” in the blogosphere, a few media moguls who direct the attention of the many. He likens it to the Old Media of ten years ago when there were only a handful of places to get media: a newspaper or magazine, one of the big 3 TV networks, radio, or movies. Karp worries that the most empowered amateurs (influential bloggers) are trending toward this model, they have too much influence, and as a result those less prominent folks with good ideas can’t or won’t be seen. He says:

‘the “system” is starting to feel a lot like Old Media, with the high-traffic blogs acting as gatekeepers for the blogosphere’s attention.’

In addition, Karp sees the gatekeepers as insufficient. He uses tech.memeorandum an example of a gatekeeper who can’t get the word out to enough people:

“If I want to reach an audience of Old Media executives who are wrestling with the painful transition to New Media, I don’t think tech.memeorandum is going to cut it. It’s not that none of them read it – it’s a matter of media fundamentals. Tech.memeorandum is highly efficient for reaching fellow geeks in the blogosphere, but much less efficient for reaching outside of it.”

Why the Chess Team Doesn’t Get Laid

This, of course, is the Long Tail in action. There are only so many BoingBoings, Slashdots, and Diggs. Only so many Winers, Rubels, and Jarvises. Even if you have something interesting to say, if you don’t have a megaphone or don’t get picked up by someone with one you won’t have a very big audience. It’s the lament of bloggers everywhere. It’s the lament of great artists who can’t get critics to see them for the genius they are. It’s also why the chess team doesn’t get laid.

To fight this model is to fight the tide. And more perhaps more interestingly, it’s fighting probability. The more you post, the more you say, the higher the probability that you’ll say something interesting, that others will hear you, and that you’ll create real relationships with fellow bloggers. That’s why Rubel, Jarvis, and Winer are where they are. They’ve written more than the vast majority of bloggers will ever write. They’ve observed, critiqued, and linked for years. They’ve taught us the fundamental rule of blogging…Don’t swim against the tide.

It’s about Relationships

The real problem for Karp isn’t that the gatekeepers are shutting him out or even that there are gatekeepers who can do the shutting out: it’s that Karp doesn’t have a relationship with anybody yet. Nobody knows what he’s talking about, what his schtick is, which way he leans. That takes time. (Even so, he’s well on his way, in two short months of blogging he’s already got more attention than most bloggers ever do.)

Most of the early attention he’s received seems to be the result of Karp asking provocative questions about the relationship between media and authority. He asks: “as Old Media gatekeepers fade, who will ultimately take their place?”. And he quotes Justin Fox: “Are there great new fortunes to be made in telling us what to pay attention to?”. These questions are fundamental to media and the Web.

The answer to these questions will come from the network, I think. Look to the new companies that are thriving: Google, Amazon, Yahoo, Netflix. These companies are harnessing their network of users to provide valuable, personalized recommendation systems that exist outside of any of the Old Media. They’re replicating our individual authority models to the point where content becomes more important than the media outlet from which it came. The amazing potential of Web 2.0 is that it distributes authority at the personal level. The next time you get a movie recommended to you from one of your friends on Netflix, think about how much more valuable that is than some review pumped through the Old Media. Did you know that roughly 2/3 of movies rented on Netflix come from recommendations?

Yes, amateurs are affecting the stock market. And publishing. For every loss in Old Media’s attention machine, there is a gain in personalized recommendation systems like Netflix that won’t ever return to either Old or even the “New” Media. It’s in the people’s hands now.

However, even though Netflix creates a wonderful tool for modeling authority, it won’t be the authority itself. No, authority will lie in individual people whom we trust, who happen to use the same systems that we do.

Gatekeepers No More

On this note, Karp rightly assumes that many bloggers won’t even consider the notion of gatekeepers in the blogosphere. I fit into this camp, as does Matt McAlister, Senior Product Manager at Yahoo:

“Insistence that there’s an editorial gatekeeper required in the media model is going to hold Old Media back from embracing New Media at any truly valuable level.”

To which Karp replies:

“My point is that media can’t function without some type of gatekeepers — otherwise you have complete entropy, with people awash in random information.”

Add this to a comment on a previous post, and you can see where Karp is headed:

“There is so much wrong with the blogger view that the monoliths of old media will be brought down and consumers will bask in the glory of infinite media choice – discussing, creating, tagging, rating (meta-ing) each other’s content in one big solipsistic frenzy.”

Just Regular People

Here’s another view. Instead of gatekeepers, I see people. Where gatekeepers simply keep the gate from all intruders, people are open to having relationships with others. And these relationships are anything but random. They’re authoritative, valuable, ever-changing, and rely on trust built up over time. You know, the whole Golden Rule thing. So instead of looking for the gatekeepers, perhaps Karp should ask himself: if hundreds of people started asking me for links, how long would it take before I started saying no?

Similarly, how long would it take a smart woman to realize that a guy who asks for sex on the first date probably isn’t worth the second?

But I got to thinking. Why is their recommendation powerful? Is it because they are two people whose opinion I trust, or because nothing else was recommended to me? What if, for example, someone else whose opinion I value recommends a different movie as highly as Kiss Kiss, Bang Bang? What should I do then?

MovieLens, a Movie Recommendation Tool

So I turned to my new movie recommendation engine called MovieLens, created by some fine folks at University of Minnesota. They are members of GroupLens Research, and have been working on recommendation systems for over a decade. To make its predictions, MovieLens uses a technique called “collaborative filtering”, which takes actions by both me and others to produce a set of recommendations for movies I should see. They explain more about the service on their about MovieLens page.

Unfortunately, Kiss Kiss, Bang Bang was not in their database yet, as it hasn’t even been released. (Jared and Christine belong to some sort of movie club that often gets to view movies ahead of release).

But it did have recommendations for me. Since I started using it, I’ve entered ratings for over 100 movies in MovieLens, and it has enough information to recommend hundreds of other titles. By combining my ratings of movies with ratings from thousands of other people who have seen the movies I haven’t, the number one recommended movie for me at the present moment is: Dear Frankie, a drama.

Now I have a choice: Kiss Kiss, Bang Bang or Dear Frankie?

There are many movie recommendation systems like MovieLens. People adore the recommendations of the Netflix web site. Blockbuster has one. Walmart suggests movies to watch. Who needs Ebert when we’ve got systems like these?

Recommendation Systems Everywhere

Recommendation systems are a growing trend, perhaps you’ve seen some of these:

• iTunes “Top Songs”
• Amazon “people who bought this also bought…”
• Bloglines “similar blogs”
• Del.icio.us “most popular” bookmarks
• NYTimes “most emailed articles”

Just to name a few. As Movielens demonstrates, recommendation systems are systems that recommend content for us by looking at certain factors including what other people are doing as well as what we are doing. The basic algorithm of collaborative filtering is to compare the trends in how I rate movies to those of other people. If other people rate movies similar to me, then they become part of my “neighborhood” of like-minded users. The movies that get recommended to me will be the ones that my neighborhood rates highly that I haven’t seen yet.

Recommendation Systems are User-Focused

Up until recently, recommendations from friends have been the most helpful way for us to find out about new things. In the future, recommendations will come from computer systems like Movielens in addition to friends. They will further infiltrate our daily lives, and I think we’ll be happy about it.

There is a lot of room for improvement in recommendation systems. Right now most of our systems prioritize content based solely on time: newest at the top, oldest at bottom. We see ads for new movies in a much higher proportion than we do old movies, even though old movies might be just as good or as important to us as the new ones.

The prioritization, of course, isn’t ours. It’s the prioritization of movie studios or distributors or whomever is trying to make money on the release of the movie. Since the prioritization isn’t ours, the relevance is not, either. This is why Netflix has grown so quickly, because it is using collaborative filtering to prioritize users over the movies.

The following list is a few ways how recommendation systems could prioritize content more helpfully. Some of these are similar to Movielens, while others are based more on other ways in which we prioritize information in our daily lives.

Ways to Prioritize

• Newness: This is how most systems prioritize things, by how new they are. New information has a higher potential of being great than old information, and therefore it is highly interesting.
• Time-sensitivity: Prioritized by whether something needs attention now vs. later. The flight status of your plane trip tomorrow would be very important. A month away…probably not.
• Popularity: Prioritized by how many other people are paying attention to it. So, anything that is getting attention now will be quickly routed your way. Anything that doesn’t won’t be seen until you get to it.
• Personal Relevance: Prioritized based on some criteria that you deem relevant. This could be your line of work, your interests, your hobbies, your family members.
• Social Network Relevance: Prioritized based on whether or not it is recommended by someone in your personal network. If someone you know (or someone close to someone you know) recommends something, that gets priority over anybody who you don’t know.
• Authority-based: Prioritized based on some metric of authority, which would take into consideration many other user’s actions. This would have to be based on some metric over time: those outlets that are most visited or referred to are given the most authority. Roger Ebert, for example, is not just the most popular movie reviewer. He’s the most authoritative movie recommender.
• Collaborative: Prioritized based on your own actions compared with others. Whereas prioritization by personal relevance would be based on some criteria you choose, collaborative-based prioritization would be based on some algorithm outside of your direct manipulation. This is what Movielens does.

By combining these various prioritization schemes, recommendation systems could make intelligent decisions about which bits of content to show us now and which we should pay attention to that we already aren’t.

We will soon see recommendation systems everywhere, and start desiring them in parts of our lives that don’t have them yet. For example, I would love to have a great recommendation system for blogs. I’ve used Bloglines, which is decent, but its not my primary blog reader and I find that most of the recommendations are for similar blogs, not just blogs of similar quality. I want to read blogs about all sorts of topics, not just about web design. Way back in November I emailed the folks building Rojo and suggested that their recommendations feature has the potential to be the #1 feature of their software. I think this is still true.

Always showing the latest things can be valuable for people. Avid moviegoers want to see what’s latest, because they’ve seen most of the old movies. On the whole, however, we don’t judge quality by equating it to newness. We judge it on other criteria, be it authority, our social network, or personal interest. Old things are sometimes even better. In fact, many people sadly say that the Golden Age of film has already passed.

The Golden Age of Recommendation Systems, however, has just begun.

The Long Tail is about focusing on the less popular content that previously couldn’t be accessed because of some physical limitation: most often shelf space. The classic examples that Anderson uses are music and books. Book and CD stores can only hold so many albums and books, so the constraint of shelf space hinders their ability to provide an exhaustive selection.

Online, there is no physical constraint like shelf space, so amazon.com can offer a much wider selection than can a physical Barnes & Noble store. Anderson points out: “The average Barnes & Noble carries 130,000 titles. Yet more than half of Amazon’s book sales come from outside its top 130,000 titles.”

Similarly, Web 2.0 is about enabling access to previously unavailable digital content. The constraint involved is political or psychological, or something akin to that.

The way that Web 2.0 enables access is by the creation of APIs on top of databases that were previously siloed for private use only. In other words, companies and organizations have databases in which they keep information. By default, that information is for their use only, as it resides on their network and their computers and they have firewalls protecting it. Some however, put valuable information on a web server and offer it up via a web site, but they only offer their interface for accessing it. However, by putting it on a web server and creating an API so that others can access it, the potential uses of the data, and its value, increase tremendously.

In most cases, Web 2.0 APIs will provide access to all content on one type. So, if a company provides access to mp3 music files, you’ll probably be able to access all of the music (even the disco genre). This is because physical storage is cheap as dirt: hard drives cost nothing nowadays.

I see lots of similarities between the Long Tail and Web 2.0. Both ideas are about improved access to previously unavailable content, both are about showing the whole catalog, and both are ultimately great at enabling user choice. They seem to overlap a lot. If I had to make a marked distinction between them I would say that Web 2.0 is about the access to information while the Long Tail is about the economics of it all.

The end result is that we now have unprecendented access to tons of great content. Drink deep.