Dear Josh,

http://movies.yahoo.com/

If you go to Yahoo.com and click on movies (left side of screen) you come to the above web-page.

The annoying thing is it took 19 seconds for this page to completely appear. Nothing wrong with my computer. Just with the page – there is tooooooo much there.

Anyway, I just thought I would send this to you as you mentioned how a cluttered web-page seems to grow even more cluttered by virtue of the ‘LAW OF WEB-PAGE SPRAWL’. And Yahoo is a perfect example of this ‘effect’. For instance, they totally screwed up their TV-listings page also.

The YAHOO! TV page used to load quickly and you could find out program information quickly and easily. So they changed it. Now it is so ungainly and slow, I have stopped using it.

‘Media-brainiacs’ at the biggest companies dictate content based on advertising over convenience….in other words….the heck with the user.

After all, he’ll say, “LIVE WITH IT”. I shouldn’t be too surprised. Media, radio and TV for example, is always trying ways to squeeze in ever more advertising.

How much do they think people will put up with???? I think that this is the question I have begun to ask myself. The media keeps pushing the limits of advertising and programming (expanding the former and squeezing the latter) …. is there any limit?

The web is now ‘THE MEDIA’ and as such (at Yahoo at least) they’ve been exploring this question – i.e. “WHAT IS THE LIMIT TO THE USERS AD TOLERANCE AND MAXIMIZING ADVERTISING REVENUES?”

I know that they are doing this. So sometimes just being aware of it is annoying in itself. At other times I just sigh and say, “LIVE WITH IT”.

I have to hand it to them. In recent years the ad-man and ad-woman has me convinced that he is informing me – at the same time he is pitching his product. If I am not mistaken, the snake oil salesman used a similar tactic.

Ed

Again, this was unsolicited. How long did he spend writing this? How many times does his frustration have to hit the boiling point before he writes this? My guess is that he’s been frustrated for a while.

Now, this is why social media (and the design of it) is so important. In most cases, an email like this dies on the vine, its recipient acknowledges the difficulty but moves on. In some cases it makes its way onto the web, as it has here. In a small number of those cases, someone from Yahoo might actually notice it. In a small number of those cases, it might actually be discussed at Yahoo. In a small number of those cases Yahoo might actually do something about it.

Here’s the major point: The person’s frustration is already there. We have evidence of it. It has little to do with what exposure the email gets. Whether it dies on the vine or whether Jerry Yang (CEO of Yahoo) emails him directly to follow up, my father-in-law is extremely frustrated. If nobody read the email, he’s upset. If it makes the front page of the NYTimes, he’s still upset. That’s the core issue…there is frustration out there…is anybody listening closely enough to do something about it?

The action of a company would depend on how many people are having this conversation. Are other people getting annoyed at the long download times and advertising on the Yahoo Movies page? Or is this an isolated incident? If it is an isolated incident then Yahoo should not do anything about it. But if this is a trend…

Notice that my father-in-law doesn’t use the TV listings on Yahoo anymore?

Yahoo has an advantage here…Yahoo already knows how many people are saying things like this because their own search engine software is scouring the web even as we speak. Or so I assume…

But what about companies who are being written about but don’t know it? What about companies who are annoying people like Yahoo is annoying my father-in-law and the only time they’ll even have an idea is when that person cancels the relationship? That’s the power of designing tools like blogs that enable customers to speak. The answer is already out there.

Just waiting to be found.

In a fascinating piece on the amazing growth of the photo-sharing site Fotolog, Jason Kottke clearly articulates a growing problem in design:

“Fotolog…relative to Flickr…has changed little in the past couple of years. Fotolog has groups and message boards, but they’re not done as well as Flickr’s and there’s no tags, no APIs, no JavaScript widgets, no “embed this photo on your blog/MySpace”, and no helpful Ajax design elements, all supposedly required elements for a successful site in the Web 2.0 era. Even now, Fotolog’s feature set and design remains planted firmly in Web 1.0 territory.”

How do sites with sub-optimal visual design and technology grow so big and become so successful? How are MySpace, Fotolog, and Craigslist so popular in an age that values stunning visual design and amazing technology above all else? Conversely, how is it that Flickr, full of beauty and Ajax, is being overtaken by a site as boring as Fotolog?

Aye, there’s the rub…a rub that defines the current state of web design.

First off, a little throat-clearing. We’re dealing with Alexa stats here, so there are no guarantees that anything is accurate. Just because Alexa shows that Fotolog gets more traffic than Flickr doesn’t mean that it is…it’s kind of like listening to a reporter who usually covers political news tell us what’s going on in Silicon Valley. Suspect, to say the least. But for the sake of argument let’s assume that the trend is right, and that Fotolog is overtaking Flickr in terms of traffic.

Page views and Ajax…a match made in Hell

Well, one reason why Fotolog might appear so successful is the very technology that Jason mentions: Ajax. Page views are a metric that Alexa uses in its traffic calculation. But when you switch to an Ajax interface, your page views plummet. For example, when people want to add a tag, change a headline, or edit a photo set on Flickr very few page views occur. You’re simply interacting with a single screen that doesn’t refresh, but sends and receives requests in the background. This undoubtedly has a huge effect on the page views on Flickr.

Fotolog, on the other hand, gets a page view anytime a person wants to change anything. Therefore, less efficient bandwidth consumption and server usage actually gets Fotolog much higher traffic numbers…which is pretty damn ironic.

What’s more ironic is that this is an increasing problem on huge advertising sites and few people want to talk about it. What’s at stake? Billions of dollars that are wrapped up in page-view models where money changes hands depending on what “traffic” a site receives. And for years that traffic depends on page requests to a server, which of course happens even when people are doing simple things like changing a photos headline. So while companies realize that using an Ajax interface, when done well, can literally save millions in bandwidth costs and actually provide a faster, easier-to-use interface, they also realize that their advertisers only know one metric: the page view.

I’ve talked to some folks at Yahoo about this, and they say that their discussions on this topic get pretty tense. This is a huge problem for them because so much of their revenue is advertising based but they know that the future of interface design is elegant Ajax. This problem has been known for some time, but we’re still at the start of the huge effort in migrating away from the page view as a valuable metric for anything.

Technology doesn’t a great value make

Jason makes a strong case that technology is over-valued. I think he’s exactly right when he says:

“Maybe tags, APIs, and Ajax aren’t the silver bullets we’ve been led to believe they are. Fotolog, MySpace, Orkut, YouTube, and Digg have all proven that you can build compelling experiences and huge audiences without heavy reliance on so-called Web 2.0 technologies. Whatever Web 2.0 is, I don’t think its success hinges on Ajax, tags, or APIs.”

This is the exact problem I’ve been talking about lately: in some cases visual design and/or technology are trumped by other aspects of design.

In my Social Design talk, which I most recently gave at the Web App Summit, I ask this question: What are the most successful web sites in the world? The answers are the ones you would expect: Google, YouTube, MySpace, Yahoo, Craigslist, Amazon, eBay.

But then I ask the question slightly differently: What are the most well-designed web sites in the world? Outside of a minimalist Google, there is no overlap for most folks. None of the others on the list are “well-designed” in their minds…they’re simply successful, poorly-designed sites. They attribute the success of these sites to other factors: being first in the market, having economies of scale, etc.

From a visual design standpoint they might be right: these sites aren’t going to win any visual design contests. But the value of these sites goes so far beyond the visual that to judge them by the way they look is to completely miss the boat. In our testing at UIE, for example, we’ve never had anyone refuse to shop at Amazon because it doesn’t look great…in fact people are most passionate about Amazon because of the value they get from reviews…and the rest of the socially-focused features there. People love Amazon, and it has nothing to do with its visual design!

And people are passionate about the other very successful sites, too. To Jason’s point, the major value of all of the successful sites doesn’t rest on what specific technology they use or whether they have tagging. Instead, the major value rests on social aspects of the design…take away the interaction of the communities on these sites and there is very little value left in them. Take away the reviews from Amazon and you’ll hear a great big sucking sound of folks rushing out to buy their wares on some other site…

Similarly to Amazon, Fotolog relies heavily on social interaction, in their case sharing photos with friends. This is the primary value of the site, not how they do it from a technological standpoint.

The usual red herring: judging a book by its cover

Ignoring visuals and technology (at least temporarily) is a big change for many designers and technologists. Why? Because technology and visuals often get the credit when things go well, but aren’t really talked about when things go contrary to our assumptions. That’s exactly Jason’s point: why is it that Fotolog uses inferior technology and visual design and still succeeds?

I think the answer is that the differentiator on the Web right now isn’t great visual design or technology, although those help out tremendously (don’t get me wrong!). An analogy might be in order here because so many people think I’m trying to denigrate visual design…I’m not! Here’s an analogy:

Every time George Bush makes his State of the Union Address he speaks very clearly, his words are well-chosen and his speechwriters are obviously top-of-the-class. They communicate very well, and for the most part every single person who listens or watches the address knows exactly what George Bush is trying to say. Speechwriters learning the craft would do well to emulate the skill and technique of Bush’s speechwriters. Even so, the address is a bunch of statements that most people disagree with: most people want the U.S. out of Iraq and observe that the efforts there have largely been a failure. Even Bush’s own party is now alienated. But the State of the Union Address itself is well-executed: it’s clear communication…Bush is just sending the wrong message.

(update: several folks are angry with me that I used a political analogy…I’m certainly open to suggestions for future analogies where the communication is clear and well-executed but fails to deliver the right message to the audience)

This is the same with visual design: you can execute beautifully but if the message you’re sending isn’t the one the audience wants to hear then the overall design will be a failure. I believe this is what Jason is talking about with his repeated references to “Web 2.0″. He doesn’t see the value in the majority of so-called Web 2.0 services…they might look great and have interesting technology but if they don’t actually improve our lives…then what good are they?

Visual design is about communicating a message well…getting the point across. The problem comes when the message being communicated isn’t the right one…and that’s exactly what we’re seeing so much of…so many sites have great visual design and great technology but just aren’t sending a valuable message…

Where are all these sites? They’re everywhere: they’re the ones you’re NOT using.

There are two primary aspects of design: communicating the right message. Why is this two aspects? Because one aspect is communicating a message well and the other is making sure it is the right message in the first place. Perhaps this second part is what is called design strategy these days. I don’t know, but I know that one needs the other in each and every design project.

Preventing valueless design

We need a new way of thinking to prevent valueless design. Valueless design is like a George Bush speech: well-executed but wrong. While it may be communicating beautifully on one level, the impact on society may be minimal or, even worse, negative. We need design that provides real value to humans.

The new model as I call it is social design: a focus on the social lives of users, the context of how people live, and the connections they have with their family, friends, and loved ones. It’s about the daily activities that people care about, that make their lives richer, more fulfilling, and that have very little to do with how a piece of software looks or works behind the scenes.

But that’s just how I see it. I’m sure that other ways to get people in the right design mindset. I believe the best designers not only execute technically well, but have the mindset to discover the right design. They’re open to new ideas, passionate about what they do, and focused on the lives of their users in order to prevent sending the wrong message.

“Based on your own family’s experience, what do you think we should do to improve health care in America?”

This is amazing on several levels.

One, Clinton is actually asking the American people what they think, rather than assuming or generalizing from the party she’s a part of. (this doesn’t mean she’ll listen, but it’s a start)

Two, Clinton is using Yahoo Answers, a publicly-accessible social software app to ask the question. In the past year Yahoo Answers has been a runaway success for Yahoo, racking up millions of users.

Three, in just two days there are over 35,000 answers!.

As Loren Baker points out, this type of content is hugely important. It’s a record of the public’s view on a question in free-form, going way beyond an black and white vote. Many of the answers that I have read so far are pretty solid: passionate and reasonable answers to the question.

Is this is an indicator of another level of political discourse going forward?

Christina Wodtke (who wrote the book on Information Architecture) is angry about its impending death:

“I recalled a recent blogpost by Adam Greenfield and I found a clue. I think he, and Peterme, and Lou and Peter Morville… well, we’re all outgrowing our favorite pair of jeans: IA. And the waistband is cutting in badly, but it’s our favorite pair, so of course we’re crabby. We’re all going to stay crabby unless we finally take them out of our “skinny” drawer and give them to goodwill.”

Yes, indeed. IA as it has lived will soon die. Not because it wasn’t valuable, not because IA’s didn’t do great work, but because the Web is moving on.

The problem is that IA models information, not relationships. Many of the artifacts that IAs create: site maps, navigation systems, taxonomies, are information models built on the assumption that a single way to organize things can suit all users…one IA to rule them all, so to speak.

Clay Shirky, in his talk Ontologies are Overrated, equates this type of categorization with organizing the world in advance. He uses the dichotomy of browse vs. search as a wedge:

“Browse versus search is a radical increase in the trust we put in link infrastructure, and in the degree of power derived from that link structure. Browse says the people making the ontology, the people doing the categorization, have the responsibility to organize the world in advance. Given this requirement, the views of the catalogers necessarily override the user’s needs and the user’s view of the world. If you want something that hasn’t been categorized in the way you think about it, you’re out of luck.”

Many IA’s won’t stand for this, however. Their response would be something along these lines: “unchanging taxonomies aren’t what IA is about…it’s about organizing information around the user’s needs, and practices such as card sorting help to do that”.

In addition, writers in information architecture have reacted strongly against ideas such as folksonomies, which are navigation structures built out of one’s own tags. Peter Morville, in his book Ambient Findability, states:

“when it comes to findability, their (folksonomies) inability to handle equivalence, hierarchy, and other semantic relationships causes them to fail miserably at any significant scale.”

This is a valid reply, of course, except that it’s completely wrong. Equivalence is handled by similar tags and tag clusters, hierarchy is handled by nested tags, and it’s pretty clear that both Flickr and Del.icio.us (and many other sites using folksonomies) can scale.

Thomas Vander Wal, in a recent reply to Beneath the Metadata: Some Philosophical Problems with Folksonomy, an article critical of folksonomies (a term he coined), gets at the heart of the problem here:

“This assumption…that taxonomies are great and help people find things by providing the authoritative terms is wrong. Taxonomies are always less than perfect and most often far less than perfect for helping people find and refind information they need. But, we do need taxonomies to provide that foundation structure. We need solutions that can help the many people whose terms and vocabulary are left out of the taxonomy.”

This is, on some level, a platonic vs relative argument. Either you believe meaning is inherent in the natural structure of the universe, or you believe that meaning is relative, personal, and different for everyone.

The biggest cleavage along these lines, as Shirky alluded to, is Google Search (meaning is relative and can be modeled by links) vs. Yahoo Directory (meaning is inherent in the structure of information). We all know who won that battle, but did you know that the Yahoo Directory isn’t even on the Yahoo homepage anymore? Yahoo has all but demonstrated that the directory model, and not the folksonomy model, doesn’t scale.

In many ways, the success of Google’s Pagerank algorithm was the harbinger of all this. The simple idea that people’s actions model meaning better than a directory (even a flexible directory) is a critical step forward in thinking about the Web. The innovation we’re seeing with folksonomies, recommendation systems, social networking sites…all have their roots in the idea that modeling what people actually do on the Web is the best way to provide answers for them. And, perhaps more importantly, it is an admission that we simply can’t predict the future…we can’t design a perfect information architecture, and to attempt to implies that the world we’re modeling doesn’t change.

That said, I’m not claiming that information architecture is bad. In all probability an IA would assume that Search is part of IA, that flexible metadata is part of IA, and most of what I’m using as counter-examples are part of IA.

But the fact is that IA is a theory about the inherent structure of information…the architecture of information…and if we are moving away from that we should call it something else.

Relationship Architecture, perhaps?

In the end, Christina suggests that it is all about change, and that explains why she’s angry:

“Anger is almost always based on fear, and change fuels fear.”

When I think of the word “video”, I immediately think of Youtube. When people want to upload “video”, they immediately think of YouTube. When people talk about where they saw the latest episode of the Daily Show, they talk about YouTube. When advertisers think of “video”, it’s all YouTube.

YouTube is what people think about when they think of the word “video”.

And because YouTube has top-of-mind, it means that people are not thinking about all the competing services out there. Google Video, Yahoo Whatever, or Microsoft Whatever, and the countless other video startups that want even a sliver of that ridiculous pie. Those services have very little top-of-mind share for the word video. (for the record, it’s MSN Soapbox and simply Yahoo Video).

Top-of-mind share is really interesting because it has the potential to be so transient, yet isn’t. We’re talking about what web site comes to mind when you think video, and right now that site is YouTube. Couldn’t that change in an instant? Couldn’t some other service easily eclipse that overnight, and tomorrow everyone will simply have another site in mind when the word video pops up?

Google doesn’t think so. That’s why they spent an unimaginable amount of money on a site who they compete with directly with their own video service. Google already has much of the technology. It’s not like they’re just blindly entering the video game…they’ve been trying! And in one year YouTube has rebuffed their attempt!

Google is making a huge bet that YouTube will stay top-of-mind for a long time to come, at least long enough to gain much of their money back on advertising and search-related ventures.

Google, of course, has top-of-mind for the word “Search”. They have for several years now, and probably will for several more. Even in an age where Microsoft can redirect you to their search engine simply because you’re using their crappy browser, Google has conquered the Search top-of-mind. So, even if other companies come up with better search than Google, it will still take years before they wrest away top-of-mind.

The importance of top-of-mind cannot be understated. If a web site has top-of-mind, it is the first thing people talk about. Like MySpace in social networking, Netflix in movie rentals, Microsoft in monopolies, Macintosh in fine computing.

And now YouTube in video. Even the founders of YouTube realize this. Check out Chad and Steve’s personal message about the announcement. They acknowledge their dominance in video, saying that “two kings have gotten together, the King of Search and the King of Video”.

Susan Mernit has a great piece on the YouTube deal. She’s looking at it from the perspective of what Google Didn’t Buy. And what Google could have bought but didn’t, in her estimation, is a little company called the New York Times. She points to the staggering implications of this, and what it means for social media:

“The point here–just to kick it a little harder–is that this is yet more evidence how social media platforms that are shifting the paradigms in a profound way–Not only does YouTube have a mass market, it’s video on the web appeal that the more high-brow Times will never have (Is YouTube the next MTV?). Furthermore, it’s a platform that gives Google the opportunity to morph into a multimedia MySpace ecosystem, way beyond what Orkut could ever be–and most cruelly, it’s something that teens and twenty-somethings care about, which may no longer be the case for The New York Times.”

In case you missed it, here’s part 1: The Lifecycle of Design: Part 1

Well, we ended up archiving it in Writely, and filling it out a bit. Luke’s got the first part up now. (I’ll be publishing some parts of it during the week).

The Lifecycle of Design: Part 1

Hansell describes the situation like this: Yahoo is consistent, turning out useful and solid designs like Yahoo Maps and Mail while Google goes for the WOW factor, turning out semi-finished designs in an attempt to impress people enough to lure them to their platform only to buff on a shine later on.

On the whole, this characterization seems reasonable. Yahoo does seem to put more work up front into their designs, while Google seems to experiment more. I, personally, find the experiments fascinating. Hansell characterizes the situation as a defining point in the “battle” to come:

“Do Internet users prefer services that are consistent and predictable, like those offered by Yahoo, or are they more interested in Google’s wow factor? These two approaches define a pivotal front in the battle for online loyalty between the major players in the Internet search business.”

While I think Hansell overplays the war-like nature of the two approaches, it does seem like the designers at Google and Yahoo come at their jobs from two very different perspectives. Yahoo is full of “designers”, while Google is full of “engineers”. As time goes on, I think these two camps will get closer and closer together until you can’t tell them apart (if they’re not already).

Unfortunately, we don’t hear from the designers in this piece, we hear from executives, who, in the case of Google, seem to have little control over what their designers build (and maybe that’s a good thing). Hansell notes a recent comment by a Google chief:

“Sergey Brin, the company’s co-founder and its president of technology, said in an interview last week that he had been encouraging engineers to develop their ideas as add-ons for existing Google products, rather than as stand-alone services.”

As many of Google’s products come out of the engineer’s personal projects, this might be a change in course for Google. It’s possible to create new, socially adopted software, but it will be increasingly difficult as the canaries in the goldmine YouTube and MySpace beckon new competitors. It makes sense for Google to start from within, and create new products that leverage the existing user base instead of trying to create entire new ones. Yahoo is good at doing that.

However, the power of Search changes things. As I wrote on BrainSparks recently, Yahoo is trying to get people to use Search from the browser, which will effectively get them away from the self-advertisement of their homepage. So, if people don’t go to the Yahoo homepage anymore for Search…which must the the biggest reason why they go there…how will they find out about all the other integrated Yahoo services?

And I wonder, did Alan Eustace, the senior vice president for engineering and research at Google, really say “We are trying to come up with something that is new and different, that makes people say ‘Wow.’”? That doesn’t sound right. Instead of “Wow”, which could mean any number of things, why not “I can’t live without that!”?

Perhaps its Google’s way of saying they want to innovate, but what company doesn’t?

“We are literally encoding the principles of freedom of speech and freedom of expression in our tools.”

I hope he’s right. It sometimes feels like we should be, but aren’t.

Clay’s pattern library is interesting, too, following closely on the heels of the Yahoo Design Pattern Library.

The interesting difference between the pattern libraries is that Yahoo’s is a library of interface elements, while Clay and Co’s is made up of social elements modeled in an interface. Both really great tools for discussion/inspection.

Web 2.0 – Leveraging the Network (2.74 MB pdf)

In the talk I spoke about how Web 2.0 companies distinguish themselves by leveraging the network of which they are a part. Brittanica, for example, has had a web site for quite some time and were slow to leverage the network in any particular way. Wikipedia, on the other hand, exists only because they used the available network to improve their contents communally. And Wikipedia, of course, is a much, much more popular site.

As in my last talk: Web 2.0 for the Rest of Us (which includes a podcast), I started down the road toward Web 2.0 from the standpoint of those Web companies who have excelled: Google, Yahoo, Amazon, and eBay. They obviously know more about succeeding online than anybody else, and have become so successful so fast that we often take them for granted, even though they are barely a decade old. So, I find it particularly useful to ask: What makes them so special? What have they done that others haven’t? And I find myself coming back to the same answer over and over: they know how to leverage the network. From Google’s pagerank algorithm to the APIs of eBay and Amazon to the movie ratings on Yahoo, these companies know how to harness the collective activity and intelligence of people to make their services better.

For those who want only the quick and dirty (without the pretty pictures), here are the talking points:

1. The home page is no longer the most important page on your site.
2. The information architecture that people use to find your content is, increasingly, not yours.
3. Each feature added to an application is more to think about – for everyone.
4. Folksonomies are a way for users to map their own, familiar vocabulary to your alien one.
5. Words are the currency of the Web. Spend the most time on your words.
6. Seducible moments are those increasingly rare moments when you can talk to your users in an appropriate context.
7. Recommendation systems are a forced move.
8. Users want control.
9. Users appreciate tools that help them make their own well-informed decisions.
10. The best software models human behavior.
11. Links model how users value content, and are only the start…
12. Sometimes it is easier to design for yourself than others.
13. There is always an opportunity for a better interface to data.
14. All things being equal, faster interfaces allow for more innovation.
15. Most people are willing to trade their personal information for good service.
16. As choices grow, so does the importance of learnability.
17. Redesigns are dead.
18. Network effects are rare, and killer.
19. Network effects work in the opposite way for teams building software.
20. Personal value precedes network value
21. People rarely do things for the “good of the network”
22. Del.icio.us, though providing very cool tagging features, is mostly about a single person remembering items for later.
23. “The accretion of tiny marvels can numb us to the arrival of the stupendous”

I would appreciate any and all feedback, as I’ll be giving this talk in the future and would like to improve upon it in any way that I can.

“the integrated DLA platform wars have begun!”

In addition to finding out that using a laptop 12-14 hours a day can affect my spine, I also found out about the amazing rise of math in business, from analyzing clickstreams to tracking blog conversations. It seems Google and Yahoo already have next year’s math grads lined up for jobs. They simply cannot get enough brain power to do what they want to do.

What do they want to do? Mine data, of course. From the mountains of search queries in Google to the ever-increasing purchase histories at Amazon, we have more data than we know what to do with. Even at the relatively tiny UIE we have more than we can handle. I simply cannot fathom what millions of users could do to a database.

Here’s an interesting bit about Yahoo:

‘At the Sunnyvale (Calif.) campus of Yahoo, chief researcher Prabhakar Raghavan heads a team of 100 mathematicians and computer scientists. Scribbling on a white board covered with equations, Raghavan describes Yahoo’s immense pool of data, featuring the online activity of 200 million registered customers, as Yahoo’s most precious resource. There is a whole world of uninvented businesses, he believes. They’ll come into being as Yahoo discovers new ways to satisfy the urges, curiosities, and desires of this customer base. The hints of these future businesses float in the oceans of Yahoo’s data. Raghavan’s mandate is to sift through that data and form new connections among consumers, e-marketers, and advertisers. Better algorithms, he says, “are critical to survival.”‘

In general, there are two kinds of user-supplied content which can be mined:

1. User-added content:
   Intentional content. That content which users input themselves. This includes blog posts, comments, reviews, ratings, links, RSS subscriptions, podcasts, and video.
2. User-generated content:
   Unintentional content. That content which accrues as a byproduct of the actions of users. This includes clickstreams, purchase history, RSS read stats, search history, and other artifacts of behavior. User-generated content serves as evidence that a user passed that way, like footprints.

This distinction may or may not be important. I don’t know. But we are seeing a tremendous amount of work in the area of aggregating these types of content in an effort to build recommendation systems out of them.

In general, though, I think we’re learning some basic rules of thumb. Recommendation systems seem to work better if they are built out of user’s direct preferences, like ratings or reviews. If you try to build them out of say, clickstreams, you won’t get the intentional feedback that you need. For example, Amazon gives recommendations built out of searches on their web site, even if it is something that you’ve only looked at as a gift for someone else. I recently did a search on knitting for my wife and now I’m stuck with knitting books for a while. However, their recommendations built on top of my wish list are much more valuable to me, and I actually find them useful.

Going back to the article, I liked this quote:

“People are complicated,…If you have a system, they figure out how to game it. Machines never do.”

During a meeting today at the Microsoft Search Champs Conference in Redmond, WA, Yusuf Mehdi, Senior VP of MSN Information Services, discussed the recent blowup involving the U.S. Government’s subpoena of personal information from major Search Engines including MSN, Yahoo, Google, and AOL. This was not the first time that the U.S. Government has requested information from corporations in this manner. It was, however, one of the most talked about, spurred on by a press release from Google, who announced that they had turned down the request. Soon after, it was revealed that both Yahoo and MSN has complied with it, casting an instant shadow over those companies. In response, Ken Moss, general manager of MSN web search, provided a few relevant details of the case on the MSN Search Blog.

Today, Mehdi added some detail concerning what actually happened when the request from the Government was made. First, the Government had asked for information that could identify people on an individual basis (most likely, an IP address). Microsoft declined this request, and instead handed the Government a watered down version of data, which Mehdi made clear did not include personal information. The information provided by Microsoft, Mehdi said, consisted only of a sample of search terms and their frequency, as well as a random sample of pages in the MSN Search Index.

Update: Ramez Naam, Group Program Manager, MSN Search, sends along a clarification: the DOJ didn’t ask for personal information specifically, they simply asked for logs.

This was a very hot topic the entire day today at the Conference. Not only are there differing viewpoints about what Search companies should and should not do, but the very relevance of data was in question. Is this a non-issue given that Microsoft didn’t hand over personal information? Did Microsoft make an error in giving in to the Government? What information did they actually give? Given that the Government has final say, does any of the MSN posturing matter? And finally, does the average Joe really care about all this?

To help answer these questions, a few members of the Search Champs crowd gathered tonight to record a podcast. The podcast members were:

• Joshua Porter ( Me – Search Champs Attendee )
• Thomas Vander Wal ( Search Champs Attendee )
• Chris Pirillo ( Search Champs Attendee )
• Dion Hinchcliffe ( Search Champs Attendee )
• Fred Oliviera ( Search Champs Attendee )
• Alex Barnett ( Microsoft )
• Brady Forrest ( MSN Search Team )

Update: Additional Coverage:
Alex Barnett (Photos)
Thomas Vander Wal
Dion Hinchcliffe
Fred Oliviera

And Robert Scoble has a writeup of the meeting: Search Champs Grilling MSN execs

This started out as a list of technological trends, with RSS, Ajax, and Ruby on Rails being the headlines, as all three had huge years in terms of implementation and being squacked about. But these things, while interesting, aren’t really trends in the way that people are using the Web. Instead, they’re trends in building. Nothing illustrates the disparity between technology and usage more than the what Yahoo had to say in their October whitepaper: RSS-Crossing into the Mainstream. They claim that while over 1/4 of all Web users consume RSS in one way or another, only 4% know it.

So, in the spirit of usage I offer the following trends, focused on the way that those in the curve use the Web. Those ahead of the curve are probably on to whatever will get mainstream next year…

The Subscription Model

Through blog and news feeds we are learning the value of subscription on the Web. Borne out of our experiences with newspaper and magazine subscriptions, we’re subscribing to information sources that are as specialized as we want them to be. These are written not by professional writers in prestigous publications, but by folks whose passion or interest is their ticket to the show. Their writing, never as clean as what you’ll read in the Times or the Post, is always as interesting.

The subscription model is about getting content on your terms and not on somebody else’s. It’s a pull technology, so we only get content when we want it. This is unlike a push technology such as email, in which you’re damned if you ever give out your email address to the wrong person. The power of the model is apparent every time we open up a feed and view an item in its perfect plainness. Imagine being asked 3-4 years ago if people would stand to read content that is styled generically and with very little logo or branding to speak of. It would take a powerful model to convince us of that, but the subscription model does just that.

Attention and Gestures

Our attention is all we have. To give or to receive. To not give or to not receive. All other value flows downhill. Attention is two way on the Web, and we’re finding it increasingly difficult to keep our attention on any one thing long enough to learn what we need to learn before moving on. The statistic that the number of scientific journals doubles each 15 years is piddling. The number of blogs doubles in a matter of months.

Gestures, as it has been explained, are the unit of attention. In other words, you can measure attention by giving weight to and counting gestures. As Steve Gillmor is quick to point out, however, gestures include the negative as well as the positive. If I link to a web page, then that means something about how I value it. If I don’t link to a web page, that can mean something too, but it may be much harder to discern. Obviously, not linking to the billions of web sites out there doesn’t mean that I value each of them as equally worthless. I simply make no gesture toward them. No gesture is different than a negative gesture. But if I have the opportunity to link to something, and don’t, then that’s a gesture of inattention. Count up those and you’ve got yourself an interesting metric…

Identity

Another problem arising from the increasingly networked world we live in is how to model our identity online. Right now we’re modeling it so poorly that we really don’t have an online identity: we have our information spread across siloes of data on dozens of web sites. Since we can’t aggregate it all, if we needed to, then we really be identified by it in any general sense. The symptom that afflicts users is that we can’t remember the passwords for each site. The bigger problem is that our data is everywhere outside of our control. We should be in control of our own identity!

Startups like Sxip aim to solve this problem by creating a single datastore that allows people to identify themselves to a third party, choose what information to divulge, and when. There are several other initiatives as well. One thing I really like about what Google and Yahoo do is that I only have one username and password for all their services, and given that I use several of each it saves me a little headache. But it’s not yet one ring to rule them all.

Identity will become important when people realize that we should be in control of our own identity data, that signing into each site differently is not only difficult but backwards, and that we have power as buyers. Until then, we’ll probably all use two or three passwords for the 20 or 30 sites we log into, thus taxing our minds less but also lessening security benefits across the board.

To be continued…

Vander Wal, as many of you know, coined the term “Folksonomy”. He used it to describe what was happening on two up-and-coming web sites: Flickr and Del.icio.us. Now those two sites belong to Yahoo!, millions and millions of dollars later.

It would be silly of me to suggest that folksonomies were the primary reason for the success of those sites. But it would be equally silly to say that they were non-factors. They were indeed factors. Big ones.

The reason is that folksonomies help us do something that taxonomies don’t. Back in January, when I started writing about folksonomies, I was grasping with what that was. I felt like they were really valuable, but I didn’t know exactly why. Remember these posts?

• A Self-Referential Demonstration of the Power of the Del.icio.us Folksonomy
• Controlled Vocabularies and Folksonomies: Why Change is Good.
• Folksonomy Notes: Considering the Downsides, Behavioral Trends, and Adaptation
• I’ve Heard of Folksonomies. Now How do I Apply them to My Site?

Yeah, me neither.

But now we know why folksonomies are valuable. It is because they do two things very well.

1. One is that they allow people to remember things in the way that works for them. Someone tagging pictures in Flickr or bookmarks in Del.icio.us can use tags that they’ll remember. Instead of working twice to shoehorn a resource into and out of a taxonomy, folksonomies let users do both tasks effortlessly. Reduced cognitive load. Don’t Make Me Think. (if you didn’t read Rashmi’s A Cognitive Analysis of Tagging, you should).
2. The second thing is that they easily enable a bottoms-up classification system. When folks tag multiple resources at the same time, it is easy to aggregate those tags and provide navigation and resources that expose them. This is the reason why popular resources by tags on del.icio.us (like Web 2.0) and Flickr Interestingness are consistently valuable.

It must be stressed, however, that over the last year that we’ve learned (nudged along by Vander Wal) that the first is much more important than the second. If something isn’t valuable personally, it will rarely be valuable for the community.

At any rate, folksonomies are now entrenched in the pantheon of design, in big part to Thomas Vander Wal. Congrats!