Are you rewarding good behavior, or just any behavior?
I was recently chatting with an entrepreneur who was considering building a leader-board for his social web app. The purpose of the leader-board, he told me, was to display those users of the app who were most active, with the idea that this would incentivize them (and those not on the leader-board) to participate more. With the lure of climbing the leader-board and sitting at the top, users would engage more deeply on the site and grow the app faster.
The entrepreneur told me that he would build the leader-board by counting the actions users take such as posting, commenting, and reviewing. The highest position on the board would be the person who had taken the most actions.
This problem of engagement is a common in social web applications. Many apps start off with very few users, and in some cases (such as a social network) the app has little value until a lot of people are using it. Naturally, one of the early goals is to get enough people using the software so that newcomers have a sense of presence there…newcomers can quickly observe what’s going on and perhaps find their friends. As more people join the site and engagement increases, a virtuous cycle is established.
In this light creating a leader-board makes sense…the site creators want to reward people who are engaged the most. The mere presence of the leader-board creates a game people can play…and the gaming world has shown us that people don’t need much more than a simple challenge in order to stay engaged. (I can’t tell you how many people play minesweeper at the local library…a game which is to me is incredibly boring but to others is a nice, simple challenge to while away the time).
Early on in the building of new systems we want any engagement…people just using the app. However, we soon discover that there are different types of engagement, some of which are desirable and some not. There is good behavior and there is bad behavior. Many times mere activity doesn’t mean good behavior. Good behavior is a targeted, concrete action that makes the experience of others better.
In other words, when we build a reward (or reputation) system (a leader-board, reward badges, awards, etc) we should try to tie the reward to not just to some behavior, but good (positive) behavior. I suggested to this entrepreneur that he identify those behaviors that he wanted to see, but were also useful to others in the system, and design the app in such a way that other members of the service could give feedback on those actions. This might come in many forms, such as a way for people to rate other people’s submissions (ex: Digg’s Digg this, Amazon’s Is this review helpful?, LinkedIn’s recommendations, and eBay’s seller ratings). It could also come in the form of actions the system can aggregate, such as the New York Times’ Most Emailed or Most Blogged About. While these last two are slightly different, they aren’t focused on counting an individual’s actions but on the collective action of many, so counting is OK in this case. Whatever it is you’re counting, you want to make sure that people can’t simply run up the count without providing real value to the service.
In general, when building a reputation system it is good practice to continually ask “Are we rewarding good behavior, or just any behavior?”.
Make them Care! - Struggling to communicate the value of your product or service? I'm writing a new book that shows you how to make people care about your product or service by clearly communicating the most important bits. For designers and marketers creating product web sites. Find out more.
Links to this Post
Comments
1. Andrew 12:43pm, Wed 15th, 2009
Also, obviously the leaderboard based entirely on cumulative actions will always favor the old timers. New users arriving after a few months of leaderboard activity could never hope to catch up (except maybe through a lot of “me too” comments to inflate their score).
2. Nate Klaiber 1:36pm, Wed 15th, 2009
There is a delicate balance that has to take place here. You can see this in some of the bigger social networking sites (traffic wise, not necessarily quality wise). Take Digg for example. Early on they had people climb their way to the top by activity, then those people were getting paid off to help people get on Digg. Digg recognized this and took action.
It takes work to truly gauge the quality. A rating system is another alternative, allowing the other members to truly rate the usefulness of the different features, but that is also flawed given the nature of trolls and others who want to ‘compete’.
I think the balance is three-fold: 1) Quantity (how much is the user doing), 2) Quality (what is the user doing to add value to the service), and 3) Checks and Balances – where you as the site owner need to constantly gauge quality, as well as allow the users to do the same. It’s a tough mixture to get right (and this is a simplified view of something very large).
3. Bander 2:33pm, Wed 15th, 2009
Mine sweeper and solitaire… I sometimes think the main drive behind new devices is just to have new ways to play solitaire…
4. Lisa Rex 3:37pm, Wed 15th, 2009
GenealogyWise is a new social networking site for genealogists (www.genealogywise.com). Today, I received an email announcing they’re awarding cash prizes for the following:
- $100 for the member with the most confirmed friends in GenealogyWise.
- $100 to the owner of the group with the most members.
- $100 to the owner of the surname group with the most members.
- $100 to the creator of the genealogy-related video on GenealogyWise that has been viewed the most times.
- $100 to the member who has uploaded the most historical photos.
- $100 to the person who adds the most genealogy-related videos.
- $100 to the person who has the most popular blog entry (most page views).
- $100 to the most active member in the forums.
While some appear to be awarding ‘good behavior’, this probably isn’t quite the case. Will members start spamming and harassing each other to ‘View my blog post’ or ‘Join my surname group’? Collecting the most friends completely devalues the system of friending others.
I’m not an active member, but the quality of GenealogyWise service has just plummeted in my opinion. Time will tell….
5. Evan Meagher 4:47pm, Wed 15th, 2009
Stack Overflow’s badge system is a perfect example of this. By targeting and offering badges for behavior that they want users to engage in, the SO team has perfectly incentivized positive behavior.
6. Mike 7:29pm, Wed 15th, 2009
I agree with Lisa Rex. There are different ways of incentivizing behavior, and some ways might lower it’s value to the community.
This is a problem of naively interpreting quantitative data. Maybe you’ve seen data that shows that pages with lots of comments are more likely to be commented on, so you incentivize commenting. But that’s a simplistic interpretation of the data that assumes that all comments are equal no matter why they were added. Incentivizing commenting changes their purpose from intrinsically- to extrinsically-motivated, and that changes everything. If the users believe that people comment to just drive up their scores, why bother reading them?
So I think you have to make sure that incentives enhance existing motivations rather than supplementing them, even if the new motivators create more activity.
Another dimension is the attitudes of your user base. If they are primarily young men and boys, competition and leaderboards might make sense. But I have personally witnessed leaderboards fail disastrously because the user base was K-12 educators. For them, humiliating your friends in competition was seen as something to avoid.
7. NatC 7:13am, Thu 16th, 2009
We just launched a new reputation system on the Tom’s Hardware Forums (http://www.tomshardware.com/forum/). We made sure to base it on a “solved topic” feature. Users earn the most points when they select a best answer to their question (25pts), or are selected as the best answer (40pts). We expect this system to put quality to the front, even though just posting an answer will still be rewarded with 4 pts.
I also agree with the comment on new users not able to catch up. This is made easier by non-linear scales for the reputation levels, so that you can reach the first levels very quickly if you post regularly. We are also considering decreasing the score of users that have been inactive for some time.
Last note, I agree with Evan that Stackoverflow is really a model to follow.
8. David Priemer 11:19am, Thu 16th, 2009
A very timely post for us at Rypple as just yesterday we released the first iteration of the Rypple leader board: a small widget on the user’s main page to highlight the “feedback gurus” at their company and the Rypple community.
Needless to say, countless discussions & email threads were exchanged trying to figure out how we could encourage the desired behavior and create “social proof” around or service by highlighting select members, while at the same time discouraging the gaming element Lisa mentioned.
A couple key takeaways from our experience:
1. Keep it agile: reputation management is ticky so start with something simple, see if it does in fact encourage the desired behavior, and then take small incremental steps over time to improve your treatment.
2. Create a reputation system that jives with your target audience: I agree with Evan that stackoverflow is a great example…but our question was, would the mainstream users of OUR service (from all walks of life), gravitate towards a badging methodology in the same way a developer community would?
Time will tell! Will keep you posted
9. tom 4:14pm, Sat 25th, 2009
An experience I’ve had many times is that someone starts one group after another (usually with attractive, relevant sounding names), and then constantly seeds those groups with new discussion threads that don’t attract any participation by others. These folks would do great by some quantitative assessments, but I find them a turn-off and add no value to the network.
Another problem is participation by people who are paid to participate in networks, blog comments, whatever, on someone else’s behalf. After exploring the online freelance writer market I was shocked at the dozens of new solicitations every day for ghost writers of such content for SEO rather than real participation. How do you weed them out of the assessment of a network?
10. Ron George 3:41am, Wed 19th, 2009
I just commented about something like this on Paul’s blog, http://www.thinkoutsidein.com/blog/2009/08/personal-analytics/
I think one thing that we need to do is start thinking about personal reward systems. All participants are not created equally. Instead of thinking of each action as a “count” I feel we should start thinking of them as a particular action that has a particular response. There are so many factors that can be used to alter the results of each action.