object1434799101

Conference PapersEven a modest application gamification can have a measureable effect, the goal of this post is to share a small gamification implementation, its components and show the results.

Introduction

Each year, HP has a global technical conference for its leading technologists. The only way to attend the conference is for individuals to submit a paper and have it reviewed and accepted into the conference by a committee of peers. About 1900 papers were submitted in 2013. Each paper is reviewed by at least 4 individuals so that a diverse perspective of its innovation can be developed. There were about 300 reviewers involved in performing these 7600 or so reviews. The process takes place over approximately a 5 week timeframe, in addition to the reviewer’s normal workload.

In recent years, there have been a negative trends in reviewer behavior directly impacting review quality. Since the only feedback provided to the authors about their paper (and how to make their work better in the future) came from the reviewers, addressing these trends was important to increasing the value of the process for the authors as well as the corporation as a whole. I led this effort and defined a gamification approach to meet some well-defined goals.

Understand the Goals

My view is that the first step for any gamification effort is to: Understand the goals. What change is needed and why. The two goals defined were:

  1. Increase the feedback per paper
  2. Increase the percentage of reviews that took place, as assigned, in the time available

Understand the Players

The second step is to understand the players. The players in this effort were the reviewers. The leadership (also players but of a different kind) was the program committee of the conference.

Understand the Metrics

The next step is to understand the metrics, how they measure the goals and how feedback would be provided to the players.

Three techniques were identified to provide feedback:

  1. A reward structure of points and badges was identified to recognize reviewers and specific reviewer behavior. Some badges were comically named to add a bit of fun to the exercise. For example, the Motor Mouth badge was for the individual who provided the greatest quantity of feedback to authors. Other badges were more performance-oriented, focusing on task performance within a specific timeframe— most feedback or reviews in a week. There were also team-oriented badges, like one to recognize the first team to complete all their reviews. Performing reviews and completing badges all added points to a reviewer’s performance.
  2. A leader board was created to show the top reviewers and review teams as measured by the points they accumulated. It was publically available and updated several times a week.
  3. An automated email was sent before and after the weekend, since that was when most of the review work took place. It was tailored to the individual reviewer’s efforts, providing relative performance information, badges they may have received, and the performance dashboard’s URL.
All information was analyzed and presented using the same review tools as the previous annual conferences, so a year on year comparison was possible. The only real change in interaction with the reviewers was the inclusion of the personalized analysis of performance and a leaderboard.

Points were awarded only after the reviewer feedback was entered into the review tracking system.

All public feedback was focused on the positive performance of top performers. Any feedback that could be construed as negative was included only in the personalized email.

A simple definition of review feedback quality was identified as word count. Obviously this is not ideal and could be easily gamed, but it was quantitative and easily compared to the previous year and viewed as a good starting point for comparison.

Throughout the review process the co-chairs of the conference were able to look at the progress and assess the standings of their team of reviewers. The results of the gamification effort demonstrated a significant positive performance shift:

Tech Con metrics

2010

2011

2012

2013

# of abstracts to review

1308

1592

1763

1880

# of reviewers

264

277

286

332

Percentage of reviews completed as assigned

98.69%

98.73%

95.99%

99.12%

Reviews per reviewer

24.78

28.74

30.50

28.31

Feedback word count to authors

Average

Standard deviation

Median

 

89.71

94.86

74

 

104.37

85.85

94

 

106.90

84.37

91

 

127.45

82.33

110

Feedback word count to other reviewers

Average

Standard deviation

 

4.64

16.07

 

4.39

15.49

 

4.73

16.97

 

5.76

17.18


The chart shows that the reviews definitely took place earlier in the process than in previous years. This shift in behavior enabled review coordinators to focus their attention on specific reviewers and situations earlier in the process, rather than waiting until the end of the review timeframe where the majority of reviews traditionally took place and there was no time left to respond in a controlled fashion.

Conference Paper Selection at HP

The two goals of providing more feedback and improving the number of reviews completed as assigned were met, reaching the highest levels recorded in recent history, even though only relatively simple techniques were used.

Review lessons learned

The most important lesson learned during the effort was making the rules for the gamification effort and the point system more public and well-defined, ideally including examples, early in the process. At the end, when some of the final performance badges were awarded, reviewers were upset that their standing continued to shift, long after they had completed their reviews. Although it was clear that there were points for team performance and other badges that couldn’t be awarded until the end of the review period, there were numerous questions about the shifts in point totals at the close of reviews. Having a well thought out communications plan, reviewed by multiple parties, would help ensure the approach is explained sufficiently for all the players, making communications more effective.

Another lesson came from a survey sent to all the reviewers to gather their perspective on the effort. Almost 50% of those involved thought that the gamification effort had no impact on the quality of the reviews provided and 27% thought it had no impact on the timeliness of the review process. Only 45% thought the gamification process should be done again in the future. These results show that a gamification effort may be effective and still not be viewed has having much impact by those involved. During the entire process no additional effort was required from the reviewers other than to read and delete the email status messages that were tailored to their performance. Once the reviewers saw the actual impact on their behavior as a whole, they changed their perspective on the efficacy of the effort, but not all.

Game elements like badges and leaderboards are an important aspect of the effort, but should not be the only component to create a game experience. The game designer should build a gamification experience where fun interaction and collaboration take place—where those involved are actually interested in greater interaction and understanding of what is happening.

When performing initial work on an effort like this, you will get things wrong. Be sure to understand what happened and survey those involved so improvements can be made. You should feel free to make adjustments along the way if the metrics gathered to not line up to the expectations.

See also my slide deck on Slideshare.com with more information:

 

 

Charlie Bess is a fellow at HP (HPES ABS Americas CT). He can be contacted via his This email address is being protected from spambots. You need JavaScript enabled to view it. and you can read further posts from him on The Next Big Thing Blog or @cebess on Twitter.