Reviewing the reviewers

A major problem with research conferences is that anonymous review allows reviwers to get away with murder. Anonymity, meant to allow fearlessness in reviewing, has become a way to hide ills such as:

• not spending enough time reading the paper, so that the review is essentially content-free
• writing a biased review of a paper written by a rival or friend
• turning down a paper for repeating past work, but with no reference to the past work
• stating that the paper does not fit into the call for papers, but not justifying this
• using reviews as a way to advance a pet theory or position

The first of these problems is probably the most common. Researchers are keen to be on a program committee, but not that keen to fully discharge the associated duties!

The result of this drop in review quality has led to discouraged authors, widespread assumptions of conference bias (every Sigcomm outrageous opinion session has a predictable riff on the fraction of the program contributed by PC members), and uneven paper quality even in the 'top' conferences. Something has to change.

Michalis Faloutsos and Kevin Almeroth recently proposed total transparency as a solution. I think this is a pipe dream.

Here is a simple alternative: let the authors judge the quality of the reviews they get, say on a scale from 1 to 10. At the end of the review process, every PC member gets to see the mean score of every other PC member. The score, however, is confidential, and not to be discussed beyond the PC. In particular, the score is not known by authors. Also, note that reviewer anonymity is preserved.

Why will this work? The idea is that the peer group of a PC member is the rest of the PC. No one wants to look bad in front of a peer group. So, every PC member gets an incentive to do a good job. Obviously, someone who cannot commit the time to do a review is likely to bow out of the review process (and the PC) altogether. This should eliminate the root cause of most of the shoddy reviews!

Second, if a PC member delegates reviews to grad students or untrained reviewers, they now have an incentive to make sure that the reviews are well-done. It is their reputation at stake.

Third, although reviewer scores are confidential, they are likely to be informally discussed in the PC member community. So, consistently poor reviewers will soon be eliminated from program committees.

Finally, the system sets the right incentive in terms of reviewer area. Currently, a reviewer not in the area of work of the paper does have the option to ask not to review the paper, but this option is rarely exercised. Instead, the reviewer simply marks his or her expertise level as low. In the new scheme, this will almost surely result in a paper with a low reviewee score. So, the reviewer has every incentive to better match papers with his or her expertise.

Of course, this process is not perfect. For instance, a reviewer who gets out the hatchet only some of the time will get away with it. However, this is unavoidable. This scheme is not perfect, but, in my opinion, much better than the current situation.

A more subtle problem is that this may discourage PC participation by senior researchers whose opinion is valuable PC discussions, but who do not have the time to do detailed reviews. One solution would be to have a PC where some PC members are expected not to do any reviews, but to participate fully in PC meetings. Their identities do not need to be known in public.

All in all, I think this solution balances anonymity and transparency. Please feel free to send me any comments.

Acknowledgement: these ideas were refined over lunch at Mobisys 2006. Unwitting guinea pigs for these ideas include Victor Bahl, Dan Siewiorek, Satya, and Nigel Davies.