Mapping the bid behavior of conference referees
The peer-review process, in its present form, has been repeatedly criticized.
Of the many critiques ranging from publication delays to referee bias, this
paper will focus specifically on the issue of how submitted manuscripts are
distributed to qualified referees. Unqualified referees, without the proper
knowledge of a manuscript's domain, may reject a perfectly valid study or
potentially more damaging, unknowingly accept a faulty or fraudulent result. In
this paper, referee competence is analyzed with respect to referee bid data
collected from the 2005 Joint Conference on Digital Libraries (JCDL). The
analysis of the referee bid behavior provides a validation of the intuition
that referees are bidding on conference submissions with regards to the subject
domain of the submission. Unfortunately, this relationship is not strong and
therefore suggests that there exists other factors beyond subject domain that
may be influencing referees to bid for particular submissions.