The Contest Judging Problem

Background:

When determining the winner of a competition like the Mathematical Contest in Modeling, there is generally a large number of papers to judge. Let's say that there are P = 100 papers. A group of J judges is collected to accomplish the judging. Funding for the contest constrains both the number of judges that can be obtained and the amount of time that they can judge. For example, if P = 100, then J = 8 is typical.

Ideally, each judge would read all papers and rank-order them, but there are too many papers for this. Instead, there are a number of screening rounds in which each judge reads some number of papers ad gives them scores. Then some selection scheme is used to reduce the number of papers under consideration: If the papers are rank-ordered, then the bottom 30% that each judge rank-orders could be rejected. Alternatively, if the judges do not rank-order the papers, but instead give them numerical scores (say, from 1 to 100), then all papers falling below some cutoff level could be rejected.