Formation of the Committee
Enough people should be chosen to cover the review needs. CVPR has been using a multi-level committee with "Area Chairs" (who might be viewed as the old fashioned Program Committee) and the 100+ members of the program committee who are responsible for reviewing the papers. The Area Chairs then meet to make the decisions.
Program Committee (or Area Chairs) Meeting
This meeting is used to decide on the content of the conference. The general format used at CVPR99 was:
- All papers have been reviewed by 3 reviewers.
- Area Chairs have asked for more information when the decision is not clearcut (i.e. the not uncommon -- absolute reject and award quality reviews for the same paper) Note: This requires effort on the part of the Area Chair and advance checking of the reviews.
- The Area Chair has made an initial decision. At this point there is no absolute numeric limit on how many papers.
- The papers are grouped by area (by number caused some problems, and there is a strong correlation between when the paper was submitted and the quality) and given to small groups of area chairs -- different areas -- to provide some uniformity. The small group can raise or lower the proposed rating. If everyone is there and all the evaluations were done before the meeting, then this can be finished in several hours -- by early afternoon of the first day.
- Go through all papers (this is important, and you can deal with some in a few seconds). Show the overall rating information, verify that the Area Chair and the small group agree, deal with what that rating means (i.e. reject, definite accept, definite oral, definite poster, etc.). Do not worry if there are papers that can not be completed at this stage. If there are major questions, the paper can be reread for a more complete evaluation.
- Go through the papers which were not decided absolutely again, i.e. all the borderline cases. It worked to have a category of "maybe" -- can not decide now, but wait until we have looked at everything. This helps keep things consistent so that after looking at all papers you can decide how some earlier papers can be handled. (I.e. early in the day, this paper looks good, but after seeing the other 400 papers, this paper looks bad.)
- Repeat this step until satisfied.
Things that make it go well:
- On line access to the conference web site with display of the reviews and decisions displayed to those present.
- All of the area chairs present.
- Ability to show the running totals (from the web site scripts) so that the committee can know how many papers can be accepted.
- Keep the reviewers names out of it, unless it is important for the discussion and authors are not present.
- It works to remind people to be professional about it and in order to not make a scene, the author of a paper can remain in the room and sit quietly. Asking the author to leave draws attention to the identity of the author and slows down the process.
- Papers by Program (or General) chairs: There are two views -- these few people should not submit a paper, or they can. (A less common view is that they should be given one paper slot and only one, without going through the review process.) It has always worked best if the person running the meeting does not have a paper (this can be dealt with by having multiple people run the meeting and trading off well before any conflicts may arise).
- When conflicts arise (or are claimed) be prepared to deal with them. The most important way to defuse the conflict is to move the decisions to another party -- if there is a conflict involving a program chair, move that decision elsewhere. (This should never arise in the case of a paper accept/reject -- the program chair must always accept the committee decision without argument. But it may arise in the case of awards -- like reviews and areas, the group that decides on the award should have no connection to the authors.)
After the Meeting
After the meeting, review the results to make sure everything is settled. At the end of the meeting you should have time to discuss overall program issues -- how to arrange papers, etc. As soon as possible, inform the authors of the decision, then prepare the information for CS-Press. With the online system, authors rarely remember their exact login name and password so that the notification should provide that information to the author of record. This can save a lot of time.