First batch of reviews completed

Figure 1: Mean proposal grade vs proposal rank. Proposals are graded on a scale of 0 (poor) - 4 (excellent), and the dashed horizontal line indicates the threshold of mean=2.0 that must be achieved for any proposal to be accepted. Vertical bars show the range of grades received by each proposal.

Astronomers taking part in the first Fast Turnaround (FT) cycle had until February 14th to submit their grades and reviews. The Gemini FT support team met this morning to go over the results. We reviewed the instrument configurations, observing conditions, target RAs, and total time requests of the top-rated proposals, as well as any special considerations such as timing constraints. We then used this information to construct a list of proposals which will fit in the available time and stand a reasonable chance of being observed. This provisional list very closely follows the reviewers' rankings even though, given that almost all GMOS configurations were requested and many of the proposals requested good weather conditions (see previous news item), we had been expecting that some fairly highly-ranked proposals may have to be excluded. Our next step will be to carry out technical assessments on the proposals that look like they will be accepted, replacing them with alternatives if serious problems are found. PIs will be informed of the outcome by the 21st.

We are planning to extensively analyze the review process as we gather data from this and subsequent cycles. However, the temptation to draw some preliminary conclusions has of course proved irresistible. Figure 1 shows the mean score of each proposal vs proposal rank, with the vertical bars indicating the range of scores received. Clearly, the range of grades for most proposals is considerable, with one even being awarded scores spanning the entire range from "poor" to "excellent". However, reviewers uniformly thought highly of the two highest-ranked proposals (accounting for ~7 hours between them), and the three lowest-ranked submissions received no "excellent" scores. The review process appears able to identify the best proposals and agree on a handful that should not be accepted, while proposals in the middle of the range receive less definitive evaluations.

Reading the individual reviews was also instructive. For example, although the need for rapid response is only 4th on the list of assessment criteria, we noticed that most reviewers appeared to give this considerable weight in their grading. Indeed, one reviewer recommended that proposals not strictly requiring Fast Turnaround should only be assessed by the regular TAC. We will therefore need to decide whether to emphasize that this criterion is less important than overall scientific merit, or whether it should indeed be given more weight. If the latter, no changes will be made until the March call for proposals is issued (to avoid changing the rules as people are writing proposals).

Initial, informal feedback about the process has been very constructive, and will help us improve the system (for instance, by building in better safeguards against accidentally assigning too many proposals to one reviewer!). PIs and reviewers will be sent a structured feedback form along with their notifications later this week.