The FIRST LEGO League Judging Deliberations process is used by the Judges at events to determine the appropriate award recipients. The process starts with Judging Sessions and ends with a series of votes to determine the award winners. The following is an overview of the entire process.
Step 1: Judging Sessions and Room Rankings
Following the Judging Briefing in the morning, Judges head to their rooms to prepare their room to interview teams. A judging pair will typically see between 10 to 12 teams at each event. The Judges interview each team to determine performance in each of the three areas.
As Judges see teams, they start a ranking of teams for their room. During the breaks between sessions, judges evaluate the team they just interviewed against the teams they have already ranked from previous sessions. This creates a “running ranking” that is updated after every session. After the final session, the judges prepare a final ranking for their room and deliver that to the Judge Advisor.
Step 2: Enter Rankings into Spreadsheet and Sort
Once all teams have been interviewed, the Judge Advisor collects Rankings from every Judging Room and enters them in to a spreadsheet called Judging Lite. This spreadsheet comes from FIRST and helps facilitate Judging Deliberations at Official Events. The Judge Advisor then acquires the most up to date record of Robot Game rankings from the Scorekeeper and enters the data into Judging Lite.
Judging Lite then calculates the “overall” ranking at the event. The Overall ranking is composed of a sum of the three room rankings plus the Robot Game ranking. Teams are then sorted by overall ranking in increasing order. Below is an example of data pulled from Judging Lite. (This is an excerpt, and only contains the top nine teams)
Reading the Table
Here is an explanation of important columns in the table:
- Overall / Overall Rank: As described above, the Overall value is composed of the sum of the three room rankings and the robot game ranking. The Overall Rank is derived from the ordering of the Overall column.
- Robot Design / Core Values / Project: The values in area columns are the final rankings from each judging room. Since there can be multiple rooms at an event, there may be multiple 1’s, 2’s, etc. in each column. The maximum value found in each of these columns will not exceed 12.
- Example: The “9” in the Robot Design column for Team 1115 comes from being ranked 9th out of 12 in their judging room.
- Robot Rank / Max / R#: These columns come directly from the robot game results.
Step 3: Discuss Top Teams and Decide Champions Award
After all rankings have been entered, the Judge Advisor begins discussion of teams eligible for the Champions Award. The Judge Advisor takes the top 2-3 teams by overall ranking and asks the Judges that interviewed those teams to speak about them. After all Judges have presented, the Judge Advisor sums up the discussion and asks for any other feedback or questions. If a judge feels another team should be considered for the award, that team is also brought into the discussion.
Once the discussion is complete, the Judge Advisor calls for a vote from the judges to select the Champions Award winners. These winners are then assigned the award in Judging Lite.
Step 4: Judges Break into Areas to Rank Area Awards
Following the assignment of Champions Award winners, the Judge Advisor instructs the Judges to break into groups based on their respective areas (Core Values, Project, or Robot Design) and decide any area awards such as the Core Values Award or Programming Award. The Judges create a ranking of teams for each award.
Step 5: Regroup and Assign Area Awards
Once the area rankings are complete, the Judge Advisor inputs all rankings into Judging Lite. In the event that a team is ranked equally in two area awards (For example, 1st Place Core Values and 1st Place Project), the Judge Advisor ask the judges to discuss this team further. Since a team can only win one award, the judges must decide which award best fits the team. Once all ranking collisions are resolved, then the Judge Advisor assigns the awards in Judging Lite.
Step 6: Discuss Judges Award Teams and Finalize Assignment
During the discussions above, Judges will nominate teams for the Judges Awards. These awards are given to teams that do not fit into the standard award categories seen in Awards and Advancement. The Judges are free to nominate any team that has not yet been assigned an award. The Judge Advisor then moderates a discussion for all nominated teams and the Judges vote on those that they wish to recognize with the Judges Award(s).
Once all recipients have been decided, the Judge Advisor assigns the awards in Judging Lite. The Judge Advisor then waits to receive the final Robot Game rankings from the Scorekeeper and confirms that the final rankings do not alter the eligibility of teams already assigned. In the event a team changes eligibility, the Judge Advisor then conducts a discussion and lets the Judges vote to change award assignments.
Step 7: (Qualifiers Only) Determine Advancing Teams
At Qualifiers, Judges also determine the teams that advance to the next level of competition. These teams are awarded the Advancing to Championship Certificates.
Teams winning any Champions Awards are automatically added to the advancing team list. The Judge Advisor lists all teams in order of Overall Rank and starts at the top of the list of teams. The Judge Advisor verifies eligibility (Robot Game ranking within limits set by Central Valley Robotics, no Red Level CV Concerns, etc) and then asks the Judges to confirm that this team should advance. Judges discuss any experiences and weigh the merits of each team. Once the Judges have selected enough advancing teams, the Judge Advisor asks the Judges to select one to two waitlist teams in the event one of the selected advancing teams is not able to attend the next level of competition.
Step 8: Prepare Award Scripts and Collect All Materials
Finally, once all awards have been assigned and time should allow, the Judge Advisor instructs the judges to prepare the award scripts for Closing Ceremonies. Judges will prepare a couple of sentences to help personalize the award.
The Judge Advisor then certifies all award winners. This process involves filling out the Award Record document, submitting the Judging Lite spreadsheet and award scripts to Central Valley Robotics, and packing the Judge Advisor materials to return to Central Valley Robotics. The Judge Advisor also collects and sorts all rubrics for processing at Central Valley Robotics.
Step 9: Closing Ceremonies
Once the scripts are ready and materials safely packed away, the Judges head to Closing Ceremonies to present the awards.