<html><div style='background-color:'><DIV class=RTE>(Back to Jim's original line of discussion...) </DIV>
<DIV class=RTE> </DIV>
<DIV class=RTE>How does one compare judging results when (as in most local contests) there are only 2 judges per round/class? When there's a spread, which judge is right? In a world where the judges oare often the 'best available' rather than ones who meet a standard, is there any value in such a comparison? </DIV>
<DIV class=RTE> </DIV>
<DIV class=RTE>For that matter, when there are three judges and one's score is significantly different from another, do we assume (with or without knowing the identities of the judges concerned) that the one 'different' judge is wrong? If so, why?</DIV>
<DIV class=RTE> </DIV>
<DIV class=RTE>Is it more or less significant when one judge's score is different than the others' in ONE maneuver rather than across the entire flight? </DIV>
<DIV class=RTE> </DIV>
<DIV class=RTE>Eric has made great points in his points on this topic - especially about the fact that two judges will focus on different aspects of a maneuver, but frequently give very similar scores - for overtly different reasons. </DIV>
<DIV class=RTE> </DIV>
<DIV class=RTE>
<DIV class=RTE>FWIW, some personal info on scores at the recent Nats (from a first year Masters Guy who finished REAL close to the bottom of the pack - and deserved to)</DIV>
<DIV class=RTE>- One of my rounds had a spread across the three judges of more than 110 points. (Maybe some non-zero 'mercy' scores here? who knows?) </DIV>
<DIV class=RTE>- As a first time FAI Finals judge, one of the fnalists volunteered the information that his score spread for the round I judged was 11 or 12 points across 5 judges... <BR></DIV></DIV>
<DIV class=RTE>I'm not entirely comfortable with the idea of comparing the scores of a panel of judges -- without knowing their background, flying experience, judging experience, etc - and assuming that the 'low' guy is automatically wrong. -- especially if this practice is institutionalized, and is recorded as 'facts' in the files of some or all of the judges concerned. </DIV>
<DIV class=RTE> </DIV>
<DIV class=RTE>It seems that recollections of my college statistics courses reminds me that this line of thinking will result not in recognition of the best absolute standard, but rather in creating a de facto standard of mediocrity... </DIV>
<DIV class=RTE> </DIV>
<DIV class=RTE>My gut says that the 'right' answer, unfortunately, is </DIV>
<DIV class=RTE> - A REAL set of standards about what the rules say and mean - and Don seems well on the way to getting that done</DIV>
<DIV class=RTE> - An effective classroom instruction plan that makes those rules clear (and again, Don has done a great job in his 2005 Judge Certification packages for students and instructors) </DIV>
<DIV class=RTE> - A REAL judge teaching program -that would involve practical exercise / live judging followed by discussion after each maneuver (lots of fuel burned there!). This is the part that seems to need the most progress. </DIV>
<DIV class=RTE> </DIV>
<DIV class=RTE>Understand the pain of driving hundreds of miles and not getting reasonable feedback on one's flights... especially in a world where we, the pilots, are judging each others' performance... </DIV>
<DIV class=RTE> </DIV>
<DIV class=RTE>just my $.02... </DIV>
<DIV class=RTE>Rick<BR><BR>>From: Jim_Woodward@beaerospace.com<BR>>Reply-To: discussion@nsrca.org<BR>>To: discussion@nsrca.org<BR>>Subject: Performance Judging? Trial Balloon<BR>>Date: Tue, 2 Aug 2005 08:36:49 -0400<BR>><BR>>Trial Balloon Email (beware) :)<BR>><BR>>Hi All,<BR>><BR>>I'm going to propose a different way of looking at judging. Of course,<BR>>the pilot is the "performer" that is on stage for all to see, however, at<BR>>the same time the judge(s) are performing as well. At the US Nationals,<BR>>the judging committee is calculating persons judging marks, and continuing<BR>>to rank the judges. Thus, the judges are performing at the same time as<BR>>the pilot. However, we typically do not ever get to see how the<BR>>performance of the judge ranks against the pilot, or peers
in an open<BR>>forum. The current atmosphere is sort of hush quiet, or a "don't tell"<BR>>type in regards to what judge gave what score. Almost as if, its<BR>>considered good pilot etiquette not to ask judging questions. Many judges<BR>>do not like to be approached after a round to discuss scoring. Instead of<BR>>this, we could turn the tables completely 180 degree around.<BR>><BR>>Judging could be made to be a completely open from start to finish of the<BR>>contest. Perhaps within the scoring system, after each round, judges<BR>>scores for all pilots are posted (tear sheets essentially) at the same<BR>>time as the round postings. The posting of side-by-side scores, could<BR>>become a POWERFUL training tool for younger pilots (judges). Currently,<BR>>there is no
award for the "performance" of judging. Flyers go to a<BR>>contest to fly, yet the task of judging takes double or more of the<BR>>combined numbers of people than contestants (at least in total effort).<BR>><BR>>If we acknowledge that in the current local contest and Nationals setting,<BR>>there is already volumes of discussion going on between pilots regarding<BR>>scoring (is there any bigger topic with 90% of the pilots?), we can then<BR>>foster an environment which turns what is currently ambivalent or negative<BR>>judge critiquing, into one in which takes these volumes of discussion and<BR>>focuses on "constructive" or "objective" results - results beyond simply<BR>>complaining amongst each other (avoid the misery loves company syndrome).<BR>><BR>>Doing something like this will address two important issues. 1.)
The<BR>>contestant should feel more "in-tune" with scoring, and perhaps use this<BR>>feedback to better their own flying and judging. 2.) The judge should<BR>>feel satisfied in knowing at the end of a round, whether or not his/her<BR>>"calibration" is more or less correct (I'm tending to think that within a<BR>>few posted rounds of these scores, judges will "self-correct" any trends<BR>>without asking if such a trend has become obvious). Individually, I hope<BR>>each judge feels compelled to "talk" to pilots about the round if asked.<BR>><BR>>Finally, recognizing that judging is an important criteria, I recommend a<BR>>"District Award" for the top 1,2,3 persons who performed the most amount<BR>>of judging in the district (or most accurate if possible to calculate).<BR>>Also possibly a District award for the "team" that
performed the most<BR>>contest scoring.<BR>><BR>>Key thoughts:<BR>>1. Contestants are more in-tune with total system of contest<BR>>running/scoring, etc.<BR>>2. Judges (or CD's???) offered an avenue to "self-correct" if & when<BR>>necessary (... could happen to anybody)<BR>>3. District awards for the massive judging/scoring effort already taking<BR>>place.<BR>>4. Re-focus the INCREDIBLE VOLUMES of discussion already taking place<BR>>regarding scoring into a beneficial training event<BR>>5. Possibility: Upon the conclusion of Saturdays competition, the CD<BR>>could offer an open-discussion or review of the days scoring and judging -<BR>>read as, a quick pilots meeting that gives folks the open forum to discuss<BR>>anything - which can also be viewed as
mini-judge-training-event.<BR>><BR>>Anyway - just thinking of ways to take the effort that is already going<BR>>on, and refocusing into beneficial paths. Definitely open to other ideas<BR>>and suggestions.<BR>><BR>>Jim W.<BR>><BR></DIV></div></html>