Inter-rater reliability is used for pass-fail decisions about the examinees. Inter-rater reliability is used when the rank-order of the examinees is crucial. Inter-rater reliability is used when certifying raters. Inter-rater reliability can be deduced from the rater’s fit statistics. The lower the mean-square fit, the higher the intra-rater reliability is. This is because high intra-rater reliability implies that the ratings given by the rater can be accurately predicted from each other. There are three families of inter-rater reliability statistics: do the raters agree with each other about the examinee’s rating; do the raters agree with each other about which examinees are better and which examinees are worse; do the raters give correct ratings to the performance. There is no generally agreed index of inter-rater reliability. The choice of inter-rater reliability depends on the purpose for which the ratings are being collected, and the philosophy underlying the rating process.