Stata Help

Kappa: Multiple Ratings and Multiple Raters

When you have multiple raters and ratings, there are two subcases. In the first case, there is a constant number of raters across cases; this is a preferable situation to the second case, in which you have a nonconstant number of raters. In the second instance, Stata can calculate kappa for each category but cannot calculate an overall kappa.

For the example below, three raters rated the moods of participants, assigning them to one of five categories. Since the data is organized by rater, I will use kap. Thus the command is kap raterA raterB raterC and it yields the following output:

If the data were in frequency format, the command would be kappa overjoyed happy neutral sad depressed and the output would not change.

In the unfortunate second case alluded to above, the output only contains a kappa rating of each category against all other possible values. Again, it is preferable to have a constant number of raters.