interrater reliability in qualitative research

On the other hand, an inter-rater reliability of 95% may be required in medical settings in which multiple doctors are judging whether or not a certain treatment should be used on a given patient. However, higher inter-rater reliabilities may be needed in specific fields.įor example, an inter-rater reliability of 75% may be acceptable for a test that seeks to determine how well a TV show will be received. In general, an inter-rater agreement of at least 75% is required in most fields for a test to be considered reliable. The higher the inter-rater reliability, the more consistently multiple judges rate items or questions on a test with similar scores. p e: Hypothetical probability of chance agreementĬohen’s Kappa always ranges between 0 and 1, with 0 indicating no agreement between raters and 1 indicating perfect agreement between raters.įor a step-by-step example of how to calculate Cohen’s Kappa, refer to this tutorial.p o: Relative observed agreement among raters.The formula for Cohen’s kappa is calculated as:

The more difficult (and more rigorous) way to measure inter-rater reliability is to use use Cohen’s Kappa, which calculates the percentage of items that the raters agree on, while accounting for the fact that the raters may happen to agree on some items purely by chance. The percentage of questions the judges agreed on was 7/10 = 70%. The results are shown below:įor each question, we can write “1” if the two judges agree and “0” if they don’t agree. This is known as percent agreement, which always ranges between 0 and 1 with 0 indicating no agreement between raters and 1 indicating perfect agreement between raters.įor example, suppose two judges are asked to rate the difficulty of 10 items on a test from a scale of 1 to 3. The simple way to measure inter-rater reliability is to calculate the percentage of items that the judges agree on. There are two common ways to measure inter-rater reliability: If a test has lower inter-rater reliability, this could be an indication that the items on the test are confusing, unclear, or even unnecessary. It is used as a way to assess the reliability of answers produced by different items on a test. In statistics, inter-rater reliability is a way to measure the level of agreement between multiple raters or judges.

Interrater reliability in qualitative research