Scoring Errors

Updated 15 hours ago by Shayna Pittman

We generally have ~3 types of errors in scoring:

Rubric errors- The scoring team correctly followed the rubric, but the rubric is wrong (or incomplete/vague/etc). These errors decrease as we get more repetitions through a scenario and more feedback from customers about cases we miss. We've had ~100 candidates through this scenario. We typically see stabilization around 500
Scoring errors- These are more rare and happen when each human scorer errs in the same direction on a given rubric item. When we've done audits, we see error rates of 2.5-3.5% so on a 24-rubric item scenario like this we'd expect 0-3 errors
Communication errors- The rubric and scoring are correct, but the way we communicate about them in the analysis is confusing, incomplete, or just wrong. These act like rubric errors where we have more of them when scenarios are new and they are slowly reduced with feedback/time