Increasing the Validity of Angoff Standards Through Analysis of Judge-Level Internal Consistency |
| |
| Authors: | Jerome C Clauser Brian E Clauser Ronald K Hambleton |
| |
| Institution: | 1. American Board of Internal Medicine JClauser@ABIM.org;3. National Board of Medical Examiners;4. University of Massachusetts Amherst |
| |
| Abstract: | The purpose of the present study was to extend past work with the Angoff method for setting standards by examining judgments at the judge level rather than the panel level. The focus was on investigating the relationship between observed Angoff standard setting judgments and empirical conditional probabilities. This relationship has been used as a measure of internal consistency by previous researchers. Results indicated that judges varied in the degree to which they were able to produce internally consistent ratings; some judges produced ratings that were highly correlated with empirical conditional probabilities and other judges’ ratings had essentially no correlation with the conditional probabilities. The results also showed that weighting procedures applied to individual judgments both increased panel-level internal consistency and produced convergence across panels. |
| |
| Keywords: | |
|
