共查询到20条相似文献，搜索用时 46 毫秒
1.
Will subscores provide additional information than what is provided by the total score? Is there a method that can estimate more trustworthy subscores than observed subscores? To answer the first question, this study evaluated whether the true subscore was more accurately predicted by the observed subscore or total score. To answer the second question, three subscore estimation methods (i.e., subscore estimated from the observed subscore, total score, or a combination of both the subscore and total score) were compared. Analyses were conducted using data from six licensure tests. Results indicated that reporting subscores at the examinee level may not be necessary as they did not provide much additional information over what is provided by the total score. However, at the institutional level (for institution size ≥ 30), reporting subscores may not be harmful, although they may be redundant because the subscores were predicted equally well by the observed subscores or total scores. Finally, results indicated that estimating the subscore using a combination of observed subscore and total score resulted in the highest reliability. 相似文献
2.
Mark L. Davison Ernest C. Davenport Jr. Yu‐Feng Chang Kory Vue Shiyang Su 《Journal of Educational Measurement》2015,52(3):263-279
Criterion‐related profile analysis (CPA) can be used to assess whether subscores of a test or test battery account for more criterion variance than does a single total score. Application of CPA to subscore evaluation is described, compared to alternative procedures, and illustrated using SAT data. Considerations other than validity and reliability are discussed, including broad societal goals (e.g., affirmative action), fairness, and ties in expected criterion predictions. In simulation data, CPA results were sensitive to subscore correlations, sample size, and the proportion of criterion‐related variance accounted for by the subscores. CPA can be a useful component in a thorough subscore evaluation encompassing subscore reliability, validity, distinctiveness, fairness, and broader societal goals. 相似文献
3.
Sandip Sinharay 《Journal of Educational Measurement》2014,51(2):212-222
Brennan noted that users of test scores often want (indeed, demand) that subscores be reported, along with total test scores, for diagnostic purposes. Haberman suggested a method based on classical test theory (CTT) to determine if subscores have added value over the total score. One way to interpret the method is that a subscore has added value only if it has a better agreement than the total score with the corresponding subscore on a parallel form. The focus of this article is on classification of the examinees into “pass” and “fail” (or master and nonmaster) categories based on subscores. A new CTT‐based method is suggested to assess whether classification based on a subscore is in better agreement, than classification based on the total score, with classification based on the corresponding subscore on a parallel form. The method can be considered as an assessment of the added value of subscores with respect to classification. The suggested method is applied to data from several operational tests. The added value of subscores with respect to classification is found to be very similar, except at extreme cutscores, to their added value from a value‐added analysis of Haberman. 相似文献
4.
Sandip Sinharay 《Journal of Educational Measurement》2010,47(2):150-174
Recently, there has been an increasing level of interest in subscores for their potential diagnostic value. Haberman suggested a method based on classical test theory to determine whether subscores have added value over total scores. In this article I first provide a rich collection of results regarding when subscores were found to have added value for several operational data sets. Following that I provide results from a detailed simulation study that examines what properties subscores should possess in order to have added value. The results indicate that subscores have to satisfy strict standards of reliability and correlation to have added value. A weighted average of the subscore and the total score was found to have added value more often. 相似文献
5.
A New Interpretation of Augmented Subscores and Their Added Value in Terms of Parallel Forms 
下载免费PDF全文
下载免费PDF全文 Sandip Sinharay 《Journal of Educational Measurement》2018,55(2):177-193
The value‐added method of Haberman is arguably one of the most popular methods to evaluate the quality of subscores. The method is based on the classical test theory and deems a subscore to be of added value if the subscore predicts the corresponding true subscore better than does the total score. Sinharay provided an interpretation of the added value of subscores in terms of scores and subscores on parallel forms. This article extends the results of Sinharay and considers the prediction of a subscore on a parallel form from both the subscore and the total raw score on the original form. The resulting predictor essentially becomes the augmented subscore suggested by Haberman. The proportional reduction in mean squared error of the resulting predictor is interpreted as a squared multiple correlation coefficient. The practical usefulness of the derived results is demonstrated using an operational data set. 相似文献
6.
Recently, interest in test subscore reporting for diagnosis purposes has been growing rapidly. The two simulation studies here examined factors (sample size, number of subscales, correlation between subscales, and three factors affecting subscore reliability: number of items per subscale, item parameter distribution, and data generating model) that affected the value of reporting subscores within the classical test theory framework. Results showed that a higher proportion of subscores of added value was related to lower correlation between subscales, more items per subscale, no guessing in responses, smaller variability in difficulty parameters, and matched average item difficulty and average examinee ability. 相似文献
7.
There is an increasing interest in reporting subscores, both at examinee level and at aggregate levels. However, it is important to ensure reasonable subscore performance in terms of high reliability and validity to minimize incorrect instructional and remediation decisions. This article employs a statistical measure based on classical test theory that is conceptually similar to the test reliability measure and can be used to determine when subscores have any added value over total scores. The usefulness of subscores is examined both at the level of the examinees and at the level of the institutions that the examinees belong to. The suggested approach is applied to two data sets from a basic skills test. The results provide little support in favor of reporting subscores for either examinees or institutions for the tests studied here. 相似文献
8.
Sandip Sinharay 《Educational Measurement》2013,32(4):38-42
Brennan ( 2012 ) noted that users of test scores often want (indeed, demand) that subscores be reported, along with total test scores, for diagnostic purposes. Haberman ( 2008 ) suggested a method based on classical test theory (CTT) to determine if subscores have added value over the total score. According to this method, a subscore has added value if the corresponding true subscore is predicted better by the subscore than by the total score. In this note, parallel‐forms scores are considered. It is proved that another way to interpret the method of Haberman is that a subscore has added value if it is in better agreement than the total score with the corresponding subscore on a parallel form. The suggested interpretation promises to make the method of Haberman more accessible because several practitioners find the concept of parallel forms more acceptable or easier to understand than that of a true score. Results are shown for data from two operational tests. 相似文献
9.
Recently, there has been an increasing level of interest in subscores for their potential diagnostic value. Haberman (2008b) suggested reporting an augmented subscore that is a linear combination of a subscore and the total score. Sinharay and Haberman (2008) and Sinharay (2010) showed that augmented subscores often lead to more accurate diagnostic information than subscores. In order to report augmented subscores operationally, they should be comparable across the different forms of a test. One way to achieve comparability is to equate them. We suggest several methods for equating augmented subscores. Results from several operational and simulated data sets show that the error in the equating of augmented subscores appears to be small in most practical situations. 相似文献
10.
Recent research has proposed a criterion to evaluate the reportability of subscores. This criterion is a value‐added ratio (VAR), where values greater than 1 suggest that the true subscore is better approximated by the observed subscore than by the total score. This research extends the existing literature by quantifying statistical significance and effect size for using VAR to provide practical guidelines for subscore interpretation and reporting. Findings indicate that subscores with VAR ≥ 1.1 are a minimum requirement for a meaningful contribution to a user's score interpretation; subscores with .9 < VAR < 1.1 are redundant with the total score and subscores with VAR ≤ .9 would be misleading to report. Additionally, we discuss what to do when subscores do not add value, yet must be reported, as well as when VAR ≥ 1.1 may be undesirable. 相似文献
11.
Subscores can be of diagnostic value for tests that cover multiple underlying traits. Some items require knowledge or ability that spans more than a single trait. It is thus natural for such items to be included on more than a single subscore. Subscores only have value if they are reliable enough to justify conclusions drawn from them and if they contain information about the examinee that is distinct from what is in the total test score. In this study we show, for a broad range of conditions of item overlap on subscores, that the value of the subscore is always improved through the removal of such items. 相似文献
12.
目的:探讨有精神病性症状的躁狂发作患者父母的心理健康状况及人格特征.方法:使用90项症状清单(SCL-90)和明尼苏达多相个性调查问卷(MMPI)评定有精神病性症状躁狂发作患者的父母46例,并将评定结果与中国常模比较.结果:1.患者父母SCL-90的总分、总均分、阳性项目数、阳性均分、人际关系敏感、焦虑、敌对、偏执因子分与常模比较有高度显著性差异或显著性差异(P＜0.01或P＜0.05),低于常模;患者父母的强迫症状因子分显著高于常模(P＜0.05).2.在MMPI测验中,父亲组的轻躁狂(Ma)、诈病(F)、癔病(Hy)量表分显著高于常模,而抑郁(D)、社会内向(Si)量表分则显著低于常模(P＜0.05);母亲组轻躁狂(Ma)、癔病(Hy)量表分显著高于常模(P＜0.05).结论:1.双相障碍患者父母的强迫症状评分偏高可能反映了患者父母的固有素质特征;2.双相障碍患者的父母存在与躁狂发作患者的临床表现方向一致的个性特征,但在程度上存在着明显的差异. 相似文献
13.
Howard Wainer Kathleen M. Sheehan Xiaohui Wang 《Journal of Educational Measurement》2000,37(2):113-140
In this study we describe an analytic method for aiding in the generation of subscales that characterize the deep structure of tests. In addition we also derive a procedure for estimating scores for these scales that are much more statistically stable than subscores computed solely from the items that are contained on that scale. These scores achieve their stability through augmentation with information from other related information on the test. These methods were used to complement each other on a data set obtained from a Praxis administration. We found that the deep structure of the test yielded ten subscales and that, because the test was essentially unidimensional, ten subscores could be computed, all with very high reliability. This result was contrasted with the calculation of six traditional subscales based on surface features of the items. These subscales also yielded augmented subscores of high reliability. 相似文献
14.
Subscore added value analyses assume invariance across test taking populations; however, this assumption may be untenable in practice as differential subdomain relationships may be present among subgroups. The purpose of this simulation study was to understand the conditions associated with subscore added value noninvariance when manipulating: (a) subdomain test length, (b) differences in subgroup mean ability, and (c) subgroup differences in intersubdomain correlations. Results demonstrated that subscore added value was noninvariant for 24–100% of replications (depending on subdomain test length) when the subgroup difference in intersubdomain correlation was equal to .30. To examine if this condition was met in practice, applied invariance analyses of three operational testing programs were conducted. Across these datasets, noninvariant subscore added value was present for some subdomains across sex and ethnic subgroups. Overall, these results indicate that subscore added value noninvariance is largely driven by differential intersubdomain correlations among subgroups, which may be present in some operational testing programs. 相似文献
15.
16.
Previous assessments of the reliability of test scores for testlet-composed tests have indicated that item-based estimation
methods overestimate reliability. This study was designed to address issues related to the extent to which item-based estimation
methods overestimate the reliability of test scores composed of testlets and to compare several estimation methods for different
measurement models using simulation techniques. Three types of estimation approach were conceptualized for generalizability
theory (GT) and item response theory (IRT): item score approach (ISA), testlet score approach (TSA), and item-nested-testlet
approach (INTA). The magnitudes of overestimation when applying item-based methods ranged from 0.02 to 0.06 and were related
to the degrees of dependence among within-testlet items. Reliability estimates from TSA were lower than those from INTA due
to the loss of information with IRT approaches. However, this could not be applied in GT. Specified methods in IRT produced
higher reliability estimates than those in GT using the same approach. Relatively smaller magnitudes of error in reliability
estimates were observed for ISA and for methods in IRT. Thus, it seems reasonable to use TSA as well as INTA for both GT and
IRT. However, if there is a relatively large dependence among within-testlet items, INTA should be considered for IRT due
to nonnegligible loss of information. 相似文献
17.
April L. Zenisky Ronald K. Hambleton Stephen G. Sired 《Journal of Educational Measurement》2002,39(4):291-309
Measurement specialists routinely assume examinee responses to test items are independent of one another. However, previous research has shown that many contemporary tests contain item dependencies and not accounting for these dependencies leads to misleading estimates of item, test, and ability parameters. The goals of the study were (a) to review methods for detecting local item dependence (LID), (b) to discuss the use of testlets to account for LID in context-dependent item sets, (c) to apply LID detection methods and testlet-based item calibrations to data from a large-scale, high-stakes admissions test, and (d) to evaluate the results with respect to test score reliability and examinee proficiency estimation. Item dependencies were found in the test and these were due to test speededness or context dependence (related to passage structure). Also, the results highlight that steps taken to correct for the presence of LID and obtain less biased reliability estimates may impact on the estimation of examinee proficiency. The practical effects of the presence of LID on passage-based tests are discussed, as are issues regarding how to calibrate context-dependent item sets using item response theory. 相似文献
18.
Min-Jeong Jeon Guemin Lee Jeong-Won Hwang Sang-Jin Kang 《Asia Pacific Education Review》2009,10(2):149-158
The purpose of this study was to investigate the methods of estimating the reliability of school-level scores using generalizability
theory and multilevel models. Two approaches, ‘student within schools’ and ‘students within schools and subject areas,’ were
conceptualized and implemented in this study. Four methods resulting from the combination of these two approaches with generalizability
theory and multilevel models were compared for both balanced and unbalanced data. The generalizability theory and multilevel
models for the ‘students within schools’ approach produced the same variance components and reliability estimates for the
balanced data, while failing to do so for the unbalanced data. The different results from the two models can be explained
by the fact that they administer different procedures in estimating the variance components used, in turn, to estimate reliability.
Among the estimation methods investigated in this study, the generalizability theory model with the ‘students nested within
schools crossed with subject areas’ design produced the lowest reliability estimates. Fully nested designs such as (students:schools)
or (subject areas:students:schools) would not have any significant impact on reliability estimates of school-level scores.
Both methods provide very similar reliability estimates of school-level scores. 相似文献
19.
《教育实用测度》2013,26(4):303-320
Calculator effects were examined using methods taken from research on differential item functioning. Use of a calculator was controlled on two experimental forms of a test assembled from operational items used on a standardized university mathematics placement test. Results indicated that calculator effects were not present based on analysis of test scores and in only two of the three subscores composed from homogeneous item types. Analyses of item-level functioning indicated, however, that a number of items, including several not included in the two significant subscore combinations, also contained calculator effects. For those items identified, use of the calculator appeared to have changed the actual objective being tested. The findings were generally consistent with previous research: Items that were easier when a calculator was used required either simple computations or use of a function key on the calculator; items that were more difficult required knowledge of a procedure either with or without additional computation. Analysis at the item level facilitated clearer understanding of the impact of calculator use on measurement of the underlying objective. 相似文献
20.
Confidence Intervals for Weighted Composite Scores Under the Compound Binomial Error Model 下载免费PDF全文
下载免费PDF全文 Reporting confidence intervals with test scores helps test users make important decisions about examinees by providing information about the precision of test scores. Although a variety of estimation procedures based on the binomial error model are available for computing intervals for test scores, these procedures assume that items are randomly drawn from a undifferentiated universe of items, and therefore might not be suitable for tests developed according to a table of specifications. To address this issue, four interval estimation procedures that use category subscores for the computation of confidence intervals are presented in this article. All four estimation procedures assume that subscores instead of test scores follow a binomial distribution (i.e., compound binomial error model). The relative performance of the four compound binomial–based interval estimation procedures is compared to each other and to the better known normal approximation and Wilson score procedures based on the binomial error model. 相似文献