| 科技项目专家评审的元评价综合模型研究 |
| |
| 引用本文： | 汪建,王裴裴,丁俊.科技项目专家评审的元评价综合模型研究[J].科研管理,2006,41(2):183-192. |
| |
| 作者姓名： | 汪建 王裴裴 丁俊 |
| |
| 作者单位： | 1.上海大学管理学院，上海200444；
2.上海市经济和信息化委员会信息中心，上海200125 |
| |
| 摘 要： | 科技项目的评选越来越多采用专家评价的方式，评审专家的打分质量直接影响科技项目的立项结果。为了提升科技项目的评价效果，管理部门通常建立评审专家库，并动态调整。本研究从第三方的角度，借鉴元评价的概念，基于评审专家的匿名打分结果对评审专家进行评价以支持专家库的动态调整。专家评审质量从三个方面进行定义，包括单项得分，最终得分，排序得分等角度；具体选取偏差系数、秩相关系数和二值法系数等评价指标。基于某政府科技部门2015年的1634个项目，分析了99位评审专家的实际评价数据。基于实证分析结果引入变异系数、参与度和立项率等调整系数建立了元评价综合模型。由管理部门结合评价过程和专家的实际身份特征等信息对综合评价模型的得分进行了验证分析，结果表明本研究提出的综合模型的可靠性和实践性满足项目管理部门的需求。最后，基于该模型，对科技项目管理部门如何提升专家评审质量给出了具体建议。
|
| 收稿时间： | 2017-07-17 |
A comprehensive model on the meta-evaluation of science and technology projects by review experts |
| |
| Wang Jian,Wang Peipei,Ding Jun.A comprehensive model on the meta-evaluation of science and technology projects by review experts[J].Science Research Management,2006,41(2):183-192. |
| |
| Authors: | Wang Jian Wang Peipei Ding Jun |
| |
| Abstract: | Founding science and technology projects is an important approach to support innovations. Most science and technology projects are selected through the evaluation of review experts. How to assess the review experts is necessary and important to select good projects. The evaluation of review experts are also called as meta-evaluation. After reviewing the theories and methods on meta-evaluation, it was found there are some limitations when referring to the existing literatures in estimating review experts in science and technology projects, i.e., some evaluation criteria used in meta-evaluation can estimate experts from different perspectives and may be used together; some criteria may be duplicated and should not be used together; there are some influential factors on evaluation criteria. Therefore, a comprehensive meta-evaluation model was proposed for the meta-evaluation of science and technology projects by review experts.
The comprehensive model regarded that three kinds of information may be used to estimate one review expert in meta-estimation, i.e., 1) the deviation degree between the expert′s scores on each review criterion with the scores given by all experts; 2) the deviation degree between the rank of all projects given by the expert and the rank given by all experts; 3) the deviation degree between the pass results suggested by the expert and the actual results. Accordingly, the comprehensive model adopted three evaluations criteria based on literature review to calculate the deviation degrees, i.e., 1) deviation coefficients on each review criterion which can be used for estimate the first deviation degree; 2) Spearman rank correlation coefficient on all projects which can be used for estimating the second deviation degree, and 3) binary value reliability coefficients on each project which can be used for estimating the third deviation degree. These criteria include all types of data used in the researches on meta-evaluation, cover two research perspectives of meta-evaluation, and include the main meta-evaluation criteria. In addition, three influential factors were identified based on literature review and discussions, i.e., 1) the variation coefficients of each review criterion among all experts which can be measured with the standard deviation of the experts′ score given on each estimation criterion; 2) the attendance degree of one expert in project review work which can be measured with the ratio of the project quantities that an expert participated in and the maximum quantity of projects that all experts participated in; 3) the actual pass ratio of projects that can be measured with the ratio of the final supported project quantities to the total applying project quantities. In order to analysis the relationships among these criteria and other influential factors, an empirical study was adopted. The empirical data was provided by one government department including the review data on 1634 projects by over 1000 experts. 99 experts with the most attendance and all of their review data were picked up for this study.
Some suggestions can be given based on the statistical analysis of the empirical data. 1) The correlation relationships among three estimation criteria are not significant (p=001). Though the relationships between Spearman rank correlation coefficients and binary value reliability coefficients are significant on p=0.02, their relationship coefficient is low at 0.24. The data suggests that three selected estimation criteria can make estimation through different perspectives and these criteria can be used together to estimate review experts. 2) There is a significant correlation relationship between the deviation coefficients of one expert and the variation coefficients of all experts on review criteria. The correlation relationship coefficient is 0.901 (p=0.00). The close relationships suggest that the variation coefficient has a significant impact on the deviation coefficients. It is not suitable to estimate one expert only according to his/her performance on the deviation coefficient on each review criterion. Therefore, a formula was founded to adjust the deviation coefficients based on the variation coefficients. 3) The experts′ attendance degrees have significant impact on the estimation performance (p=0.01). Therefore, an adjustment model was founded to revise Spearman rank correlation coefficient with experts′ attendance degrees. 4) There are significant correlation relationships between binary value reliability coefficients and the pass ratios of projects. The higher (lower) the pass rates of science and technology projects is, the more (less) the experts recommend the projects. Accordingly, an adjustment model was built to revise the binary value reliability coefficients with pass ratios of projects.
At last, in order to integrate these three criteria in the comprehensive meta-estimation model, their weights were decided based on an empirical survey among 8 specialists from industries and schools. The survey data suggested different weight for each criterion, i.e., the deviation coefficients on each review criterion (0.2), Spearman rank correlation coefficient on all projects (0.4), and binary value reliability coefficients on each project (0.4). Based on three estimation criteria and three influential factors, a comprehensive meta-estimation model was built. In order to illustrate the reliability of the model, an actual case was adopted for testing. In this case, there are seven review experts participating in reviewing 15 projects. The results of the comprehensive model were compared with the traditional meta-estimation methods. The comparison results show that the results from the comprehensive model can give out more practical results.
Finally, some suggestions are provided on how to improve the expert review quality for science and technology project management departments. 1) It is not enough to adopt few estimation criteria during meta-estimation of review experts. A comprehensive meta-estimation model is necessary to integrate different criteria and influential factors. 2) It is necessary to dynamically manage review experts, and the attendance degree of review experts is also important for review quality improvement. Therefore, it is necessary to improve the stability of review experts. 3) It is necessary to strengthen the communication between project management departments and review experts on the pass rate of projects and other relevant information. 4) It is helpful to combine quantitative and qualitative estimation criteria so as to reduce the deviations of review data given by review experts.
In this study, a single review expert was estimated based on their deviation with the total experts. This may be not good in some cases, such as when reviewing the projects with originality. Some standard projects may be suggested for testing review experts. In addition, this study did meta-estimation only based on the quantitative data, an integrated meta-estimation model based on quantitative and qualitative data may be proposed in the future study. |
| |
| Keywords: | |
|
| 点击此处可从《科研管理》浏览原始摘要信息 |
| 点击此处可从《科研管理》下载免费的PDF全文 |
|
