The Undergraduate Research Student Self-Assessment (URSSA): Validation for Use in Program Evaluation     

Timothy J. Weston  Sandra L. Laursen 《CBE life sciences education》2015,14(3)

This article examines the validity of the Undergraduate Research Student Self-Assessment (URSSA), a survey used to evaluate undergraduate research (UR) programs. The underlying structure of the survey was assessed with confirmatory factor analysis; also examined were correlations between different average scores, score reliability, and matches between numerical and textual item responses. The study found that four components of the survey represent separate but related constructs for cognitive skills and affective learning gains derived from the UR experience. Average scores from item blocks formed reliable but moderate to highly correlated composite measures. Additionally, some questions about student learning gains (meant to assess individual learning) correlated to ratings of satisfaction with external aspects of the research experience. The pattern of correlation among individual items suggests that items asking students to rate external aspects of their environment were more like satisfaction ratings than items that directly ask about student skills attainment. Finally, survey items asking about student aspirations to attend graduate school in science reflected inflated estimates of the proportions of students who had actually decided on graduate education after their UR experiences. Recommendations for revisions to the survey include clarified item wording and increasing discrimination between item blocks through reorganization.Undergraduate research (UR) experiences have long been an important component of science education at universities and colleges but have received greater attention in recent years, as they have been identified as important ways to strengthen preparation for advanced study and work in the science fields, especially among students from underrepresented minority groups (Tsui, 2007 ; Kuh, 2008 ). UR internships provide students with the opportunity to conduct authentic research in laboratories with scientist mentors, as students help design projects, gather and analyze data, and write up and present findings (Laursen et al., 2010 ). The promised benefits of UR experiences include both increased skills and greater familiarity with how science is practiced (Russell et al., 2007 ). While students learn the basics of scientific methods and laboratory skills, they are also exposed to the culture and norms of science (Carlone and Johnson, 2007 ; Hunter et al., 2007 ; Lopatto, 2010 ). Students learn about the day-to-day world of practicing science and are introduced to how scientists design studies, collect and analyze data, and communicate their research. After participating in UR, students may make more informed decisions about their future, and some may be more likely to decide to pursue graduate education in science, technology, engineering, and mathematics (STEM) disciplines (Bauer and Bennett, 2003 ; Russell et al., 2007 ; Eagan et al. 2013 ).While UR experiences potentially have many benefits for undergraduate students, assessing these benefits is challenging (Laursen, 2015 ). Large-scale research-based evaluation of the effects of UR is limited by a range of methodological problems (Eagan et al., 2013 ). True experimental studies are almost impossible to implement, since random assignment of students into UR programs is both logistically and ethically impractical, while many simple comparisons between UR and non-UR groups of students suffer from noncomparable groups and limited generalizability (Maton and Hrabowski, 2004 ). Survey studies often rely on poorly developed measures and use nonrepresentative samples, and large-scale survey research usually requires complex statistical models to control for student self-selection into UR programs (Eagan et al., 2013 ). For smaller-scale program evaluation, evaluators also encounter a number of measurement problems. Because of the wide range of disciplines, research topics, and methods, common standardized tests assessing laboratory skills and understandings across these disciplines are difficult to find. While faculty at individual sites may directly assess products, presentations, and behavior using authentic assessments such as portfolios, rubrics, and performance assessments, these assessments can be time-consuming and not easily comparable with similar efforts at other laboratories (Stokking et al., 2004 ; Kuh et al., 2014 ). Additionally, the affective outcomes of UR are not readily tapped by direct academic assessment, as many of the benefits found for students in UR, such as motivation, enculturation, and self-efficacy, are not measured by tests or other assessments (Carlone and Johnson, 2007 ). Other instruments for assessing UR outcomes, such as Lopatto’s SURE (Lopatto, 2010 ), focus on these affective outcomes rather than direct assessments of skills and cognitive gains.The size of most UR programs also makes assessment difficult. Research Experiences for Undergraduates (REUs), one mechanism by which UR programs may be organized within an institution, are funded by the National Science Foundation (NSF), but unlike many other educational programs at NSF (e.g., TUES) that require fully funded evaluations with multiple sources of evidence (Frechtling, 2010 ), REUs are generally so small that they cannot typically support this type of evaluation unless multiple programs pool their resources to provide adequate assessment. Informal UR experiences, offered to students by individual faculty within their own laboratories, are often more common but are typically not coordinated across departments or institutions or accountable to a central office or agency for assessment. Partly toward this end, the Undergraduate Research Student Self-Assessment (URSSA) was developed as a common assessment instrument that can be compared across multiple UR sites within or across institutions. It is meant to be used as one source of assessment information about UR sites and their students.The current research examines the validity of the URSSA in the context of its use as a self-report survey for UR programs and laboratories. Because the survey has been taken by more than 3400 students, we can test some aspects of how the survey is structured and how it functions. Assessing the validity of the URSSA for its intended use is a process of testing hypotheses about how well the survey represents its intended content. This ongoing process (Messick, 1993 ; Kane, 2001 ) involves gathering evidence from a range of sources to learn whether validity claims are supported by evidence and whether the survey results can be used confidently in specific contexts. For the URSSA, our method of inquiry focuses on how the survey is used to assess consortia of REU sites. In this context, survey results are used for quality assurance and comparisons of average ratings over years and as general indicators of program success in encouraging students to pursue graduate science education and scientific careers. Our research questions focus on the meaning and reliability of “core indicators” used to track self-reported learning gains in four areas and the ability of numerical items to capture student aspirations for future plans to attend graduate school in the sciences.  相似文献   

Breaking the Cycle: Future Faculty Begin Teaching with Learner-Centered Strategies after Professional Development     

Diane Ebert-May  Terry L. Derting  Timothy P. Henkel  Jessica Middlemis Maher  Jennifer L. Momsen  Bryan Arnold  Heather A. Passmore 《CBE life sciences education》2015,14(2)

The availability of reliable evidence for teaching practices after professional development is limited across science, technology, engineering, and mathematics disciplines, making the identification of professional development “best practices” and effective models for change difficult. We aimed to determine the extent to which postdoctoral fellows (i.e., future biology faculty) believed in and implemented evidence-based pedagogies after completion of a 2-yr professional development program, Faculty Institutes for Reforming Science Teaching (FIRST IV). Postdocs (PDs) attended a 2-yr training program during which they completed self-report assessments of their beliefs about teaching and gains in pedagogical knowledge and experience, and they provided copies of class assessments and video recordings of their teaching. The PDs reported greater use of learner-centered compared with teacher-centered strategies. These data were consistent with the results of expert reviews of teaching videos. The majority of PDs (86%) received video ratings that documented active engagement of students and implementation of learner-centered classrooms. Despite practice of higher-level cognition in class sessions, the items used by the PDs on their assessments of learning focused on lower-level cognitive skills. We attributed the high success of the FIRST IV program to our focus on inexperienced teachers, an iterative process of teaching practice and reflection, and development of and teaching a full course.  相似文献   

How Custom Agents Really Detect Lies     

Timothy R. Levine  Yasuhiro Daiku 《Communication Research Reports》2019,36(1):84-92

Research has found discrepancies among folk beliefs about lie detection, deception cue utility, and retrospective accounts of detected lies. Elite customs agents (N = 37) were surveyed regarding their overall beliefs about how to detect lies (general strategy), their opinions about best practices (best strategy), and about a successfully detected lie (actual strategy). Responses were coded in three nonexclusive broad categories: (a) behavioral cues, (b) communication content and evidence, and (c) interactive-questioning approaches. Behavioral cues, especially nonverbal cues, were listed most frequently as general beliefs. Opinions about the best approach were evenly split across the three categories. Communication content and evidence was the most frequently reported actual detection method and was mentioned in 97% of the responses. The results are discussed in relation to Truth-Default Theory and their implications for lie detection practice.  相似文献   

INTERNAL FACTOR STRUCTURE AND CONVERGENT VALIDITY EVIDENCE: THE SELF‐REPORT VERSION OF SELF‐REGULATION STRATEGY INVENTORY     

Timothy J. Cleary  Leah Dembitzer  Ryan J. Kettler 《Psychology in the schools》2015,52(9):829-844

Using a sample of 348 middle school students, we gathered evidence regarding the internal consistency of scores, as well as the internal factor structure and convergent validity evidence for inferences from a self‐report questionnaire called the Self‐Regulation Strategy Inventory–Self Report. Confirmatory factor analysis revealed that the fit indexes for a hierarchical model (composite, three factors) and a single‐level, three‐factor model were highly similar but mixed. Respecification of the hierarchical model based on conceptual overlap of items led to substantial improvement in the overall fit of the model, as indicated by the root mean square error of approximation, chi‐square/df, and the comparative fit index. Correlational analyses also provided strong convergent validity evidence, as the three subscales exhibited statistically significant relations with four motivation beliefs (i.e., self‐efficacy, perceived instrumentality, task interest, perceived responsibility) and two distinct markers of regulation‐related behaviors (i.e., teacher ratings, office discipline referrals).  相似文献   

Do more hours in center-based care cause more externalizing problems? A cross-national replication study     

Catalina Rey-Guerra  Henrik D. Zachrisson  Eric Dearing  Daniel Berry  Susanne Kuger  Margaret R. Burchinal  Ane Nærde  Thomas van Huizen  Sylvana M. Côté 《Child development》2023,94(2):458-477

Whether high quantities of center-based care cause behavior problems is a controversial question. Studies using covariate adjustment for selection factors have detected relations between center care and behavior problems, but studies with stronger internal validity less often find such evidence. We examined whether within-child changes in hours in center-based care predicted changes in externalizing problems in toddlers and preschoolers (N = 10,105; 49% female; data collection 1993–2012) in seven studies, including from Germany, Netherlands, Norway, two from Canada and two from the U.S. Race/ethnicity data were only collected in the United States (57% and 80% White; 42% and 13% African-American; 1.2% and 5% Latinx). Meta-analyses showed no association (r = .00, p = .88) between hours in center-based care and externalizing problems.  相似文献   

Social Perspectives on the `Hard' Problem of Consciousness     

Timothy Koschmann 《学习科学杂志》2013,22(3-4):521-523

  相似文献   

Review and criticism     

M. Timothy O'Keefe  Kenneth G. Sheinkopf 《广播与电子媒介杂志》2013,57(4):403-412

An earlier version of this paper was presented to the AEJ convention at Boulder, Colorado in 1973 by Dr. Milan Meeske for the authors. Dr. O'Keefe is an associate professor of communication at Florida Technological University where Mr. Sheinkopf is a student.  相似文献   

Establishing the structure of reality for an industry: model and anti-model arguments as advocacy in Nike's crisis communication     

Timothy Sellnow  Jeffrey Brand 《Journal of Applied Communication Research》2001,29(3):278-295

This study illustrates the potential role of model and anti-model arguments in organizational crisis communication. Specifically, model and anti-model arguments are described as a strategy for moving the focus of a crisis from an organization to its industry. Model arguments enable organizations to establish their corrective action as industry standards that merit imitation. Conversely, organizations can set minimum standards for their industries with anti-model arguments. Phil Knight's May 12, 1998, speech announcing Nike's new initiatives in global manufacturing is analyzed as a case study. The essay concludes that model and anti-model arguments can suspend criticism of the organization, create the foundation for a return to industry prominence by the organization, and establish proposed new industry standards that are favorable to the organization.  相似文献   

Teacher e-mail message strategies and students' willingness to communicate online     

Jennifer Waldeck  Patricia Kearney  Timothy Plax 《Journal of Applied Communication Research》2001,29(1):54-70

In light of the prevalent use of e-mail between teachers and students, researchers have highlighted the need to assess its uses and effects. Relying on the research and thinking of teacher immediacy and extra-class communication, we developed a scale that measures those e-mail message strategies that influence students' willingness to communicate online with their teachers. Next, we isolated those reasons that students use e-mail to interact with their teachers: (1) to clarify course material and procedures, (2) as a means of efficient communication, and (3) for personal/social reasons. Finally, we examined student characteristics such as gender and ethnicity, and history of e-mail use that may (or may not) influence their attitudes toward e-mail exchanges with teachers.  相似文献   

Introduction     

Timothy Koschmann 《Distance Education》2002,23(1):5-9

  相似文献