A systematic review of empirical studies using log data from open-ended learning environments to measure science and engineering practices     

Karen D. Wang  Jade Maï Cock  Tanja Käser  Engin Bumbacher 《British journal of educational technology : journal of the Council for Educational Technology》2023,54(1):192-221

Technology-based, open-ended learning environments (OELEs) can capture detailed information of students' interactions as they work through a task or solve a problem embedded in the environment. This information, in the form of log data, has the potential to provide important insights about the practices adopted by students for scientific inquiry and problem solving. How to parse and analyse the log data to reveal evidence of multifaceted constructs like inquiry and problem solving holds the key to making interactive learning environments useful for assessing students' higher-order competencies. In this paper, we present a systematic review of studies that used log data generated in OELEs to describe, model and assess scientific inquiry and problem solving. We identify and analyse 70 conference proceedings and journal papers published between 2012 and 2021. Our results reveal large variations in OELE and task characteristics, approaches used to extract features from log data and interpretation models used to link features to target constructs. While the educational data mining and learning analytics communities have made progress in leveraging log data to model inquiry and problem solving, multiple barriers still exist to hamper the production of representative, reproducible and generalizable results. Based on the trends identified, we lay out a set of recommendations pertaining to key aspects of the workflow that we believe will help the field develop more systematic approaches to designing and using OELEs for studying how students engage in inquiry and problem-solving practices.

Practitioner notes

What is already known about this topic

• Research has shown that technology-based, open-ended learning environments (OELEs) that collect users' interaction data are potentially useful tools for engaging students in practice-based STEM learning.
• More work is needed to identify generalizable principles of how to design OELE tasks to support student learning and how to analyse the log data to assess student performance.

What this paper adds

• We identified multiple barriers to the production of sufficiently generalizable and robust results to inform practice, with respect to: (1) the design characteristics of the OELE-based tasks, (2) the target competencies measured, (3) the approaches and techniques used to extract features from log files and (4) the models used to link features to the competencies.
• Based on this analysis, we can provide a series of specific recommendations to inform future research and facilitate the generalizability and interpretability of results:
  • Making the data available in open-access repositories, similar to the PISA tasks, for easy access and sharing.
  • Defining target practices more precisely to better align task design with target practices and to facilitate between-study comparisons.
  • More systematic evaluation of OELE and task designs to improve the psychometric properties of OELE-based measurement tasks and analysis processes.
  • Focusing more on internal and external validation of both feature generation processes and statistical models, for example with data from different samples or by systematically varying the analysis methods.

Implications for practice and/or policy

• Using the framework of evidence-centered assessment design, we have identified relevant criteria for organizing and evaluating the diverse body of empirical studies on the topic and that policy makers and practitioners can use for their own further examinations.
• This paper identifies promising research and development areas on the measurement and assessment of higher-order constructs with process data from OELE-based tasks that government agencies and foundations can support.
• Researchers, technologists and assessment designers might find useful the insights and recommendations for how OELEs can enhance science assessment through thoughtful integration of learning theories, task design and data mining techniques.

  相似文献   

Exploring the development of executive functions in children in a digital world     

Veraksa  N. E.  Veraksa  A. N.  Bukhalenkova  D. A.  Säljö  R. 《European Journal of Psychology of Education - EJPE》2022,37(4):1035-1050

European Journal of Psychology of Education - Children in many parts of the world spend increasing time using digital devices (tablets, mobile phones etc.). In the present study, the developmental...  相似文献   

Human and artificial intelligence collaboration for socially shared regulation in learning     

Sanna Järvelä  Andy Nguyen  Allyson Hadwin 《British journal of educational technology : journal of the Council for Educational Technology》2023,54(5):1057-1076

Artificial intelligence (AI) has generated a plethora of new opportunities, potential and challenges for understanding and supporting learning. In this paper, we position human and AI collaboration for socially shared regulation (SSRL) in learning. Particularly, this paper reflects on the intersection of human and AI collaboration in SSRL research, which presents an exciting prospect for advancing our understanding and support of learning regulation. Our aim is to operationalize this human-AI collaboration by introducing a novel trigger concept and a hybrid human-AI shared regulation in learning (HASRL) model. Through empirical examples that present AI affordances for SSRL research, we demonstrate how humans and AI can synergistically work together to improve learning regulation. We argue that the integration of human and AI strengths via hybrid intelligence is critical to unlocking a new era in learning sciences research. Our proposed frameworks present an opportunity for empirical evidence and innovative designs that articulate the potential for human-AI collaboration in facilitating effective SSRL in teaching and learning.

Practitioner notes

What is already known about this topic

• For collaborative learning to succeed, socially shared regulation has been acknowledged as a key factor.
• Artificial intelligence (AI) is a powerful and potentially disruptive technology that can reveal new insights to support learning.
• It is questionable whether traditional theories of how people learn are useful in the age of AI.

What this paper adds

• Introduces a trigger concept and a hybrid Human-AI Shared Regulation in Learning (HASRL) model to offer insights into how the human-AI collaboration could occur to operationalize SSRL research.
• Demonstrates the potential use of AI to advance research and practice on socially shared regulation of learning.
• Provides clear suggestions for future human-AI collaboration in learning and teaching aiming at enhancing human learning and regulatory skills.

Implications for practice and/or policy

• Educational technology developers could utilize our proposed framework to better align technological and theoretical aspects for their design of adaptive support that can facilitate students' socially shared regulation of learning.
• Researchers and practitioners could benefit from methodological development incorporating human-AI collaboration for capturing, processing and analysing multimodal data to examine and support learning regulation.

  相似文献   

Examining socially shared regulation and shared physiological arousal events with multimodal learning analytics     

Andy Nguyen  Sanna Järvelä  Carolyn Rosé  Hanna Järvenoja  Jonna Malmberg 《British journal of educational technology : journal of the Council for Educational Technology》2023,54(1):293-312

Socially shared regulation contributes to the success of collaborative learning. However, the assessment of socially shared regulation of learning (SSRL) faces several challenges in the effort to increase the understanding of collaborative learning and support outcomes due to the unobservability of the related cognitive and emotional processes. The recent development of trace-based assessment has enabled innovative opportunities to overcome the problem. Despite the potential of a trace-based approach to study SSRL, there remains a paucity of evidence on how trace-based evidence could be captured and utilised to assess and promote SSRL. This study aims to investigate the assessment of electrodermal activities (EDA) data to understand and support SSRL in collaborative learning, hence enhancing learning outcomes. The data collection involves secondary school students (N = 94) working collaboratively in groups through five science lessons. A multimodal data set of EDA and video data were examined to assess the relationship among shared arousals and interactions for SSRL. The results of this study inform the patterns among students' physiological activities and their SSRL interactions to provide trace-based evidence for an adaptive and maladaptive pattern of collaborative learning. Furthermore, our findings provide evidence about how trace-based data could be utilised to predict learning outcomes in collaborative learning.

Practitioner notes

What is already known about this topic

• Socially shared regulation has been recognised as an essential aspect of collaborative learning success.
• It is challenging to make the processes of learning regulation ‘visible’ to better understand and support student learning, especially in dynamic collaborative settings.
• Multimodal learning analytics are showing promise for being a powerful tool to reveal new insights into the temporal and sequential aspects of regulation in collaborative learning.

What this paper adds

• Utilising multimodal big data analytics to reveal the regulatory patterns of shared physiological arousal events (SPAEs) and regulatory activities in collaborative learning.
• Providing evidence of using multimodal data including physiological signals to indicate trigger events in socially shared regulation.
• Examining the differences of regulatory patterns between successful and less successful collaborative learning sessions.
• Demonstrating the potential use of artificial intelligence (AI) techniques to predict collaborative learning success by examining regulatory patterns.

Implications for practice and/or policy

• Our findings offer insights into how students regulate their learning during collaborative learning, which can be used to design adaptive supports that can foster students' learning regulation.
• This study could encourage researchers and practitioners to consider the methodological development incorporating advanced techniques such as AI machine learning for capturing, processing and analysing multimodal data to examine and support learning regulation.

  相似文献