This paper presents a study on the practical application of Adaptive Comparative Judgment (ACJ) in assessing student exams within legal education at the University of Bergen, Norway. ACJ is a method used to evaluate and compare the quality of qualitative work, involves assessors judging pairs of items relative to each other. Unlike traditional assessment methods, which rely on absolute judgments, ACJ requires assessors to assess items in pairs, determining relative quality rather than absolute quality. The study examines the feasibility of integrating ACJ into existing educational frameworks, addressing key challenges such as resource allocation, grading transparency, and assessing exams with multiple tasks. Specifically, the research focuses on assessing student exams within an undergraduate course on Property and Intellectual Property Law, with a total of 300 exam papers and eight examiners participating in the study. Utilizing RM Compare as the ACJ software, each assessor was instructed to dedicate 35 hours to comparative judgment, aiming to establish a reliable rank order of the exam papers. The findings reveal several challenges associated with the practical implementation of ACJ, including issues of resource allocation, grading transparency, and the assessment of exams with multiple tasks. Potential solutions to these challenges will be discussed.

The poster will present the outline of an ongoing findings from a study on the use of adaptive comparative judgment by examiners as a method for assessing, ranking, and grading student exams. The study aims to investigate if and how this method may support examiners in making better judgments when assessing student works for grading. In this study, approximately 300 essays will be assessed by 8 examiners using adaptive comparative judgment. These essays have already been assessed and graded using a traditional criteria-based approach with one examiner per 30 essays. These grades will serve as a point of reference for the study. The examiners using comparative judgment will use the same assessment criteria as in the traditional approach, but the essays will be assessed in pairs. In this pairwise comparison, the examiners chose which essay is better and justify their choice based on assessment criteria for each pair. Using learning analytics, the judgment of all essays will be ranked in terms of quality. The professors will then read through the ranking and try to identify the benchmarks for the lowest quality work for each grade. The benchmarking will be based on the professor’s assessment of quality considering the assessment criteria, not on a normal distribution. 

The poster will present findings on the formative use of comparative judgment and criteria-based assessment in a course on administrative law as part of a master's program in legal education. Students used the two assessment approaches to assess the quality of administrative decisions. A student population of 300 was divided into two cohorts and given six administrative decisions of varying quality to assess. One cohort assessed the decisions using a comparative judgment approach, and the other used a criteria-based assessment. In both cases, the students assessed the legal decisions focusing on a) method, b) language, and c) content and ranked them from best to worst. Before the assessment activity, the students had practiced writing administrative decisions on their own and participated in a lecture that prepared them for the assessment activities. After the assessment activity, the students discussed the quality of the decisions they had assessed in groups with participants from both cohorts. The focus of the study was to investigate whether and how the students experienced the two assessment approaches contributed to developing their understanding of quality in administrative decisions. Data was collected through a student survey and the student ranking of the legal decisions.

Technology education internationally has for some time struggled to achieve continuity between what is depicted in policy and curricular documents and the reality of day-to-day practices. With its focus often articulated through the nature of activity students are to engage with, technology teachers are recognised as having significant autonomy in the design and implementation of their practices. From this, it is important to understand teachers’ beliefs about technology education, as their conceptions of the subject will inform practice. As such, this study sought to investigate teachers’ conceptions of the purpose of teaching technology through reflection on their enacted practices. A constructivist grounded theory methodology was employed for the design of the study and analysis of data. According to our analysis, despite similarities between the nature of student activity that teachers designed and implemented, teachers represented the purpose of the subject in different ways. Three different conceptions of the purpose of teaching technology were identified; obtaining knowledge and skills for application, ability to act in a technological way, and ability to think in a technological way. Central to the three conceptions were contentions in the representations of what constituted subject matter knowledge in the subject, and the role that different application cases played in teaching technology. Without consideration and explicit articulation of the purposes for teaching technology, this lack of clarity and differences in rationale for teaching technology are likely to continue.

This chapter will describe a project that aims to provide teachers and students with necessary and efficient digital assessment approaches for the development of students’ transversal skills in STEM education. These approaches were developed, implemented, and evaluated through large-scale classroom piloting, leading to policy recommendations at national and European level for the further transformation of education. This international project included partners from Austria, Belgium, Cyprus, Finland, Ireland, Slovenia, Spain, and Sweden and was co-funded by partners and the European Commission under its Erasmus+ KA3 Policy and Experimentation initiatives. The aim of this innovative policy experimentation project was twofold. On the one hand, to “know and explain” the process of implementing the Assessment of Transversal Skills (ATS) STEM program and, on the other hand, to “understand” how it works in different contexts (schools, classrooms, countries) and thus make suggestions for improvement. This chapter will attempt to provide an overview of the commonalities and differences among the national standards in the countries involved in the ATS STEM project and compare them to the context of the Standards for Technological and Engineering Literacy: Defining the Role of Technology and Engineering in STEM Education(STEL).

This paper reports on a pilot project focused on the use of the formative assessment resource Make Design Learning Visible (MDLV) in different national settings. The MDLV resource centers on a design model involving seven interactive design skills and a formative assessment model involving five strategies. A team of researchers from seven countries in three continents, are working collaboratively with a teacher practitioner from each country to develop the structure of the research project and trial of a design activity that utilises the MDLV skills and approach to formative assessment. This paper reports on ongoing exploratory early work with the teacher practitioners trialing a short design project with an overarching theme of sustainability, a focus on developing two design skills (Empathy and Sharing ideas) and two formative assessment strategies (‘Activatinglearners as resources for one another’ and ‘Activating learners as owners of their learning’). The project centers on a design brief customised for each national setting and relevant for learners aged between 10 and 15 years of age. A structured portfolio supporting an iterative design process forms the basis of tangible evidence of learner responses. The pilot is providing insights into the effectiveness of the MDLV resource in developing design skills and formative assessment acrossnational settings. Additionally, the pilot contributes understandings of comparative participatory research involving teacher practitioners across countries. The exploratory nature of the early stages of the project intentionally allows some aspects to be customised locally by teachers as we seek to understand rather than dictate how the resource is best used in the local educational contexts. To this end, the overarching structure and pedagogy of the activity is fixed, but aspects such as lesson timings, choices around the use of MDLV tools and the further MDLV design skills are flexible.