University of Reading
By testing and evaluating a range of available methodologies, this project will draw conclusions on what might be the right combination of instruments for the measurement of learning gain. These measure include degree classification outcomes, UKES, NSS, Student Wellbeing Survey and CLA+.
Project methodologies: Grades; Surveys; Standardised test; Mixed methods
Learning gain: Can attainment data inform pedagogy?
Our project evaluates a range of methodologies (attainment data, the UKES, the Student Wellbeing Survey, CLA+ and student self-assessment of learning [perceived learning gain]) for the measurement of learning gain. This case study looks at attainment data modelling with an objective of measuring and comparing learning gain in different programmes and cohorts.
Aims and objectives
- This aspect of the mixed model project aimed to model student and programme trajectories with an objective of measuring and comparing learning gain in different programmes and cohorts.
- We fit four-level growth models. This approach enabled us to capture not only the global achievement and improvement of learning gain in the university, but also the relative differences in the intercepts and rates of change of students and programmes.
- Residual analyses were conducted to compare between- and within-programme differences within the same cohort, as well as between-cohort differences.
Directors of Teaching and Learning were given programme trajectories from 2008-2016 and asked the following questions:
- What should learning gain look like in terms of these trajectories?
- What changes in the student- and programme-level achievement trajectories have taken place in the University of Reading?
- What University policies or School approaches to assessment might have enabled or constrained the achievement at the level of specific programmes?
- Can anything identified above actually be associated with the trajectory results?
- What do the data tell you about assessment and marking of your programme?
- Can you identify issues in your programmes that are of concern?
- If so, what will you do about it?
Experiences and outcomes
From a student perspective:
We have not yet reached the stage of applying our findings in ways that will impact positively on student learning. However, plans are underway to use data and involve students in curriculum design.
From an institutional perspective:
This aspect of the learning gain project has positively informed and influenced institutional learning and teaching strategies. Learning gain has become one of the strategic projects in teaching and learning (T&L) at the University of Reading. Furthermore the attainment data has been incorporated into T&L dashboards to inform future strategies for each School and staff in planning and strategy have been trained to use the model.
The statistical model visualises the trajectories of students between their first and final years, as well as the average trajectories of the courses they were in. We now have a manual that explains, step-by-step, the required procedure to extract the sample being studied from the university database and therefore allows other researchers at the university to replicate the findings. This approach can easily be replicated with SITS data in other universities.
Internal dissemination of attainment data has been via seminars, a webpage and T&L workshops. The aim of the workshops were to ask staff to reflect on the expectations of attainment in relation to learning gain and to consider whether their current programmes and assessments achieve this. The modelling of student progression presented allows staff and senior managers to compare attainment across programmes and between student groups within programmes. This has already highlighted where some programmes need to reconsider their assessments and marking criteria. Even if no changes are necessary, the modelled data presented encourages reflection and internal discussion on how learning gain can be measured and the value of such information.
Some Schools have run focus groups to interpret the modelled attainment data and draw conclusions about possible links between the attainment data and pedagogical practices. In some cases, Schools have questioned their marking practices.
Elsewhere, the learning gain workshop has triggered a School to probe individual module attainments to explain the modelled patterns in various degrees (i.e. where the slope is negative over time).
Impact of the project
Staff in direct contact with students:
“Although not directly relevant to my current role, it was an insight into how people learn and what you should be thinking about when you are trying to plan teaching activities for any audience, not just students.”
“In response to the attainment analysis data, we are planning to introduce an additional module to prepare students for independent research.”
“We plan to monitor data for individual modules more closely.”
“The learning gain workshop on the interpretation of the attainment model data has prompted my rethinking the link between the curriculum and assessment.”
A major issue was related to use of the SITS data. In order to apply the model, data had to be cleaned to remove students who did not complete their degree in three years.
In addition there was difficulty in analysing and comparing specific groups of students due to changes affecting the categorisation of programmes within JACS and UCAS codes over that period. Some variables had a large proportion of missing data, although the risk of inferential bias was partly mitigated by using advanced data imputation techniques (chained equations).
Data and analytical methods
The statistical model currently draws from data from about 20,000 students who completed their HE studies between 2008 and 2016. The model visualises the trajectories of all these students between their first and final years, as well as the average trajectories of the courses they were in.
Each of these cohorts included a total of 1,700–2,000 undergraduate full-time students who graduated on time from a three-year course. This sample represented about 75% of all undergraduate full-time students graduating on time in a given year and just over half of the total population of graduates. Both UK and overseas students were included in the analysis.
To model student and programme trajectories we fit four-level growth models. Our modelling approach enabled us to capture not only the global achievement and improvement of the university, but also the relative differences in the intercepts and rates of change of students and programmes. Details of the methodology will be available online within the next couple of months.
Publications and forums
Where has the work been publicised?
- Institutional website
- Institutional TEF document
- At external conferences i.e. HEA Conference, SHRE meeting. Download learning gain presentation slides
Contact: Amanda Callaghan: email email@example.com, tel 0118 378 4428
Case study authors: Amanda Callaghan, Nina Brooke and Cesare Aloisi