The Open University
This project will carry out secondary data analysis of existing Affective-Behaviour-Cognition (ABC) model data for previous academic years. It will then carry out a mixed-method study of new ABC data for the academic years 2015 to 2017, including interviews with participants as well as work on learner analytics.
Partners: Oxford Brookes University; University of Surrey
Project methodologies: Grades; Surveys; Mixed methods; Other qualitative methods
Using students’ academic performance to estimate their learning gains
Aims and objectives
The main aim of the project is to test whether (or not) we can establish and evaluate:
- effective predictors for affective, behavioural and cognitive learning gains using principles of learning analytics
- contrast the relative merits and drawbacks of affective, behavioural and cognitive approaches to measure learning gains
- test the validity and generalisability of affective, behavioural and cognitive learning gains proxies across Open University, Oxford Brooks University and University of Surrey.
Our longitudinal analyses across 100 modules and 40 degree qualifications indicate substantial variation in learning gains that are found between students, modules, and qualification.
Similarly, substantial variation is present when comparing results across other universities, highlighting a need for advanced statistical modelling to unpack complex learning gains across institutions.
Experiences and outcomes
There are three main pillars in learning gains research:
- what learning gains to measure
- how to measure them
- how to compute learning gains.
Whether students are judged to have made learning gains or not depends on whether learning gains are interpreted within the context of discipline, course and university or within the student entity as a whole.
Interpretation outside the context can give advantages to some disciplines and not others, and even to some modules and subjects. Analysis of over 100 modules in science and social science disciplines at the Open University revealed that when comparing overall progress across all modules there is a slight upward trajectory in both disciplines. This would lead to the conclusion that there is a small positive learning gain in general and it does not matter which discipline students choose to study.
However, looking at every module within each discipline separately leads to a somewhat different conclusion. In both social science and science disciplines, there are individual modules with upward trajectories, and others where trajectories are downwards. Some groups of students show higher learning gains on some modules than on others. Thus, module level data suggests that there are clear advantages in studying particular modules where a majority of students obtain higher learning gains.
Interestingly, social science students with initially low achievements tend to progress at a similar rate to students with high initial achievements, whereas in science students who were initially low achievers are generally able to obtain higher learning gains over time. This suggests that studying science is more advantageous for weaker students and studying social science is more advantageous for stronger. However, it is twice as hard to predict social science students’ grades as those of science students, although, overall, science disciplines still seem preferable in terms of learning gain to social science subjects, in particular for weaker students.
So, aggregating data at different levels can lead to different interpretations. To make assessment of learning gains a more even playing-field it is important to look at the context. In both science and social science disciplines there are students who show positive and negative learning gains, but each module also has its own trajectory.
In order to effectively measure appropriate measurements of learning gains, our project indicates that it is paramount to look at the trajectories of students in relation to the trajectories of modules, courses, and disciplines. Students’ attainment can go up or down, but if it goes up or down in the context of a module, discipline, and university, that matters. As such, if we fail to consider the context of subject or module choice, it is likely to put some disciplines at an unfair disadvantage.
From the student perspective:
Although so far this project concentrated on analysing students’ cognitive learning gains quantitatively using big data and learning analytics approaches, the preliminary findings from qualitative interviews with more than 40 high–medium–low learning gains students indicate that students also make effective and behavioural learning gains. Below are examples of quotes from students when asked, what have they learned while studying at OU?
From the institutional perspective:
The Open University UK is a distance-learning institution with an open-entry policy, and is the largest university in the UK. Given that, the OU is open to all people and no formal qualification requirements are present at level 1 modules. This means that anyone can enrol onto a degree programme subject to a successful completion of 1st year modules. As such, OU is faced with unique challenges of helping students to achieve their academic potential. Analysis of students’ learning trajectories across modules and also across degree qualifications enabled us to see how current practice impacts students’ learning gains and identify areas where OU can improve.
The learning gains project findings have been included into the Scholarly Insight Report (Rienties et al., 2017) that was disseminated widely across the OU in May 2017. There is strong interest from across faculties in this project as well as from the managerial level.
The impact of the project
From the student perspective:
On effective learning gains:
“I feel more confident and I am happier because I am doing something I have always wanted to be doing and something that interests me.”
“Now I say, ‘you know what, I can do that in future.'”
Behavioural learning gains:
“I am much better at time management, I am much more organised now and planning things in advance.”
“[In my new job], there will reports and planning to be drawn and I think that this will be an aspect of my job where I can say yes the OU study and discipline I’ve received from the OU has actually contributed to that.”
On cognitive learning gains:
“Academic appreciation, the training of my academic side of my mind has been the thing I think I’ve been able to improve.”
“I think I am more openly critical, I mean in the positive sense.”
From the staff perspective:
Staff have indicated that the Scholarly Insight report and pathways of students were very useful. We are still collecting data about impact on staff at this moment.
From senior managers' perspective:
”[…] This is very helpful. This needs to be part of the building block for the new teaching model.”
From researchers' perspective:
In general there is good uptake and interest from researchers on our learning gains project. See @learninggains.
The quantitative component of Academic Behaviour Confidence (ABC) learning gains project requires universities to share large quantity of data including socio-demographic and administrative data as well as virtual learning environment data. Data sharing across institutions has been the biggest challenge. These involved storage of the data, anonymization of the data and merging multiple datasets.
At this stage of our project we primarily use administrative data and socio-demographic data to estimate student’s learning gains.
The biggest challenge within learning gains research that is often overlooked is identification of an adequate computational method for estimating learning gains. Review of the learning gains research identified a number of techniques that can be used, like computation of true gain, residual gain, normalised gain, average normalised gain, normalised change, ANOVA and ANCOVA on residuals or pre-post test scores. Although these alternatives address the issue of measurement error, all of these methods assume that errors between participants are uncorrelated and, as such, assume that pre-test and post-test observations from one participant are independent from pre-post test observations of another participant.
This assumption is not true, as students from the same discipline, same class, and/or same university have shared variance due to the similarity of experiences, and this variance is usually overlooked, which leads to the incorrect results. To address this limitation and to estimate learning gains of students on the institutional level, our project use multilevel growth-curve modelling and estimated individual learning trajectories, within module learning trajectories and qualification trajectories.
Publications and forums
Where has the work been publicised?
Academic publications since October 2016:
- Rienties, B., Rogaten, J., Nguyen, Q., Edwards, C., Gaved, M., Holt, D., Ullmann, T. (2017). Scholarly insight 2017: a Data wrangler perspective. Milton Keynes: Open University UK.
- Rogaten, J., Rienties, B., Cross, S.J., Whitelock, D., Sharpe, R., Lygo-Baker, S., & Littlejohn, A. (under review). Reviewing affective, behavioural and cognitive learning gains in higher education. Review of Educational Research. Impact factor: 5.235.
- Rogaten, J., Whitelock, D., & Rienties, B. (2016). Assessing learning gains Paper presentation at European Association for Research on Learning and Instruction Biannual Conference (EARLI 2017).
- Rogaten, J., Rienties, B., Whitelock, D., Cross, S.J., & Littlejohn, A. (2017). Using big data to understand learning gains in knowledge and understanding in Social Science and Science. Paper presentation at European Association for Research on Learning and Instruction Biannual Conference (EARLI 2017).
- Rogaten. J., Rienties. B., & Toetenel. L., (2017). Learning gains as a function of learning design in Higher Education. Paper presentation at HEA Annual Conference 2017.
- Rogaten, J., Rienties, B., Whitelock, D., Cross, S.J., Littlejohn, A., Sharpe, R., Lygo-Baker, S., Scott, I., Warburton, S., & Kinchin, I. (2016). Multilevel modelling of learning gains: The impact of module particulars on students’ learning in Higher Education. Symposium Presentation. SRHE conference, Celtic Manor, UK
- Rogaten, J., Whitelock, D., & Rienties, B. (2016). Assessing learning gains. Paper Presentation. Technology Enhanced Assessment Conference 2016, Tallinn, Estonia
- Bart Rienties, Principal Investigator, email Bart.Rienties@open.ac.uk
- Jekaterina Rogaten, Research Associate, email Jekaterina.firstname.lastname@example.org
Authors of case study: Jekaterina Rogaten and Bart Rienties