The Manchester College
Partners: The Blackpool and Fylde College; Bradford College; City College Norwich; Doncaster College; Gloucester College; Grimsby Institute of Further and Higher Education; Hull College; Leeds City College; New College Durham; Rotherham College; Tameside College; South Devon College; Wakefield College; West Herts College; York College
Project methodologies: Grades; Surveys; Mixed methods
Measuring learning gain
The project aims to:
- develop measurable metrics for both personal skills development and academic development (learning gain)
- investigate related variables impacts on both learner’s engagement as well as their learning, in regards to both personal and academic development.
Experiences and outcomes
At this stage of the learning gain project, we have collected the learning gain survey data from both our college and our partner colleges and are now in the process of collecting management information systems (MIS) data from each college that includes grades as well as an array of personal, demographic and socio-economic data. Upon merging and finalising the array of data sets, we will then begin to measure and analyse the procured data.
From the student perspective:
At this stage of the project, student involvement has been related to collecting the learning gain survey data. Students view the learning gain survey as an enlightening tool to a range of issues related to their learning and skills development. Multiple students have expressed that undertaking the survey has made them think more about how they learn. They have also shown a level of appreciation that their voice is being heard, especially in regards to the ability to freely express their positives and negative experiences in higher education so far.
From the institutional perspective:
The Manchester College and our partners’ colleges’ management teams have been supportive of the learning gain project. At The Manchester College, leadership have expressed an understanding of how the project outcomes can benefit the college going forward and wish to be fully informed of the results of the project, in order to incorporate these outcomes into the college’s new strategic plan going forward which focuses on student learning, engagement and support. Furthermore, we have cooperated with the head of student experience and engagement to help her develop new questions and measures for their purposes, and incorporated these questions at the end of the survey circulated at The Manchester College.
The majority of literature we have encountered would indicate that when the level of academic challenge is higher, students interact with each other more frequently in an academic context. However, in both the overall aggregated sample and the sample across the institutions that data was collected from, it could be seen that this was not the case in our research.
The impact of the project
From the student perspective:
“It is nice to have the opportunity to say what we think about the course, I feel like my opinion is being valued.”
“It’s clarified what I want to do with my future. It pushed me to work harder at my place of work because I want to achieve the best.”
“Doing the survey made me think about my own learning more, helped me realise a few things about myself.”
From the staff perspective:
On the project looking at a wider scope on learning gain:
“I think the soft skills are important because it’s impossible to completely align your career with the course choice. You know, there’s always going to be that element where, a lot of jobs aren’t degree specific, and it’s what transferable skills you develop, I think, can be just as important."
On how they feel the general student profile at the College learn:
“Giving them enough time to try things out and develop, make mistakes, work out their own way of improving things. Making it as practically relevant as possible. Making the academic things as relevant, the thing about an adult learner is you can’t make them do something they don’t really see the point of. They have to see the point of it - I think that’s always the key thing on how you devise your assessments and do your teaching. It’s about setting very practical, interesting assessments for them.”
On how they see student views on research around their areas:
“They don’t do it as much as we need them to. It takes a while for them to see the relevance of it. Some of them do feel that they are doing it just for your assessment, or for you, not to actually make their end artefact better. But that’s about trying to embed that process, and there can be a struggle there.”
Reflecting on the process of learning and grade reflection:
”Yeh, so looking and reflecting on the students that achieve at the end of the third year – it's very consistent in what they achieve in the first and second year – there is a definite follow through, a definite path, which I think suggest that level of knowledge gained is not only stored, but, kind of utilised and applied, in the progressing years.”
On independent study at the College:
”There is a huge correlation between that, and very clear, with marks and achievement, and knowledge and contribution in sessions, with who is doing that additional independent study – sorry, not even additional, it is essential – and who isn't, it is quite clear.”
On the project in relation to the students:
“This is your opportunity to get the way that you want to work and the way that you want to learn, understood by the College and governing bodies.”
From the senior managers' perspective:
"The learning gain project is a much-welcomed project, which will lead to a much richer understanding of student engagement and of students entering into higher education and the journey they take throughout. This data should not only inform a provider’s strategy for supporting students, but it has much potential to inform students of the benefits of independent study and student ownership across the academic community. This project will likely confirm what we already know but give it much more credibility with students and to inform students, through appropriate means.”
From the researchers' perseptive:
“Understanding student learning, from both the process and results of the project, should hopefully allow us to identify where the college can improve in regards to both better understanding what students need in order to learn best, and also improve their overall experience while at the college.”
“An ancillary benefit of the project is that the depth of data gathered through both MIS collection and detailed, wide-scoping, learning gain student surveys will allow us to better understand the demographics of our learners – and in turn see which types of learner need more support, and what kind of support and to plan for meeting their needs.”
“The comments sections for the students to air the positives and negatives of their experiences have been interesting, and have allowed us to identify several areas – both learning gain and institutional related – where the college can improve going forward due to high levels of student consensus on such issues.”
Scheduling was difficult, changes to classes and finding convenient slots for every class was the main concern. This was addressed by closely liaising with Department Heads and tutors to create an in-depth timetable for the surveys.
- A tight schedule across multiple campuses meant two surveys had to be re-arranged at short notice.
- Attendance in classes was virtually never 100%. In addressing this, we had to go back to several classes to try to get more responses.
- 50% is a success considering that the alternative, online surveys, generally garner around 25% response rates.
- Certain colleges struggled to collect survey data. To help with this we created and online version of the survey, and had a meeting in which we gave them pointers as to how we had gone about collecting the data and planning the collection.
- Data sharing an issue for some partners’ colleges. In order to address this, we re-wrote the data sharing agreement multiple times.
- Some colleges simply did not collect certain items of data that in an ideal world we would use. Therefore, we examining which demographic and personal data was essential and narrowed the scope to include just those.
- Certain colleges had all the data but in separate databases and without the capacity to merge them. In order to address this, The Manchester College’s MIS team have agreed (in collaboration with the Learning Gain Team) to handle some of the data manipulation for other partner institutions.
The project got underway late due to delays in staff hiring. Key staff members were not on board until months after the first year of data should have been collected. This has in effect delayed the project by an academic year.
While the majority of partners in the project were supportive and engaged from the beginning, there were certain institutions that were unresponsive and did not engage at all. This led to the removal of multiple colleges from the project. To combat the dwindling responses this would have meant a further two colleges were added to the project. It is also envisageable that further partners may have to be removed in the near future if they are incapable of providing survey or MIS data. However, we feel that we have already garnered enough responses for the project to continue moving forward without problems in this area.
We developed a learning gain conceptual model based on both an array of academic literature as well as specific and individual long-term teaching and learning experiences.
- A mathematical model estimating learning gain from a combination of grades and soft skills has been developed.
- A purpose-built survey measuring the learning gain was developed. This survey is used for data collection.
- The estimated learning gain for soft skills is presented in this case study.
- Management Information Systems' data has been collected from The Manchester College. This data includes grades and demographics.
- Metrics estimating learning gain from soft skills and grades are in development going forward.
- Advanced statistical analyses including Structured Equation Modelling (SEM) will be developed once we receive further data from collaborating colleges, using the collected data from both The Manchester College and our partners’ colleges.
Publications and forums
Where has the work been publicised?
The fascinating world of learning: a research outlook. An article published in Quality matters, Q Magazine, October 2016, Issue No. 16 pages 28-31. 2016.
Academic publications since 2016:
- Academic challenge’s impact on learning and personal development: An empirical study from the HE colleges. Paper submitted for publication in an academic refereed journal, 2017
- Learning Gain Modelling. Paper presented in the Sunderland College Higher Education Conference, Sunderland College, Sunderland. 03 July 2017
- Measuring Learning Gain in Higher Education, Model development challenges and opportunities. Paper presented in the HEFCE Learning Gain Pilot Projects. 26 April 2017, London
- Overview of TEF and Learning Gain. Paper presented in the Regional AoC Northwest, 30 March 2017, Leyland Best Western Hotel, Leyland, Preston
- HEFCE - Learning Gain Model Development. A workshop presented in the AoC College HE Conference and exhibition, 15 March 2017, 99 City Road Conference Centre, London.
Contact Hassan Al-Zubaidi, Case study author and Statistician (Learning Gain Project Lead); email Hal-Zubaidi@tmc.ac.uk