You need cookies enabled

HEFCE closed at the end of March 2018. The information on this website is historical and is no longer maintained.

Many of HEFCE's functions will be continued by the Office for Students, the new regulator of higher education in England, and Research England, the new council within UK Research and Innovation.

The HEFCE domain - www.hefce.ac.uk - will continue to function until September 2018. At this point we will close the site entirely and all its information will only be available from the National Web Archive.

Cookies

You need cookies enabled

University of Warwick

This project comprises four work packages, one in curriculum areas and three related to employability. They will use a range of methods, including longitudinal and cross-sectional approaches, to test and develop tools to effectively measure student learning gain.

Led by the University of Warwick, in partnership with: University College London; University of Birmingham; University of Bristol; University of Cambridge; University of Durham; University of Exeter; University of Leeds; University of Liverpool; University of Manchester; University of Nottingham; University of Southampton; University of Sheffield; University of York; Imperial College London; King’s College London; Newcastle University; Queen Mary University of London

Project methodologies: Grades; Surveys; Standardised test; Mixed methods; Other qualitative methods

Pilot case study - University of Birmingham lead

Student international experience: motivation and aims for learning gains

Led by the University of Birmingham, in partnership with: University of Cambridge; University of Exeter; Kings College London; University of Southampton; University of Liverpool; University College London.

This qualitative project explores the aims, motivation and self-perceived employability learning gains of a sample of UK students who have studied abroad, worked abroad or undertaken an on-campus experience designed to enhance internationalisation, as research has identified that widening participation students are disadvantaged in their ability to undertake work or study abroad. 


Aims and objectives

  • To provide the sector, and especially careers practitioners and those supporting study abroad, with better evidence on what students believe they gain from international experience
  • To enable better design and promotion of international experiences
  • To create enhanced pre-experience preparatory resources and post-experience reflection
  • To identify, and enhance, on-campus opportunities to produce similar levels of employability learning gain to traditional international experience.

Experiences and outcomes

To date 90 semi-structured interviews have been completed with students undertaking the Study abroad, Work abroad and On-campus experiences. Approximately 20 post-experience interviews are currently being transcribed with a further 70 being planned and arranged.

As no post-experience interviewees have yet been analysed it is not possible to identify final outcomes at this time. However, the project timeline will see a final report submitted in early spring 2018. 

From the student perspective: The project has increased participants’ awareness of motivation and learning aims by asking them to explicitly consider these.

From the institutional perspective: The project aims to inform institutional strategy (see project aims and objectives).

New learning

Pre-experience interviews identified that participants do not undertake international travel experiences primarily due to differing perceptions of risk leading to risk averse behaviour, as well as being influenced by perceptions of relevance to subject of study and practical considerations e.g. extra year, money, course of study restrictions.

Pre-experience interviews identified that participants’ motivation and aims for international travel and experience tends to vary depending on whether the experience is compulsory or voluntary. For those for whom the experience is voluntary, intrinsic motivations (e.g. personal development and ‘make a difference’) were more emphasised in accounts, although extrinsic factors were also mentioned such as CV and relevant experience.

Some differences were identified between the aims of those undertaking work experience and study experience. Generally for those undertaking study abroad the three main themes for gain were academic related factors, employability related factors, and personal interest and development - with no noticeable difference in frequency or emphasis between them.

However, for those who were undertaking voluntary work abroad, the same three themes were present but there was a greater emphasis on the personal development aspect of the experience (for those who were undertaking a compulsory work placement there was a similar pattern to those undertaking international study abroad).


Impact of the project

Senior managers:

"As the senior manager leading the Internationalisation project I am extremely hopeful that this research will produce both valuable evidence that will inform the general mobility/employability debate and ultimately provide university careers services with practical tools and interventions to enhance their work in this area." Eluned Jones, Director of Student Employability, University of Birmingham

"I am excited by this project and the rich data it will provide us in terms of understanding the learning gain students undergo as a result of a variety of internationalisation experiences. It will enable our students to increase their awareness of the skills and behaviours they have developed and will inform ways we support, and encourage, our students to undertake more of this type of activity, ultimately improving their prospects in the labour market after graduation." Sue Welland, Deputy Director External Engagement and Internationalisation Strand Project Manager, University of Birmingham

Researcher:

"The main concerns and barriers with experience abroad tend to be cost, personal well-being, and lack of language skills (Broadening Horizons, 2015). I hope this project will help those students who cannot travel abroad to gain similar skills & attribute enhancement as those who can." 


Challenges

Recruitment of students to the project proved to be an initial challenge. However, a mixture of small incentives and the persistence of careers service colleagues ensured that pre-experience targets have been met.


Methodologies

Using predominantly qualitative methodology, enriched by repeated measures of quantitative data, this project is researching three distinct aspects of international experience. The first is Study abroad - where typically a student undertakes a full academic year studying their subject at an overseas university.

The second is Work abroad - where students undertake a period of overseas employment and/or work experience usually during the summer vacation.

The third area provides the contrast as this explores activities with an international focus that students undertake while remaining on their home campus.

All students are interviewed using a standard set of questions before they start their experience and are then interviewed again upon their return to the UK or completion of the on-campus experience. The project seeks to discover if the students believe that their skills and abilities have been enhanced by their experiences.


Publications and forums

  • LEGACY webpage
  • NAGCAS annual conference, University of Adelaide, 28–30 November, 2016
  • SRHE Annual Research Conference, Celtic Manor, Newport, 7-9 December 2016
  • AMIN Seminar, University of Warwick, 17 May 2017

Further information

Contact:

Case study authors: Eluned Jones, Toni Wright

Find out more about the LEGACY learning gain project

Pilot case study - University of Cambridge lead

Engaging students in a longitudinal study testing measures of learning gain across disciplines in higher education

Led by the University of Cambridge in partnership with: Queen Mary University London; University College London (UCL); University of Warwick; University of Birmingham; University of Manchester; University of Exeter; Durham University; University of Newcastle; University of Nottingham; Sheffield Hallam University

The Cambridge Strand of the LEGACY project is generating new empirical evidence on student learning gains, exploring both the student perspective on learning and learning gain in higher education, and testing out questionnaire-based and test-like instruments in a large-scale longitudinal survey.


Aims and objectives

The aims of the Cambridge Strand of the LEGACY project are:

  • To identify the core dimensions of student learning gains across different disciplines
  • To develop a measurement instrument that captures the core dimensions of student learning gains
  • To establish the validity, reliability and usability of the developed measurement instrument for student learning gains across different disciplines
  • To produce new scientific knowledge about the longitudinal development of student learning gains throughout higher education in different disciplines
  • To test a longitudinal explanatory model of student learning gains including a range of personal and environmental factors.

Experiences and outcomes

The project has to date engaged with a large number of students, in two distinct phases.

In the first phase, 34 undergraduate and postgraduate students were interviewed extensively about their views on learning in higher education, with a specific emphasis on the set of skills, competencies and abilities that they saw as the most important non-subject-specific learning dimensions.

In the second phase, over 30,000 students in four large disciplinary areas were contacted via email and invited to take part in a longitudinal survey measuring dimensions of learning identified by the initial interviewees, as well as by an extensive literature and measurement instrument review.

Over 5,000 students engaged with the first wave of the survey, and over 3,000 continued their involvement in the second wave, a response rate well above industry standards, which would suggest that the topic of learning and learning gain is of interest to the student population in the study.

New learning

The project is ongoing: it is deriving evidence as to the quality of the measurement instruments, both existing and newly-developed, particularly in relation to how reliably different dimensions of learning gain can be measured, and also revealing the extent to which these different dimensions work together as a larger explanatory model for cross-discipline learning gain in higher education.


Impact of the project

Senior managers: Academics and senior managers in higher education institutions have noted the scientifically robust manner in which the study is being conducted, in particular the attention to the development of theory and a conceptual framework to underscore the development of measurement instruments that can capture non-subject-specific learning and learning gain.


Challenges

Engaging students in a two-year, three-wave survey is a difficult task, with competing pressures on students’ time. Therefore, at scale, measurement of learning gain that is voluntary (and even paid) is not straightforward, and may require inventive ways of keeping students engaged.

Our findings suggest that student engagement and response rates can be improved by:

  • engaging with senior staff from partner universities about the purposes of the study, and communicating clearly about expectations for both researchers and partners
  • informing students precisely as to the purposes of the study
  • managing expectations about what the survey questionnaire will be like
  • using high-quality measurement instruments that are short as possible, but also interesting for students, and potentially slightly challenging
  • engaging in personalised communication to students, highlighting the importance of every single response to meeting the study’s aims
  • timing the survey wave so as to avoid exam periods, reading weeks, or student holidays.

Methodologies

This strand relies on a sequential mixed-methods design, whereby the initial qualitative phase (consisting of interviews with students) informed the development of data collection instruments for the second, quantitative phase (the longitudinal survey). Additionally, administrative data will be linked in to the survey data in as much as possible. Tying both phases together, a comprehensive conceptual framework was developed, to include dimensions of non-subject-specific learning gain.

The measurement instruments chosen for the longitudinal survey reflect this conceptual framework and also take into account issues surrounding practicality of at-scale measurement. Twelve separate measurement scales are included in the survey: some used with permission from other authors (e.g. deep learning); some adapted from existing instruments (e.g. social and emotional engagement); others compiled from an existing item bank; and others derived specifically for the purposes of this research (e.g. attitude to research).


Publications and forums

The work has been featured at Teaching and Learning Forums at several universities, as well as at international conferences.

Academic publications are in preparation.


Further information

Case study authors: Dr Sonia Ilie, Prof Jan Vermunt, Prof Anna Vignoles

Find out more about the LEGACY project

Pilot case study - University of Nottingham lead

Using the Career Adaptability Scale-UK with first year students

The project measures how those who ‘least engage’ with career readiness develop their career adaptability across the psycho-social areas of concern (engagement and positive attitude towards the future); control (over one’s situation); curiosity (about opportunities and possibilities); and confidence (believing in yourself and your ability to achieve your goal). It also explores employability enhancement behaviour via reported engagement with career support activity.


Aims and objectives

  • To test the efficacy of quantitative and qualitative measures of perception of learning gain for employability over students’ first two academic years as measured by:
    • Outcomes of repeated applications of career adaptability questionnaire
    • Engagement with career support activities over time
    • Students' qualitative evaluations of career readiness over time
  • To facilitate building knowledge and capacity for measures of learning gain by training and developing of careers professionals to use the career adaptabilities questionnaire with their students

Additional project objectives:

  • To enable participating institutions to pilot a new approach to interaction with level 1 students on a course where engagement has traditionally been a challenge
  • To develop an online version of the Career Adaptabilities questionnaire which could be used across the sector to support students in their career development.

Experiences and outcomes

To date two administrations of the questionnaire have taken place: one at the beginning of the first year and one 8 months later near the end of the first year. Illustrative qualitative interviews are also taking place which are intended to flesh out possible reasons behind any observed quantitative change.

As the project is only halfway through it is not possible to identify final outcomes at this time, although there is a recorded increase in 'concern' (student engagement with their future after university) between commencement of the project and the end of the first year.

From the student perspective:

The project has increased participants’ awareness of the importance of considering their employability related psycho-social resources by asking them to explicitly consider these via the administration of the questionnaire and via questionnaire feedback.

The online feedback offers participants advice and guidance on engaging with resources and activities which are designed to enhance employability.

From the institutional perspective: 

The project is designed to test new ways of engaging students with their employabilities which may in turn impact on DLHE results and thus the TEF.

New learning

One set of repeated questionnaire administrations has taken place and a significant difference has been shown between scores on the ‘concern’ factor of career adaptability over the 8 months between administrations. This suggests that over the 8 months the project has been running, first year participants have significantly increased their engagement with developing a positive attitude towards their future.


Quotes about the impact of the project

First year students:

"It definitely made me think about wanting to be more active in seeking advice for careers…"

"It did prompt me to […] think and to research a bit more in depth on my own; I’ve been inclined to go away and do my own research on top of the career adaptability”

"It promoted me look more into the careers advice services […] which I hadn’t been to before; it encouraged me to seek out what assistance they could give whereas I hadn’t done before."

"It definitely prompted me in terms of how my [current] learning could help [going forwards]."

Staff in direct contact with students:

"One student told me that filling in the first questionnaire had made them aware of the need to do something."

Senior managers:

"Participation in the learning gain project provides an interesting opportunity to explore collaboratively a methodology that helps individual students with their career development, and collectively could provide a measure of employability learning gain. The sharing of experience between partners will be used to develop recommendations for the use of the career adaptability instrument to benefit, and potentially assess, the development of student employability within an institutional context." Dr Nalayini Thambar

Researchers:

"The Career Adaptability Scale has the potential to serve as a measure of employability-related learning gain and an employability enhancement intervention. This project is an exciting opportunity to test its potential uses in the HE setting."


Challenges

Recruitment of students to the project proved to be a challenge, and retaining participants for repeated administrations was an even greater challenge. However, the project is specifically working with cohorts who tend to be hard to engage so this is not a surprising finding.

Ongoing communication in a variety of styles as well as incentives are being used to address the issue. The persistence and dedication of careers service colleagues has been no small factor here too.


Methodologies

The CAAS-UK (Wright, Bimrose & Frigerio, in review) has been developed from the CAAS2-International (Savickas & Porfeli, 2012) for use in the UK. Nottingham University has developed an online version of this questionnaire which incorporates feedback and advice emailed to the student as a result of their completion of the questionnaire. This new resource could be further developed for the benefit of the whole of the HE sector. 

Publications and forums

  • LEGACY webpage
  • NAGCAS annual conference, University of Adelaide, 28–30 November, 2016
  • SRHE Annual Research Conference, Celtic Manor, Newport, 7-9 December 2016

Further information

Contact:

Authors of case study: Toni Wright, Stephen Sharkey, Nalayini Thambar

Find out more about the LEGACY project

Pilot case study - University of Warwick lead

The impact of a R2 Strengths profile (with 1-1 feedback) on the employability of final year undergraduates

Led by the University of Warwick, in partnership with Durham; Manchester; Newcastle; QMUL; Sheffield.

To capture the impact of the R2 Strengths profile on Home and EU Finalist Undergraduates’ self-awareness, self-efficacy, confidence, career readiness and ability to secure a graduate job or further study offer. 


Aims and objectives

  • To identify the impact of specific interventions designed to support learning gain for employability as measured by quantitative and qualitative distance travelled relating to R2 intervention
  • To facilitate dialogue within the sector and with experts to facilitate shared critical understanding of the efficacy, scalability and viability of different measures of learning gain:
    • Creation of a resource bank of all project and other relevant ancillary materials
    • Seminar series drawing on academic, policy and employer interests and engagement.

Experiences and outcomes

From the student perspective:

Students’ feedback from the qualitative interviews highlights that completing the R2 Strengths profile at the beginning of the final year of their undergraduate degree had a range of positive impacts, both tangible and intangible. As the quotes below illustrate, students described how gaining greater awareness of their different strengths gave them a greater sense of efficacy in job search, more motivation to pursue their goals, and greater clarity and self-assurance around their career choices.

"I’ll give the example of the job I got playing in this band, I knew the piano player I would be taking over from. I was under confident about that because I was not sure how good I was compared to him. But I thought that I am good at meeting people and making connections and I think that sometimes that is half the battle when you are trying to get a job. And that made me apply for it and that gave me the confidence to put myself out there for it really."

"My main thing is using the unlearned strengths to motivate your learned strengths, especially for someone who’s ever stuck in a rut – that is such a good tool because you start looking at situations differently. Instead of looking at it like – oh, I have to do this again - you start looking at it being like 'right, how can I really get the best enjoyment out of the situation?', […] 'what out of this situation am I going to really enjoy and I should really focus on?' So yes, that really helped."

"So there's little things that I have maybe changed my opinion on, so it's sort of made me more sure about certain things. So, from doing the testing, and having the chat, I sort of was... it confirmed more in my mind that, okay, why I'm applying for these type of things, I think that is actually what I want to do, and I think that is hopefully something that I'll be quite good at."

Students also reported that the R2 Strengths profile provided them with a holistic tool of self-reflection and evaluation of their learning gain at the end of their higher education journey.

"I think that something like this is actually much more useful than perhaps the student survey. Because I think it’s much more holistic…. […] Definitely not everyone can have a one-to-one, but something like this is much more effective in deciding what you want to do occupationally or whether you want further education. And it seems much more personal than just filling out…. I know it’s all about finance and marketization and you can’t change that but it’s a comment from my perspective. It’s a much more personal way of finding out about you rather than being just a student of the University of x, and what was your experience in that institution. That doesn’t include the strengths you acquired. When they just ask you what are you doing six-months down the line, it doesn’t really tell you which strengths you acquired in the past three years."


Impact of the project

Students:

Students’ experiences with the profile were reported to be overall very positive, with most students enthusiastically recommending similar tools for their peers. The 1-1 feedback with a careers adviser emerged as a significant factor that had considerable impact in helping students gain the fullest insights from their Strengths profile.

"I Just that I think stuff like this should be open to every student to do as part of your first and last year, that would be amazing... if you go to university and get one done then and then you leave and have another one just before you leave, just to show you what university does for you…"

"The careers adviser was really great in being able to summarise and put things into perspective with the whole Strengths Finder because, other than that, it was still helpful to see it on paper but to have it explained was really helpful, really useful."

Staff in direct contact with students:

Careers consultants undertaking the 1-1 R2 Strengths feedback informally reported that students had largely seemed enthusiastic and appreciative of the discussion. They had also enjoyed providing feedback.

Senior managers:

"For students in their final year, it is highly valuable that they know what to expect in a job interview or an interview for further study. It will be highly valuable to understand how R2 Strengths can increase this understanding, and I am eager to find out the result of this research." Dr Giles Carden, Director of Strategic Planning & Analytics, University of Warwick

Researchers:

"LEGACY’s R2 strand promises valuable results which will progress the sector's understanding of students’ learning gain. However, it would be very helpful to understand more about the part-time student experience and include the long-term impact of learning gain on employment and personal outcomes." Dr Grace Clifton SFHEA, Academic Lead - Enhanced Employability and Career Progression, The Open University


Challenges

Key challenges experienced have been how to keep students interested in the project. The initial enthusiasm for participating in the project waned slightly with the first questionnaire and again with the follow-up questionnaire.

Timings were an issue; whilst it was the right time to be providing an intervention that was aimed at supporting finalists before they were making applications, this time of year is very busy with applications and the timing of the follow-up questionnaire coincided with final exams and students finishing their courses.


Methodologies

Using a mixed methods approach, the research utilised a bespoke quantitative questionnaire and an in-depth research topic guide. Both instruments covered students’ opinion about strengths in general, their self-efficacy, and their career readiness. The quantitative survey was conducted online, and the qualitative interview took place face-to-face. The data is being analysed using SPSS and NVivo.

All participants complete Before and After Likert scale questionnaires aimed to capture distance travelled. Two of three groups complete an R2 Strengths online questionnaire. These students all received a personalised R2 Strengths profile report; half received 1-1 feedback from a careers consultant who is also R2 Strengths trained. The control group completed Before and After questionnaires only.

36 participants who received 1-1 feedback took part in qualitative interviews with a PhD Research student to explore students’ learning in more depth.


Publications and forums

  • LEGACY webpage
  • AGCAS Heads of Service annual conference workshop
  • HEA Conference
  • Legacy meetings and seminars

Further information

Contact:

Authors of case study: Anne Wilson, Heike Behle, Arianna Tassinari

Find out more about the LEGACY project

Learn more about the University of Warwick's project

See all learning gain pilot projects

 

Page last updated 13 December 2017

For further information