University of Portsmouth
This project will develop psychometric tests of learning gain. It will focus on non-cognitive elements, such as resilience and employability identity. The project will also explore ways to develop learning gain and the relationship with early preferences for learning, demographics, faculty input, student engagement, UCAS points and degree outcomes.
Partners: University of Southampton; Royal Holloway; Roehampton University
Project methodologies: Grades; Surveys; Standardised test; Mixed methods; Other qualitative methods
Education as transformation: Developing measures of non-cognitive development
In exploring the views of higher education (HE) stakeholders (students, staff, parents and employers) what became clear is the value placed on HE as a vehicle for transformation. Hence, using mixed methods, we are seeking to devise and test measures of learning gain as defined by the development of non-cognitive skills, including resilience, graduate capital and identity, self-theories and implicit theories.
Aims and objectives
- To gain an understanding of stakeholders’ perceptions of learning gain and “graduateness”
- Explore the similarities and differences between stakeholders’ perceptions
- To use these perceptions to inform psychometric tool development.
- To develop reliable and valid psychometric tests of undergraduates’ learning gain (non-cognitive skills). We are focussing on the development of students’ non-cognitive skills, including resilience, graduate capital, and self-theories (self-esteem and self-efficacy), and implicit theories (implicit theories of intelligence) as a route to understanding the transformational potential of HE
- To explore the development of these factors and relationships between them
- Understanding the degree to which engagement with university life determines transformation
- Understanding if any demographic factors (BME status, gender, age, first generation status) determine engagement with university life and transformation.
Experiences and outcomes
From the student perspective:
Not yet applicable. However, students taking part in the quantitative survey will be offered the opportunity to receive a report at the end of the project. This report will show students how they have developed on each of the measures over the course of the study. This will potentially be useful for students to inform their own development action planning.
From the institutional perspective:
Not yet applicable. However we envisage that the project will enable us to provide measures or evidence to support some of our recently refined “Hallmarks of a Portsmouth Graduate”.
Although somewhat anecdotal at this stage, the data does appear to show that stakeholders place greater value on many of the features that we consider to be the Hallmarks of a Portsmouth Graduate.
For example, a simple word frequency analysis of the first 50 Parent Postcards that we have collected shows that parents most frequently mention non-cognitive factors as factors that they hope their child will gain during their time at university. The most frequent word to arise is “independence”. Other frequent words include “confidence”, “opportunities”, “enjoy”, “happy” and “develop”.
We have also conducted some preliminary analysis* of the cross-sectional data from the Portsmouth-only cohort. For the measures that we have explored, analysis has revealed a number of interesting significant changes across the year groups, with Year 3 (Level 6) students having higher Self-Efficacy, Learner Resilience, and Graduate Capital than Year 1 (Level 4) students. Students in Year 3 also appear to have a significantly higher strategic approach to learning and a significantly lower surface approach to learning (measured using the ASSIST) than Year 1 students. Self-Esteem, and deep approach to learning, both appear to increase across the year groups, but not significantly so. There is no change however for implicit theories of intelligence.
*This really is preliminary analysis: it is cross-sectional, based on relatively small samples, and the new tools that we are developing throughout this project are yet to be properly refined and validated. Furthermore, the real confirmation will come from seeing if these changes or differences appear in the longitudinal data.
We have already met our success criteria for three of the four qualitative strands of the project (Student Focus Groups, Parent Postcards and Employer Interviews). We are collecting more data for two of these (Parent Postcards and Employer Interviews) to ensure saturation. We aim to begin more formal data analysis in autumn-winter 2017. We will then be in a position to share what we learn from these strands of the project.
We have conducted some preliminary analysis on our quantitative data, but are still awaiting registry data from some of our partner institutions. We will not be collecting Time 2 data for the full cohort until September 2017.
Due to various difficulties with getting contracts signed etc. we were not in a position where all institutions were ready to collect Time 1 data when initially planned. This has meant that we have moved to having two cohorts:
1) The Portsmouth-only cohort who we will be able to follow for three academic years (we also collected some cross-sectional data in the first year of this); and
2) Full-team cohort who we will follow for two academic years (with three time points) across all of the institutions.
As with many of the other projects we have faced difficulties with recruitment. We are concerned that retaining the samples we have will present an equal challenge. Time 2 data collection for the Portsmouth-only cohort is currently underway, with a response rate of approximately 50% following three reminder emails. We are due to collect Time 2 data for the full-cohort in September. We are planning to boost responses for this cohort by using underspend on the budget to offer more incentives.
We are also facing challenges with data sharing between institutions - this has been much more difficult for some than others.
We had particular difficulties with engagement from one institution (Roehampton), who has now been replaced with University of the Arts, London.
- Parent views: being explored using “Parent Postcards” which ask parents to respond to one question: “What benefits do you hope your child will gain from their time at university?”
- Employer views: being explored via one-to-one interviews
- HE staff views: being explored via focus groups
- Student views: being explored via focus groups.
We are developing new psychometric tools to measure the following:
- Graduate capital
- Learner resilience
- Self-efficacy and self-esteem
We are also including other items to explore engagement and expectations.
We are using the following existing tools:
- Implicit Theories (Dweck, 1999)
- ASSIST (Tait, Entwistle & McCune, 1998)
We are also collecting the following:
- Demographic info (age, gender, BME status, first generation status etc.)
- UCAS data
- Transcript data
- KIS data.
We will also aim to collect final outcome and Destinations of Leavers from Higher Education (DLHE) data (although this will have to be done post-award).
Publications and forums
Other than HEFCE events, work on this project has only been presented at internal events at the University of Portsmouth - the Annual Teaching and Learning Conference held on 9 June 2017 being the most recent.
- Sherria Hoskins, Project Lead, email firstname.lastname@example.org
- Emily Mason-Apps, Lead Research Fellow, email email@example.com
Case study authors: Emily Mason-Apps and Sherria Hoskins