University of Manchester
This project will investigate ways of:
- standardising entry and exit qualifications in (and out of) HE
- designing and validating new measures of dispositions and critical skills
- modelling change in learning outcomes, for robust measurement of the ‘distance travelled’.
Project methodologies: Grades; Surveys; Standardised test; Mixed methods; Other qualitative methods
The role of dispositions in measuring and modelling learning gain
Our project involves a mixed-method longitudinal design approach to measuring and modelling learning gain. There are three principal data collection methods which are expected to be linked together:
- quantitative analysis of administrative/secondary data
- longitudinal self-report student questionnaires capturing dispositions and experiences
- interviews with students.
Aims and objectives
- To develop and design self-report instruments to measure various aspects of students’ academic dispositions at different stages of their undergraduate degree
- To design new generic instruments for the assessment of students’ critical skills
- To collect data from students at various stages of their study via grades achieved and via purpose-designed surveys and tests to validate our instruments' cross-sectionally and to assess the comparability of these measures at different time points and on different disciplines
- To use existing (secondary, administrative) data to investigate the feasibility of standardisation of current entry and exit qualifications across different subject areas
- To model learning outcomes (dispositions and grades) cross-sectionally and longitudinally in order to test value added models of learning gain and the effect of students’ perceptions of academic support and experience, as well as their background characteristics on their trajectories.
Experiences and outcomes
From the student perspective:
The project has worked closely with representatives from the Student Union, even though it was decided to drop initial plans to involve them in data collection.
The research findings will be shared with the Student Union who will disseminate results via their communications channels. So far representatives of the Student Union attended our dissemination workshops and contributed with feedback and reactions to the findings.
The Research Team is considering a number of different methods to feedback findings to students and raise the profile of the learning gain project. The University has a number of social media communication links to students and the Research Team have met with marketing officers that can facilitate the dissemination of information through these channels. The style and format of the messages will be important in connecting with students and various forms are being considered.
From the institutional perspective:
We are working with a range of academic schools holding individual meetings with each School to understand what kind of outputs they would like from the research. Particularly useful have been our meetings and discussions with the Undergraduate heads of teaching and learning. In the context of the TEF, academic Schools have supported the project and are planning to use the research findings to inform their future teaching and learning practices.
A range of dissemination techniques have been developed to ensure that academic Schools understand the key findings that are relevant to their students:
- a series of detailed reports were produced for individual Schools that covered the initial set of results generated by the project. The initial research findings have provided a wealth of information regarding students’ expectations of teaching and learning across a range of subject areas and also students’ confidence across a range of academic skills.
- we hosted a dissemination event to engage with key contacts across the University in relation to the research findings. This helped in understanding how they can be applied to inform learning and teaching strategies, and student recruitment.
Reported findings are used by Schools to refine how courses are marketed and how courses are meeting the range of academic skills required by students.
Results from our first data collection provided the basis for a paper (under review) on the validation and potential use of new measures of students’ confidence:
- ‘confidence in learning though traditional university transmission pedagogy’
- ‘confidence in learning through social means’, and
- ‘confidence in learning through problem solving, reflection and critical thinking'.
Such measurement allowed us then to model learning gain and provide insight into the relevance of measures of disposition, as well as background variables. Preliminary evidence suggests that learning gain (measured as value-added between A-level entry qualifications and first semester results) was significantly associated with students' confidence with ‘traditional’ learning, but not with ‘social’ and ‘problem solving’ confidence.
We have also started exploring how other student perceptions, dispositions and background variables are implicated.
The data will also be explored further within the doctoral thesis of Martyn Edwards which aims to develop new understandings of the relationship between academic dispositions and learning gain.
Gaining ethical approval for the project through the University’s internal ethics board was a lengthy process. A key component of the research design is linking survey data to the wealth of administrative and academic data held by the University, and ensuring that this process met the ethical requirements was an issue that had to be worked through with the ethics board.
Securing support from Academic Schools across the University has been very resource intensive. The Project Team initially worked with senior colleagues in the Professional Support Services to develop a data collection model. The most suitable model of engaging Academic Schools was to approach Heads of School and ask to work with Undergraduate Directors of Teaching and Learning. Subsequently meetings were held with representatives from Academic Schools to discuss the project and develop bespoke data collection models across each Academic Schools that capture students across years 1 to 3. The University of Manchester has 17 Academic Schools; meetings have been held with the majority of Schools - which has been a time consuming process.
A variety of data collection methodologies have been developed to access students at various points of engagement with the University (e.g. laboratory work, lectures, academic advisor meetings). This process ensured that a good response rate was achieved across various discipline areas providing the wealth of data required to fully pilot the surveys.
The development of the survey instrument has involved engagement and support from a range of academic colleagues including piloting the questionnaire with two groups of students. Working across disciplines has created the need for two versions of the survey instrument linked to STEM and non-STEM subjects. Extensive work has also been carried out to link learning outcomes measured by the survey to the learning outcomes across academic programmes. The development of the survey instrument has involved a number of stages and has taken longer than expected but has created a sound data collection tool.
Collecting longitudinal data from across the academic career of students is anticipated to be challenging. It has been easier to access students in year one of their academic studies due to larger lectures resulting from more common modules and lower levels of survey fatigue amongst this group. Gaining the crucial longitudinal data across years 2 and 3 of students’ academic studies has been more difficult, particularly using online surveys. Working with Academic Schools to develop bespoke data collection models has been successful and this will be taken forward into the final year of the learning gain research project.
Using the IRT Rasch Measurement framework so far we have validated the three scales presented above in the context of the submitted paper.
We have also produced the following (some new some recalibrations):
- Confidence in general maths skills
- Confidence in applied maths skills
- Disposition to complete the chosen course
- Critical thinking perception
- Critical thinking competence
- Perceptions of the transitional gap
- Positivity towards the transitional gap.
Publications and forums
Pampaka, M., Swain, D., Jones, S., Williams, J., Edwards, M., & Wo, L. (under review). Validating constructs of learners’ dispositions for measuring learning gain. Higher Education Pedagogies (Special Issue, Learning Gain).
Contact: email firstname.lastname@example.org
Authors of case study: Maria Pampaka, Daniel Swain, Steven Jones, Julian Williams, Lawrence Wo, Martyn Edwards