This project will analyse the measurement of student research skills as an aspect of HE skills development in six academic disciplines, and student achievement data.
Partners: City College Plymouth; Bridgwater and Taunton College; Cornwall College, Petroc; Strode College; South Devon College
Project methodologies: Grades; Surveys; Mixed methods; Other qualitative methods
Self-reporting measures for learning gain; engaging and supporting students in measures to develop self-efficacy
This project is based on evaluating the value of using research methods learning as a proxy measure of learning gain. Understanding and applying research methods is a core component of higher education across disciplines and this project examines how students knowledge, skills and experience can be captured and contextualised across disciplines.
Aims and objectives
The project aims to:
- provide an evaluation of the feasibility of measuring learning gain through students’ skills, knowledge and confidence in research methods (RM), which then provides a contextual appreciation of the diverse learning journeys of students
- provide a balanced evaluation and conclusion on the feasibility of using RM to conceptualise, define and measure learning gain
- use a range of methods that enable evidence-based comparative judgments on facets of learning gain
- contribute to raising awareness of leaarning gain and dialogue within the HE sector including College HE.
- establish developmental capacity, building a suite of resources and makes recommendations regarding the potential for research methods to predict and measure learning gain.
Experiences and outcomes
A core part of the project has been the development of a self-reporting survey for students to engage with during level 5 and 6 of their studies. The survey was developed to be extensive and inclusive for all discipline areas within the project: Business and Social Science, Humanities and Science.
Students completed this survey at the beginning and end of level 5, and finally toward the end of their level 6 studies. These timeframes were selected to capture baseline data prior to engagement in substantial research methods training, then subsequently to record their learning as they completed this training and engaged with research projects (either independently and in groups).
From the student perspective:
Student reflections have been used to evaluate the impact of the self-reporting questionnaire to their learning. Students spoke about their understanding of the value and purpose of research methods in relation to their cognitive development and skills acquisition. Most students were able to identify the value of research methods in relation to learning within their own disciplinary activities.
“Research is the process of evaluating documents, images and other information from the past to assist understanding of actions and events.” Humanities student
“Opens up the mind - allows for more unbiased view and an educated view, broadens understanding.” Science student
Final student reflections on the project show clear evidence of the value of undertaking self-review activities (surveys and reflections). Students identified the survey as a mechanism to benchmark their own learning and development. Revisiting the survey enabled them to see how they had progressed since the first time that they had completed the survey.
“There were some [items from the survey] in the first year that I didn’t know of and I went off and looked them up and interacted with them. It helped me build [my understanding of research].” Science student
The project team believes this provides a strong case for integrating the self-reporting survey into research methods provision, to allow students to capture current levels of learning, and engage in dialogue with peers and research method tutors to direct future learning. This would allow students to take a proactive role in their learning gain.
Recent work has involved piloting a research methods self-evaluation tool with level 6 students. This resource is based on the collective learning made by the project team from the ongoing analysis of the survey (and related) data. Initial feedback from the students indicates this tool would assist in identifying gaps in their understanding of research methods.
“I just really like this feedback from it, instead of just doing a survey, you get something back”. Science student
From the institutional perspective:
The staff involved in the project have been positive about the value and use of a self-evaluation tool to support teaching and learning. However, early work to explore the implementation of such a tool on an institutional level has received a mixed reaction. For some staff who do not engage with research, or do not using research-led or informed methods of teaching, the value of this tool as a measure of LG is questionable.
Some of these responses mirror the concerns expressed by staff at the start of the project and can be related to potential negative conclusions made about teaching. This seems to be the case in programmes where research methods are embedded within a broader framework or there is a focus on applied research methods delivery and secondary research. The issue for this group of staff is the relevance of the tool to the specific programme. It is important to note that many staff will not be aware of the alternatives in measuring LG which would not offer any such contextualisation.
Staff considering this on an institutional level have pointed to its value in teaching and learning, supporting programme design and review, and enabling students to gain a wider perspective on the value of skills developed within research methods.
The project has developed from a focus on evaluating research methods across disciplines and environments to understanding the value of research from a student perspective. The potential of research methods teaching to develop self-efficacy across disciplines and environments has become clearer as the project has progressed.
Quotes about the impact of the project
“It is also necessary because it enhances not only your knowledge but your ability to think critically about knowledge.” Social science student
“Research is finding out the best way to do things through trial and error, experiments and observation. Changing variables in x will lead to observable changes in results such as profitability, so this can be used to work out how things should be done.” Business student
“I find research to be the most interesting part, in which the majority of what is found can be criticised and looked at from various perspectives.” Humanities student
“It kind of nudged me towards the MA as well as I know now that I have the relevant skills to progress, so I think it’s pointed out and honed the ones that would be useful.” Humanities student
“It is eye-opening how you’ve changed from the first year to now, and it does build your head a little bit, an ego boost.” Social science student
Staff in direct contact with students:
“I think it’s good to do, because… I will put my head above the parapet and say that sometimes I think I don't put myself in their shoes perhaps as often as I should do.”
“I think the principle of using this as a form of self-assessment would be good. If done online, with staff able to access overall data, it could also be used to help identify areas where students as a cohort are struggling.”
“Some staff have reservations that the tool could be used to ‘judge’ teaching rather than be a genuine tool to help students and to get them reflecting on their own progress.”
“The issue in any project such as this is buy-in from the staff and then the students. If it’s made too complex and imposed from outside it won’t happen.”
“Staff found it useful to know which topics their students were least and most confident in. Providing staff with a summary of students' self-evaluations prompted them to critically reflect on certain assumptions they were making about how students conceptualise and approach research methods.”
“Some stage 3 students felt the tool helped make their progress in research methods more apparent, and hence in their final year, increased their confidence in pursuing future goals whether in higher education or future careers.”
Challenges have taken the form of staff and student buy-in. Where staff engaged and supported the project activity - and tried to actively build it into research methods sessions or discussions - student engagement was good. Staff needed to know that there would be a positive outcome from engagement (i.e. for students' experience, teaching enhancement, even staff development) – some staff did not engage where they felt uncertain about the way that the results would be used.
Working on contextualisation of the survey and laying out the analysis that may be useful to support teaching activity was the attraction for those who engaged. Tracking and engaging students across years and institutions where students progressed from a partner college to the university was difficult. Personalised communication was used wherever possible.
Student incentives were used but the main incentive for students was in feeling that this engagement was a part of their programme that they would benefit from. In developing the research methods toolkit, the project team feel explicit integration in the delivery of research methods provision should promote student engagement. Integration of the outcomes of students' self-evaluation into goal setting and discussions with tutors would allow students to recognise current strengths, areas for development and shape their learning, allowing them to take a proactive role in their learning gain.
An existing validated research methods survey was used as a starting point for the survey (Gray et al., 2015). This was reviewed and developed with staff from across subject disciplines to ensure that language was appropriate for all students.
Analysis from the surveys has been used to modify and contextualise the survey so that it can be applied according to the needs of all discipline areas. This will be the basis of the toolkit that is developed from the survey. Some parts of the new toolkit are generic and apply to all HE students, but other sets of questions are more discipline-focused and a third set of questions can be contextualised to a specific programme.
Publications and forums
Where has the work been publicised?
Turner, R., Gray, C., Sutton, C., Swain, J., Muneer, R. & Stone, M. (2017). Can students’ knowledge, skills and experiences of research methods serve as a proxy for learning gain? Workshop delivered at SEDA Spring Conference ‘The quest for teaching excellence and learning gain: issues, resolutions and possibilities’, Marriott Victoria and Albert Hotel, Manchester: 11-12 May 2017.
Turner, R. (2017). Employing research methods to measure and characterise learning gain in HE. Invited talk presented at ‘Next steps for Learning Gain: supporting excellence and improving student outcomes’, Westminster Briefing, ETC Venues, London 28 February 2017.
Turner, R. (2017). Employing research methods to measure and characterise learning gain in HE. Invited talk presented at the AoC Scholarship Project’s Scholarship Development Managers mid-term meeting, Woburn House, London: 16 January 2017.
Sutton, C. (2016). Gathering and analysing student achievement data. Inside Government Learning Gain Event, Congress Centre, London: 11 May 2016.
Turner, R., Swain, J., Gray, C., Sutton, C., Muneer, R., Stone, M. & Kneale, P. (2016). Employing research methods to measure and characterise learning gain in HE. Paper presented at the 14th VCs' T&L conference, University of Plymouth: 30 June 2016.
Academic publications since October 2016
Turner, R., Gray, C., Sutton, C., Swain, J. & Muneer, R. Developing a ‘generic’ measure of learning gain for integration into the teaching of research methods with undergraduate students. Under review with Higher Education Pedagogies for a special issue on Learning Gain.
Contact: Claire Gray, email firstname.lastname@example.org
Authors of case study:
Claire Gray, Reema Muneer, Carole Sutton, Julie Swain, Rebecca Turner