1. This report describes the findings of a research study to inform the development of the 2017 National Student Survey (NSS). The research involved running a pilot questionnaire, which was offered to students on an opt-in basis after they had completed the 2016 NSS. The pilot combined existing NSS questions with new and revised questions developed following a consultation held as part of the review of the provision of information. The aim was to examine whether the revised questionnaire was valid and reliable across a range of students, provision and institution types, and whether the survey worked as a whole.
2. The research tests the robustness of the questions and whether the new question scales work. The actual responses of the students have not been analysed.
3. Since the online version of the NSS may be taken on multiple types of device, the pilot survey also tested alternative interfaces and response scales in an attempt to find the most suitable layout for each. Research on the different interface and response scale options is included in analysis of the cognitive testing of the 2016 pilot NSS carried out by IFF Research Ltd. The investigation of the new interfaces and response scales is therefore not covered in this report.
4. This report belongs to a series published as part of the review of information about learning and teaching, and the student experience.
The questions are all broadly relevant to most people
5. Questions 14 (‘Good advice was available when I needed to make study choices’), 20 (‘I have been able to access subject specific resources […] when I needed to’) and 26 (‘Students’ academic interests on my course are effectively represented by the Students’ Union […]’) have the highest proportion of ‘not applicable’ responses. Questions 25 (‘It is clear how students’ feedback on the course has been acted on’) and 26 are answered the least positively overall.
The new question scales are working as expected
6. When split into eight groups, based on students’ responses, the eight question scales are identified as the most appropriate groupings for the questions.
7. Question 26 does not display particularly high correlation with any other question, although it is most closely related to Question 25.
8. This analysis excludes ‘not applicable’ responses, and thus over half of the distance learners who responded. When their responses are included, the scales no longer work as expected, meaning that the scales may not work in the same way for distance learners.
Removing the students’ union question improves the internal validity of the survey
9. All scales are found to have acceptable levels of internal consistency. However, reliability analysis suggests that the final question scale, ‘Student voice’, would work better if Question 26 was removed.
10. The analysis implies that removing any other question from the survey would only weaken the question scale in which it sits.
The ‘Learning community’ and ‘Student voice’ question scales are answered much less positively by distance learners than all other students
11. Distance learners are between 11 and 33 percentage points less satisfied than other students for Questions 21 to 26. Outside these two scales, distance learners’ level of agreement does not differ greatly from that of other students, except for ‘Assessment and feedback’, where (in line with the main survey results) distance learners have a considerably greater level of agreement.
12. In their free text responses, many distance learners mentioned that parts of the survey were not relevant to them. Over a quarter of distance learners students responded ‘not applicable’ to Question 26.
13. No action is required.