Home > Publications & reports > Publications

UK review of the provision of information about higher education: National Student Survey results and trends analysis 2005-2013

July 2014 | ref: 2014/13

To:

Heads of HEFCE-funded further education colleges, Heads of HEFCE-funded higher education institutions, Heads of universities in Northern Ireland
Heads of UK publicly funded higher education institutions, Heads of HEFCW-funded further education institutions

Of interest to those responsible for:

Quality assurance and enhancement, Student services and Planning practitioners

This document presents the results and trends from an analysis of the first nine years of data from the National Student Survey (NSS). The two main areas of investigation are the robustness of the NSS results, which is considered by investigating response patterns, and the difference between the NSS results split by a variety of student and course characteristics such as ethnicity, gender and subject.

Correction: This document was corrected on 12 December 2014: a number of percentages and graphs plus Annexes H, I, J and K have been revised. None of the changes impact substantially on the conclusions drawn about student satisfaction.

Executive summary

Purpose

1. This document presents the results and trends from an analysis of the first nine years of data from the National Student Survey (NSS). The two main areas of investigation are the robustness of the NSS results, which is considered by investigating response patterns, and the difference between the NSS results split by a variety of student and course characteristics such as ethnicity, gender and subject.

Key points

Robustness of the results

The NSS question scales are valid across all years of the NSS

2. The National Student Survey (NSS) originally contained 22 questions which were split into six sets of questions, also known as NSS question scales. These six scales were:

  • Teaching and learning
  • Assessment and feedback
  • Academic support
  • Organisation and management
  • Learning resources
  • Personal development.

3. There is an additional standalone question regarding overall satisfaction.

4. Statistical methods can be used to determine whether the individual NSS questions can be grouped into related sets of questions. This analysis of the responses indicates that the individual questions cluster into the same groups as the NSS question scales listed in paragraph 2. Hence the way in which the questions were originally grouped formed a solid structure for the survey which has remained consistent over the nine years of the NSS.

When presenting results, ‘definitely agree’ and ‘mostly agree’ can be combined as a measure of satisfaction. However, ‘definitely disagree’ and ‘mostly disagree’ should not be combined as a measure of dissatisfaction

5. Response patterns of related questions differ more depending on whether ‘definitely disagree’ or ‘mostly disagree’ is the chosen response, compared with those who choose ‘definitely agree’ or ‘mostly agree’. This suggests that the results for ‘neither agree nor disagree’, ‘mostly disagree’ and ‘definitely disagree’ should be presented as separate percentages.

The questions have become more similarly answered over time

6. The relationship in the responses between individual questions in the NSS has become stronger over time. This means that the responses to individual questions are more similar in 2013 than they were in 2005.

7. There are close relationships between the response to the overall satisfaction question and the responses to the individual questions. These relationships suggest that all the responses to the questions reflect the overall satisfaction of students.

Over 5 per cent of respondents tick the same answer for every question

8. The act of choosing the same answer to every question is known as acquiescence bias or yea-saying. In 2005 only 1.0 per cent of respondents were found to be yea-saying, but by 2013 this had risen to 5.4 per cent. This is much higher than would be expected, even after accounting for the very high levels of satisfaction and strong relationship between the answers to different questions.

9. The vast majority of yea-saying occurs with respondents choosing the ‘definitely agree’ response category.

10. There is no indication of any link between the incentives or prizes offered by institutions to their students for completing the survey and the proportion of yea-saying. There is also no relationship between the size of an institution and the proportion of yea-saying.

11. We have tested the impact of yea-saying on the NSS results by removing all responses with identical answers to every question from the analysis, and have found no material difference in the sector results. However, if the proportion of yea-saying continues to rise, there is a possibility that it could affect the robustness of the NSS results.

Online responses are the most similar

12. When the responses are split by the three survey completion methods (online, postal and phone), the strongest relationship between responses to the questions is found in the online responses. In other words, the responses of any given individual are most similar when responding online.

13. Yea-saying is present in responses amongst all completion methods, but most predominant in online responses. When yea-sayers are removed from the data, the results of the analysis in paragraph 13 still hold true.

Student satisfaction

Students have become more satisfied over time

14. The percentage of students who agree with each of the six NSS question scales and the overall satisfaction question has increased over time.

Black Caribbean students are less satisfied than the average, but Black African students are more satisfied

15. The ‘overall satisfaction’ of Black Caribbean students has always been lower compared with White students. An expected ‘overall satisfaction’ score for different student characteristics can be generated by modelling the NSS results. The largest difference between this and the actual score for ‘overall satisfaction’ from Black Caribbean students was seen in 2011, when the satisfaction score for Black Caribbean students was 73.6 per cent compared with the expected satisfaction score of 80.2 per cent. Black Caribbean students were therefore 6.7 percentage points less satisfied than expected in 2011.

16. Conversely, the proportion of Black African students who agree with the overall satisfaction question is higher compared with White students. This difference peaked in 2010, when Black African students had an expected ‘overall satisfaction’ score of 80.3 per cent and an actual ‘overall satisfaction’ score of 83.3. Hence, in 2010, Black African students were 3.1 percentage points more satisfied than expected.

The largest variation in satisfaction scores is observed when considering subject of study

17. By far the biggest variation in satisfaction is between subjects of study. The results of modelling 2013 ‘overall satisfaction’ are spread across a range of 15 percentage points when split by subject.

Students who declare a disability are less satisfied than those with no known disability

18. Across the five years of modelled overall satisfaction results, those with a declared disability have been consistently less satisfied than those with no known disability.

Action required

19. In addition to this report, there is an opportunity to explore the NSS data further using an online NSS tool which can be found at www.hefce.ac.uk/whatwedo/lt/publicinfo/nss/nsstrend/. The online tool contains data from all nine years of the NSS, and allows investigation of the results split by a variety of student characteristics (such as age, gender and ethnicity).

20. Over the course of the NSS we have had over 2 million responses which have been linked back to the student characteristics of individual respondents.

Download:

Main report and Annexes A and B

Download the HEFCE2014_13 - corrected 12 December 2014 as PDF (2,090 KB)Download the National Student Survey results and trends analysis 2005-2013 as MS Word (2,745 KB)

Annex C: Correlation heat maps for all years of the National Student Survey in chronological order

Download the Annex C: Correlation heat maps for all years of the National Student Survey in chronological order as PDF (322 KB)Download the Annex C: Correlation heat maps for all years of the National Student Survey in chronological order as MS Word (377 KB)

Annex D: Heat maps of correlation between questions split by method of response

Download the Annex D: Heat maps of correlation between questions split by method of response as PDF (967 KB) | Download the Annex D: Heat maps of correlation between questions split by method of response as MS Word (1,056 KB)

Annex E: Correlation matrices for the NSS results

Download the Annex E: Correlation matrices for the NSS results as MS Excel Spreadsheet (44 KB)

Annex F: Correlation matrices for the NSS results split by method of response

Download the Annex F: Correlation matrices for the NSS results split by method of response as MS Excel Spreadsheet (106 KB)

Annex G: Heat maps from principal component analysis in chronological order

Download the Annex G: Heat maps from principal component analysis in chronological order as PDF (376 KB)Download the Annex G: Heat maps from principal component analysis in chronological order as MS Word (442 KB)

Annex H: Modelling results for the 2013 NSS for the core population

Download the Annex H: Modelling results for the 2013 NSS for the core population as MS Excel Spreadsheet (70 KB)

Annex I: Modelling results from 2009 - 2013 NSS

Download the Annex I: Modelling results from 2009 - 2013 NSS as MS Excel Spreadsheet (46 KB)

Annex J: Modelling results for the 2013 NSS results

Download the Annex J: Modelling results for the 2013 NSS results as MS Excel Spreadsheet (58 KB)

Annex K: Institutional effects

Download the Annex K: Institutional effects as MS Excel Spreadsheet (89 KB)

Annex L: Results from a principal component analysis performed on the NSS results

Download the Annex L: Results from a principal component analysis performed on the NSS results as MS Excel Spreadsheet (44 KB)

Enquiries should be directed to:Emily Thorn, tel 0117 931 7268, e-mail e.thorn@hefce.ac.uk

Page last updated 12 December 2014

Share this: