You are in :
  HEFCE HEFCW DENI SHEFC

Report 99/66

December 1999

Performance Indicators in Higher Education in the UK

Note: This file contains the main text of the report. There are separate files for the Tables of indicators and the Annexes. All Excel tables may be downloaded together as a single .zip file (600K). Printed copies of the report are available from HEFCE, price £25.

Table of Contents

Preface

  1. Introduction
    Structure of document

  2. An overview
    Purpose
    Indicators included
    Background
    Adjusted sector benchmarks

  3. The indicators
    Description of indicators

  4. Summary of performance indicators for the sector
    Percentage from schools and colleges in the state sector
    Percentage from social classes ‘skilled manual’, ‘semi-skilled’, and ‘unskilled’
    Percentage from low participation neighbourhoods
    Non-continuation rates after first year at institution
    Projected outcomes and efficiency
    Research indicators

  5. How to use the tables
    Adjusted sector benchmarks
    Significant differences
    How to interpret the indicators
    A simple example

  6. Tables
    Introduction to Table T1
    Table T1 Participation of under-represented groups in higher education
              (a) Young full-time first degree entrants 1997-98
              (b) Young full-time undergraduate entrants 1997-98

    Introduction to Table T2
    Table T2 Participation of under-represented groups in higher education
              (a) Mature full-time undergraduate entrants 1997-98
              (b) All part-time undergraduate entrants 1997-98

    Introduction to Table T3
    Table T3 Non-continuation following year of entry to institution
              (a) Full-time first degree entrants 1996-97
              (b) Young full-time first degree entrants 1996-97
              (c) Mature full-time first degree entrants 1996-97

    Introduction to Table T4
    Table T4 Resumption of study after year of inactivity
              Full-time first degree entrants 1995-96

    Introduction to Table T5
    Table T5 Projected learning outcomes and efficiencies
              Full-time students on first degree courses 1996-97

    Introduction to Table R1
    Table R1 Share of research output per share of research input weighted by cost centre, 1997-98

Annexes

Annex A General definitions

Annex B Adjusted sector benchmarks - technical notes and more detailed information
         Adjusted sector benchmarks
                  A simple example
                  Technical notes
                  Standard deviations
         Subject and entry qualifications breakdown for the sector
                  Subject categories
                  Entry qualifications
         Tables of sector results

Annex C Calculation of projected outcomes and efficiencies
         A simple example
         Technical notes

Annex D Research indicators - technical notes

Preface

  1. The publication of this set of performance indicators is an attempt by the funding councils to provide greater transparency in the way the higher education sector operates. These indicators provide measures of performance with respect to widening access, student progression, outcomes of learning and teaching, learning and teaching efficiency and research output.
  2. Publication follows an extensive consultation with the sector, and more than two years’ work by the Performance Indicators Steering Group.
  3. The indicators should be taken as a whole. Non-completion rates for an institution cannot be considered separately from the access indicators, and both should be viewed in the context of the institution’s mission. The higher education sector is so diverse that no single measure can adequately describe an institution. However, these indicators are the first step along the road towards providing measures that will reflect this diversity.
  4. The indicators will provide more open information about higher education institutions, provide benchmarks against which performance can be compared, and allow future decisions about the sector to be made on the basis of information that is widely accepted. The funding councils are committed to helping institutions to analyse their data, where this will prove useful.
  5. Because they are mostly based on data from 1997-98, these indicators will not reflect recent activities by institutions, or the impact of funding to recognise the extra costs of recruiting and supporting mature students and those from disadvantaged backgrounds. However, they do provide a baseline against which future indicators can be compared.

1. Introduction

  1. This is the first publication in what will be an annual series providing performance indicators relating to higher education. Over time, we shall be adding to this initial set of indicators.

    Structure of document

  2. The document is designed to be read at various levels. The overview provides an introduction to the indicators, and this is followed by more detailed information about how they should be interpreted. The indicators themselves are followed by annexes which contain more technical information.
  3. The components of the document are:
    • Section 1: Introduction
    • Section 2: an overview of the document, and background information
    • Section 3: a description of the indicators included in the document
    • Section 4: summary of results for the sector
    • Section 5: context statistics, and how to use the tables.
    • Section 6: tables containing the performance indicators
    • Annexes: definitions and technical information

    Further information

  4. More detailed information will be available on the HEFCE web-site at www.hefce.ac.uk. The tables of indicators will be available in Excel format, with additional statistics that have not been included in this document. A brief overview (HEFCE 99/67) is also available.

2. Overview

  1. A common set of performance indicators have been introduced for the first time for all 175 publicly funded higher education (HE) institutions in the United Kingdom.
  2. This overview explains the purpose behind indicators in higher education, and provides the background to their introduction.

    Purpose

  3. Recognising the increasing diversity of higher education, the purpose of performance indicators is to:
    • provide better and more reliable information on the nature and performance of the UK higher education sector as a whole
    • influence policy developments
    • contribute to the public accountability of higher education.
  4. We hope that performance indicators will encourage institutions to increase access, and maximise achievement for all who can benefit from higher education, by providing the information with which they can monitor their activities.
  5. Using these indicators, higher education institutions and funding councils will be able to identify good practice and disseminate it throughout the sector.
  6. Once indicators have been available for a number of years, we will be able to look at changes over time in the sector as a whole and in individual universities and colleges. Cumulative indicators should prove even more informative and of greater interest than those for any individual year.

    Indicators included

  7. The present indicators fall into four categories:
    1. Tables T1 and T2 provide access indicators, that is what proportion of entrants come from various under-represented groups.
    2. Tables T3 and T4 provide indicators of the non-continuation rates of institutions, in this case what proportion of students do not continue at an institution beyond their first year there.
    3. Table T5 gives the projected outcomes for students who are at an institution, in other words what proportion are projected to qualify at the institution, transfer to another institution, or leave higher education with no qualification. The same table provides the related measure of efficiency, the ratio of the average time it should take to obtain a qualification to the projected actual time taken.
    4. Finally, Table R1 provides indicators of research output, looking at quantitative outcomes of research which will change from year to year. These are different from the ratings of quality produced by the Research Assessment Exercise (RAE), and are designed to complement, rather than replace, them.

    Background

  8. The need to establish a common system of measuring aspects of the performance of higher education institutions was identified by the National Committee of Inquiry into Higher Education (the Dearing Committee). In November 1997, the Secretary of State for Education and Employment, and the Secretaries of State for Scotland and for Wales, asked the funding councils to discuss with government ways of developing performance indicators for the higher education sector. A steering group was set up with membership from interested government departments, the funding councils, the Higher Education Statistics Agency (HESA), and representatives of the institutions.
  9. The group began its work in March 1998. Its aim was to develop appropriate performance indicators which would recognise the diversity of the sector, and the requirements of different stakeholders with an interest in higher education. At the outset it was agreed that the indicators should make maximum use of existing data sources. This first set of indicators therefore concentrates primarily on full-time undergraduate students, for whom the existing data are most comprehensive.
  10. A wide variety of stakeholders have an interest in performance indicators. As it would not be possible to satisfy all of these initially, the group agreed to focus on the requirements of government, the funding councils, and the institutions themselves. However, the indicators will also be of interest to a wider audience.
  11. The group also agreed to concentrate on producing indicators for teaching and learning, and for research. These are the indicators produced here. Next year, the range of indicators will be extended.
  12. The first report of the steering group was published in February (HEFCE 99/11). This was followed, in May 1999, by a consultation document sent to all higher education institutions, with provisional values of the institution’s indicators. The results of this consultation exercise were evaluated by two independent consultants who advised on the best way forward. A number of changes were agreed, principally to the presentation of the indicators, and their explanation; and the proposed indicators on module completion rates were dropped for this year because of concerns raised about the data quality.
  13. Because of the diversity of the higher education sector, many of the indicators produced are complex, and need to be interpreted with care. Nevertheless, they should provide valuable information to stakeholders.

    Adjusted sector benchmarks

  14. None of these performance indicators attempts to demonstrate who or what is ‘best’. Higher education is too diverse for that.
  15. The tables enable comparisons to be made both between institutions, and between an institution and the sector. However, differences between institutions, such as subject mix or the qualifications on entry of their students, can obscure comparisons. Therefore we have established a system of adjusted sector benchmarks to help make comparisons meaningful. These benchmarks provide a sector value with which an institution’s values can be compared.
  16. Tables also include a ‘standard deviation’ which can help to assess whether the difference between an indicator and its benchmark is significant. Where this difference is not shown to be significant, or is less then 3 per cent, the value of the institution's indicator can be taken to be effectively the same as the 'adjusted sector benchmark'.
  17. We have marked with an asterisk those cases where the difference is significant. In such cases, this does not necessarily mean that the institution is performing better or worse than average: there may be some other factor at work which is not allowed for in the benchmark. But it does mean that those concerned should look closely and try and understand what is going on.

3. The indicators

  1. The performance indicators are presented here in six tables. All 175 publicly funded higher education institutions in the United Kingdom have been included. However, as most of the tables are concerned only with full-time undergraduate students who are residents of England, Scotland, Wales, or Northern Ireland, not all institutions are represented in all tables.
  2. We have not provided information this year about module completion rates, because of concerns raised over the data quality.

    Description of the indicators

  3. The following paragraphs explain briefly the indicators in each table. More detailed definitions and explanations are provided in the annexes.

    Access indicators

  4. Tables T1 and T2 provide information about the participation of groups that are under-represented in higher education relative to the population as a whole (the access indicators). These indicators have been produced separately for young and mature students, and for full-time and part-time students.
  5. There are three different access indicators for young full-time students, based on data from different sources. All are shown in Table T1. They are:
    • the percentage of students who attended a school or college in the state sector
    • the percentage whose parents’ occupation is classed as skilled manual, semi-skilled, or unskilled
    • the percentage whose home area, as denoted by its postcode, is known to have a low proportion of 18 and 19 year olds in higher education.
  6. The three indicators allow different facets of access to be measured. For most institutions, the three indicators tell a similar story, but for some institutions one value may be out of line with the others. In that case special factors, such as the locality of the institution or the nature of its courses, may be at work.
  7. For example, art schools tend to have high percentages of students from state schools, even when the percentage of parents whose occupation is skilled manual, semi-skilled, or unskilled is low. This is an unusual combination, but can be explained because further education (FE) colleges are included in the state school sector and nearly all Art students do a foundation course at an FE college before going on to a university or higher education college.
  8. Entrants are defined as mature if they are aged 21 or over at 30 September of their year of entry. Since no information is collected on the social class of the parents of these students, nor on the type of school they attended, these indicators are omitted for these students.
  9. The indicator used, in a slightly adapted form, is the percentage of mature students who come from ‘low participation’ neighbourhoods. Many mature students start a course at a higher education institution having previously obtained a higher education qualification. For example, a student may have obtained an HND, then worked for a few years and want to ‘top up’ the qualification to a degree, or even change direction and undertake a degree in a different subject. When this student was originally in higher education, he or she would have been included in the ‘access’ calculations at that time. Therefore, for mature students, we have taken the percentage of entrants who do not have a previous HE qualification and who have come from a low-participation neighbourhood as the access indicator.
  10. For part-time students, the information available is often not as comprehensive as that for full-time students. Therefore the access indicator is the same as for mature students: the percentage of entrants who have no previous HE qualification and come from a low-participation neighbourhood. This is shown in Table T2, part (b).

    Non-continuation rates

  11. The non-continuation rates for students at an institution are of widespread interest, but need to be carefully defined and interpreted. We have looked at these rates in two different ways.
  12. The first way, and the easiest to explain, is to consider what happens to a student who enters a course at an institution in a particular year, in this case in 1996-97. In the following year, 1997-98, that student may continue at the same institution, transfer to another institution, or be absent from higher education completely. A few students may obtain a degree after being on the course for just one year, for example if they have transferred from an HND course, but as the numbers are small these students have been included with those ‘continuing at the institution’.
  13. Table T3 provides the percentage of entrants who continue, transfer, or leave higher education. These figures have been produced for full-time first degree students only, but have been calculated for young and mature students separately. They have also been calculated for young students from low-participation neighbourhoods, young students from other neighbourhoods, mature students with no previous higher education qualification, and mature students with previous HE qualifications. Note that these rates take no account of progression, that is of moving from year 1 to year 2, nor of course changes within an institution. They simply look at whether or not a student is still in higher education after a year.
  14. Students may leave higher education at various times during their first year, or simply not return after the end of the year. We have found that when a student leaves very early in the academic year, there are different institutional practices about whether such students are recorded on the HESA student record. To prevent such differences affecting the indicators, we have removed all students who are recorded as leaving before 1 December in their first academic year from the figures. (Such early leavers are of interest. As soon as we have ensured that institutions are following the official guidance, and are returning records for all those who start, we will investigate if further statistics on these early leavers are needed.)
  15. Of course, some students who leave higher education after the first year will return after a year out. Table T4 includes statistics about such returns. These are not used as indicators, but provide some extra information which should be used together with the indicators in Table T3. Table T4 gives the percentage of students who were absent from HE in 1996-97 (having entered in 1995-96) who returned to higher education, either at the same institution or at another institution, in 1997-98. The final column expresses the number of such students who did not return to HE as a percentage of the entrants in 1995-96.

    Projected outcomes and efficiencies

  16. Table T5 presents another way of looking at non-completion rates. Because it is impractical to track a cohort of students completely through the HE system, because of the time involved, we have used a method of projection. We have calculated the eventual outcomes for a cohort, based on the assumption that they will move through the system in the same way as students currently in the system. This assumes, for example, that if 85 per cent of students who are in year 2 of their programme of study in 1996-97 move into year 3 of the programme of study in 1997-98, then the same proportion will move from year 2 to year 3 in the following years. This assumption that the present is a guide to the future may not always hold true, but it does allow us to produce a summary of the present movements of students as a set of eventual outcomes. We can show what proportion of students are projected to obtain a degree, leave the institution but transfer into HE elsewhere, or leave the institution without any qualification and without moving into HE elsewhere.
  17. By using the projection method we can also record the average study time for all the starters (regardless of whether they have a successful outcome or not) over the period of the projection - this is termed the average actual time. We can also calculate the average study time that might be expected for the successful outcomes reached by the starters - this is termed the average efficient time. For example if a student is awarded a degree on the third year of a programme of study we might expect that to take three years of study. If a student transfers directly into the second year of programme at another institution we might take this outcome to be worth one year of study (since they have skipped the first year at the second institution). Both the actual time and the efficient time are shown in Table T5. The ratio of the efficient time to the actual time is termed the efficiency and this is also shown on Table T5. Leaving the institution without a qualification and repeating years of programme (perhaps through exam failure or illness) are the main factors that reduce the efficiency.

    Research output

  18. The final table, Table R1, provides indicators of annual research output. These indicators are designed to complement the RAE quality ratings, not to replace them. The RAE will continue to provide the best measure of research at institutions. The indicators produced here attempt to measure the research output relative to the resources consumed, and will allow us to see how this changes from one year to the next. They are indicators based on quantitative measures, not quality. The measures used here are the number of PhDs produced, and the amount of Research Grants and Contracts obtained, by each cost centre within an institution, and the indicators are weighted to reflect the relative importance of each cost centre to the institution.

4. Summary of performance indicators for the sector

  1. These performance indicators provide information about individual institutions. However, in aggregate they also give valuable information about the nature and performance of the sector as a whole. This in turn provides a general context in which to evaluate the indicators of a particular institution.
  2. The following paragraphs summarise this sector-wide information for each of the indicators.

    Percentage from schools and colleges in the state sector

  3. The data for Chart 1, below, come from Table T1, and relate to young full-time undergraduate entrants to higher education, who apply through the Universities and Colleges Admissions Services (UCAS). The indicator is the proportion of students from state schools, that is all schools or colleges that have some state funding. (See definitions in Annex A).

    Chart 1 : Percentage of entrants from state schools

  4. Nationally, about 95 per cent of 17 year olds in full-time education attend schools or colleges in the state sector. About 82 per cent of young entrants to undergraduate courses in 1997-98 had attended such schools. Chart 1 shows that the majority of institutions take over 80 per cent of their young students from state schools, with about half taking 85 to 95 per cent from such schools.
  5. However, nearly one in six institutions take less than 70 per cent from state schools, and a few take less than half of their students from such schools.

    Percentage from social classes ‘skilled manual’, ‘semi-skilled’, and ‘unskilled’

  6. The data in Chart 2 again come from Table T1, and relate to young full-time undergraduate UCAS entrants to higher education.

    Chart 2 : Percentage of entrants from social classes IIIm to V

  7. This group forms about 50 per cent of the UK population. Nationally, 25 per cent of young entrants to HE come from this section of the population, with wide variations between institutions. Most institutions take between 20 and 40 per cent of young entrants from the group, but a few take more than 40 per cent and some take less than 15 per cent.

    Percentage from low participation neighbourhoods

  8. Data in Chart 3 are from Tables T1 and T2. They relate to young and mature full-time and part-time undergraduate entrants to higher education.

    Chart 3 : Percentage of entrants from low participation neighbourhoods

  9. We have defined low participation neighbourhoods as those where the rate of participation in higher education for young people (under 21) has been less than two thirds of the national average over the past four years. About one-third of young people live in such areas.
  10. Nationally, 12 per cent of young entrants and 14 per cent of mature entrants to full-time first degree programmes come from low participation neighbourhoods. Most institutions take between 5 and 20 per cent of both their young and mature full-time entrants from these areas. A few take less than 5 per cent of their students from such areas; about 10 institutions take more than 25 per cent of their mature entrants from such areas, with slightly fewer taking more than 25 per cent of their young entrants from such areas.
  11. As far as part-time entrants are concerned, there is a difference between young and mature entrants. 16 per cent of young entrants and 7 per cent of mature entrants to part-time undergraduate courses come from low participation neighbourhoods. Most institutions take between 5 and 25 per cent of their young part-time entrants from these areas, with only two taking less than 5 per cent. However, the number of young part-time students is generally low, so these figures need to be treated with caution. For mature entrants, most institutions take less than 10 per cent from low participation neighbourhoods, and very few take over 20 per cent. This contrasts markedly with the position for full-time students, and needs to be investigated further.

    Non-continuation rates after first year at institution

  12. The data in Chart 4 come from Table T3, and relate to young and mature full-time first degree entrants to higher education.
  13. In general, a higher proportion of mature entrants than young entrants do not continue in higher education after their first year, 15 per cent of mature compared with 8 per cent of young entrants. The non-continuation rate for young entrants is below 15 per cent at nearly all institutions, and for mature entrants it is between 5 and 20 per cent at most institutions, but over 20 per cent at about one in eight institutions.

    Chart 4 : Non-continuation after one year at institution

  14. Table T4 provides further information about students who do not continue after their first year. It relates to students who started at university or college in 1995-96, but were not in higher education in 1996-97. Nationally, about 12 per cent of such students returned to their original institution in 1997-98, with a further 10 per cent transferring to another institution. Young students are slightly more likely to return to higher education after a year out than mature students: 13 per cent of young students returned to the same institution, and 14 per cent transferred; the equivalent figures for mature students were 12 per cent and 5 per cent.

    Projected outcomes and efficiency

  15. The data for Charts 5 and 6 come from Table T5, and relate to all students starting a full-time first degree.
  16. Nationally, 80 per cent of starters are projected to obtain a degree eventually, perhaps having changed institutions en route, and a further 2 per cent to obtain a different qualification. 18 per cent are expected to discontinue their studies and not resume at any UK HEI. They are assumed to have gained no qualifications. This compares favourably with the rates in other countries. In fact, of all the countries with data held on the Organisation for Economic Co-operation and Development (OECD) database, only Hungary and Japan have lower non-completion rates. ('Education at a Glance', OECD, 1998, page 198.)
  17. At most institutions, between 70 and 90 per cent of entrants are projected to graduate at the institution where they started. There are a small number of institutions where this projected figure is less than 60 per cent. The projected percentage of students who leave before gaining any award, and who do not return to study or transfer to another institution, is less than 30 per cent for the great majority of institutions.
  18. The efficiency of an institution has been defined as the ratio of the time students should ideally take to obtain a qualification or transfer, to the projected study time taken before gaining a qualification, transferring or discontinuing. Nearly all institutions show an efficiency of over 80 per cent. Across the sector as a whole, following students through transfers to their eventual outcomes, the efficiency is 85 per cent. Such efficiency measures are not available for other countries, but it is known that UK students graduate earlier than in most other European countries. ('Education at a Glance', OECD, 1997, page 335) This is due partly to the shorter course lengths in the UK, and partly to the low incidence of repeated years, which is reflected in the high efficiency rate.

    Chart 5 : Projected outcomes of students starting a full-time degree course in 1996-97

    Chart 6 : Projected efficiencies

    Research indicators

  19. The data for Chart 7 are from Table R1. They are different in kind to the other indicators, in that they do not wholly relate to the student population.
  20. The indicators are all standardised to a value of 1 and take account of the differing ratios of output to input in different cost centres. They are based on two input measures (academic staff costs and funding council funding for research), and two outputs (number of PhDs and income from research grants and contracts). A value of 1 for an indicator shows that the institution is producing the same as the rest of the sector, relative to its input. A value below 1 shows it is producing less than the sector, and a value greater than 1 shows that it is producing more than the sector, again relative to its input.
  21. The indicator based on share of PhDs per share of academic staff costs has a value between 0 and 2 in most institutions. Only 13 institutions produce more than twice their share of PhDs compared to their share of staff costs. The largest value obtained is 3.5, from a specialist institution.
  22. When looking at funding council funding as the input, we get a similar picture, except that the values above 2 are more spread out. The largest indicator here is 10.9. The large values occur mainly for institutions whose funding council income for research is very small, and so even a few PhDs can represent more than their share of this funding.
  23. For research grants and contracts, the indicator values are widely spread. The share of research grants and contracts per share of academic staff costs is again under 2 for most institutions, but there are a few who have values greater than 4. These are specialist institutions who are likely to be the main player in their field.
  24. The indicator of research grants and contracts per share of funding council funding has the widest range of values of all these research indicators. The largest values tend to occur in institutions which have a very low share of the research funding from the funding councils. If these institutions obtain research grants or contracts, then these can form a large share of the sector figure, producing large values for the indicator.

    Chart 7 : Research indicators

5. How to use the tables

  1. The performance indicators cover access, non-continuation rates, projected outcomes and efficiencies, and research output. The higher education institutions in the UK are very diverse, and the range of indicators reflects this diversity. Some of the factors which make up this diversity have been taken into account in producing ‘adjusted sector benchmarks’ (see below) which are included in most of the tables. The benchmarks enable a comparison to be made between an institution’s indicator and a comparator which allows for differences between institutions in entry qualifications and subjects of study. In addition, many of the indicators are provided separately for young and mature students. Although there will be situations where it may be appropriate to compare an indicator with the actual figure for the rest of the UK, in most cases it is more meaningful for comparison to be made with the benchmark.
  2. In looking at differences between institutions, small differences between indicators should generally be ignored. What is difficult to determine is when a difference stops being ‘small’. The same is true when comparing an indicator and its benchmark; small differences can be ignored, but we must define small. We have dealt with this problem using a combination of methods, which are outlined below, and given in greater detail in Annex B.
  3. Definitions used throughout these tables are provided in Annex A. Each table is preceded by a brief explanation of what it contains.

    Adjusted sector benchmarks

  4. An institution’s indicators can be compared with a benchmark which makes allowance for both the subject mix of the institution and the qualifications on entry of its students.
  5. These benchmarks are used for the indicators relating to access and to progression, outcomes and learning efficiency. The social profile of students varies across subjects so that, for example, entrants to courses in medicine include only 13 per cent with parents with skilled manual, semi-skilled and unskilled occupations, compared with 25 per cent of entrants to the sector as a whole. The social profile of students with differing entry qualifications is also different. The same factors are also associated with the various measures of success, that is progression, learning outcomes and efficiency.
  6. These are not the only characteristics that affect how students perform, but they do account for a large proportion of the differences between institutions. Without a benchmark of some sort, comparisons would have to be made either with the overall sector value, or directly with other institutions, with no allowance made for any factors that might have a bearing on results. This benchmark is the value that the whole UK sector would have if it had the same subject and entry qualification profile as the institution. More complex statistical methods could be used to produce benchmarks, but the approach used here balances the needs for transparency and for rigour.
  7. The benchmark has a second purpose. Institutions whose benchmarks are very different should not be compared, as they will have quite different characteristics. Two institutions with similar benchmarks will not necessarily be similar, but the benchmarks, in conjunction with other knowledge, can provide a first indication of whether it makes sense to compare particular institutions.
  8. The method of calculating the adjusted sector benchmark is the same for all indicators produced. It is described in detail in Annex B.

    Significant differences

  9. To guide users of these indicators, we have provided in the tables a measure known as the ‘standard deviation’, which can answer the question of whether or not the difference between an indicator and its benchmark is small. Taking this into account, we have also indicated, by an asterisk, when this difference may be considered significant. In general, if there is no asterisk against an indicator, it can be taken that the difference between the indicator and its benchmark is not large enough to be important. Full details of the assumptions made in obtaining the standard deviation, and how differences have been designated significant, are provided in Annex B.
  10. A significant difference between an indicator and its benchmark suggests that the institution is doing either very well or very badly with respect to that indicator. Institutions that are doing ‘badly’ may like to investigate why this large difference exists; there may be factors which the benchmark does not take into account, or there may be some areas within the institution that are performing less well than others. Institutions that are performing ‘well’ may be able to look at their performance and provide pointers to good practice. The funding councils will make data available to institutions who wish to investigate these factors further.

    How to interpret the indicators

  11. The higher education sector is very diverse, and this creates a problem for those who would like just one measure of what is ‘best’. The indicators in this report are designed to be taken together, and even so do not cover all facets of the sector. This should be borne in mind.
  12. Care should be taken to ensure that two institutions are alike enough to compare. There is no point, in the extreme case, in trying to compare a small specialist College of Art and Design with a large multi-faculty university. However, there are less extreme cases where comparison is still not meaningful. To help decide if two institutions can be compared, we have included adjusted sector benchmarks in each table, as explained in paragraphs 67-70. In general, if two institutions have very different benchmarks, they should not be compared.
  13. The indicators apply to various sub-groups of the student population, such as young full-time degree students, or mature part-time students. By splitting the population in this way, there is a danger that the numbers on which a particular indicator is based may be rather small. In such cases, interpretation is particularly difficult, as changes to the characteristics of only one or two students can lead to large swings in the value of the indicator. For example, if there are only 50 students in the base population, a change in status of only two of the students would change the indicator by 4 per cent. The indicator values have been included even for small base numbers, but care should be taken in their interpretation.

    A simple example

  14. The following example of two institutions, X and Y, may help to interpret the tables. The two institutions have very different characteristics, and as such they should not be directly compared. However, they are similar in the number of undergraduate students they admit each year. University X is a highly selective institution with a major interest in research. It takes relatively few students from disadvantaged backgrounds, but is excellent at retaining those students it recruits. Half of its funding council income comes from research, and its research output is relatively high. University Y takes in a high proportion of students from disadvantaged backgrounds, partly because of where it is located. Its retention rates are not as good as those of university X. It has very little research funding from the funding council.
  15. The information about the participation of under-represented groups is taken from Tables T1 and T2. Information about young full-time students comes from Table T1, for mature full-time and all part-time students from Table T2. School type and social class background are only used for young students, as school type for mature students is likely to be irrelevant, and social class background is defined in a different way. The definition of low participation neighbourhoods is based on the participation rates in higher education of young people. However, once an area has been classified as low participation, the postcode of any student, young or mature, can be used to identify whether or not he or she comes from a low participation neighbourhood.

    Table E1 : Percentage from state schools, social classes IIIm to V, and low participation neighbourhoods

     

    From state schools, FE colleges, etc

    From social classes IIIm - V

    From low participation neighbourhoods

    Institution

    Percent from group

    Benchmark

    Percent from group

    Benchmark

    Percent from group

    Benchmark
     

    %

    %

    %

    %

    %

    %

    University X

    52

    62

    8

    11

    4

    6

    University Y

    93

    87

    37

    30

    21

    14
  16. The first point to note is that the adjusted sector benchmarks are quite different for the two institutions. We should therefore not be trying to compare them. They are different types of institution, with different missions, and this is brought out by the benchmarks.
  17. Looking at the three indicators for young entrants to the two institutions, given in Table E1, it is clear that university X takes far fewer students from state schools, from social classes IIIM to V, and from low participation neighbourhoods than university Y. Even with no knowledge of which institutions they are, it can be seen that X appears to be highly selective, while Y seems to be an institution from a disadvantaged area where it takes in many local students. Comparing the proportions from each group with the adjusted sector benchmarks, which are themselves much lower for university X than for university Y, university X takes in far fewer students from these groups than the sector as a whole, even allowing for entry qualifications and subjects studied. University Y takes in considerably more of such students than the sector as a whole.
  18. It is also important to note that the three indicators tell the same story for each institution: that is, the difference between the indicator and the benchmark is negative for all three indicators for institution X, and positive for all three indicators for institution Y. This will generally be true. Where the differences are a mixture of positive and negative, particularly if the differences are relatively large, then the factors leading to this will need to be looked into.

    Table E2 : Percentage from low participation neighbourhoods

    (a) Full-time undergraduates
     

    Young entrants

    Mature entrants

    Institution

    Percent from group

    Benchmark

    Percent from group

    Benchmark
     

    %

    %

    %

    %

    University X

    4

    6

    6

    7

    University Y

    21

    14

    29

    18

    (b) Part-time undergraduates
     

    Young entrants

    Mature entrants

    Institution

    Percent from group

    Benchmark

    Percent from group

    Benchmark
     

    %

    %

    %

    %

    University X

    7

    17

    2

    3

    University Y

    23

    20

    11

    8
  19. Table E2 repeats the low participation figures from Table E1, in order to allow a comparison of different types of entrant. For full-time mature entrants, university X takes in about what would be expected from these areas, while university Y again takes in more than expected. For mature part-time entrants, both institutions follow a similar pattern, although the difference between university Y’s indicator and its benchmark is not so large in this case. Neither institution has a large number of young part-time entrants, so the differences between the indicators and the adjusted sector benchmarks do not tell us very much. The significance of the differences in both cases is low because of the small numbers involved.
  20. Non-continuation rates after the first year at the institution are taken from Tables T3 and T4. The percentage not found shows what proportion of 1996-97 entrants to the institution were not at any HEI in 1997-98. The percentage still not studying after a year out shows what proportion of the 1995-96 entrants who were not found at any HEI in 1996-97 were still not in higher education in 1997-98. Note that the total number of entrants is not the same in Table T3 as in Tables T1 and T2; Table T3 relates to 1996-97, and Tables T1 and T2 to 1997-98.

    Table E3 : Non-continuation after one year

     

    Young entrants

    Mature entrants

    Leavers from 1995-96 still absent from HE in 1997-98

    Institution

    Not continuing

    Benchmark

    Not continuing

    Benchmark

    Young entrants

    Mature entrants
     

    %

    %

    %

    %

    %

    %

    University X

    1

    2

    4

    6

    24

    57

    University Y

    10

    11

    13

    16

    86

    93
  21. Looking at the non-continuation rates given in Table E3, the effect of university X’s selectivity is obvious. Only 1 per cent of its young entrants, and 4 per cent of its mature entrants, do not continue in higher education in the year after entry. In addition, if we look at the proportion of those who left after a year who are still not in HE after a year out, only a quarter of the young entrants and half of the mature entrants who originally left are still not at an HE institution. In other words, a large number of the non-continuers have actually returned to higher education after one year out.
  22. University Y, on the other hand, has a non-continuation rate after one year of 10 per cent for young entrants and 13 per cent for mature entrants - very much greater than the rates for university X. However, the rates are still below the adjusted sector benchmarks. It is actually doing rather well, considering its student intake and subject mix. Most of those who leave university Y do not return to higher education after a year.

    Table E4 : Projected outcomes

     

    Degree

    Neither award nor transfer

    Efficiency

    Institution

    Projected

    Benchmark

    Projected

    Benchmark

    Projected

    Benchmark
     

    %

    %

    %

    %

    %

    %

    University X

    96

    88

    1

    7

    98

    91

    University Y

    65

    70

    26

    22

    73

    80
  23. The projected outcomes and efficiency, which come from Table T5 and are shown here in Table E4, in general reinforce the above picture. As this table does not take the age of students into account, university Y, with a higher percentage of mature students than university X, will look worse against its adjusted sector benchmark than university X. While only 1 per cent of university X’s students are projected to leave without a qualification, compared to its adjusted sector value of 7 per cent, over a quarter of university Y’s students are projected to leave without a qualification, compared to the adjusted sector value of 22 per cent.
  24. Efficiency is defined in terms of how long students spend at an institution compared with how long they would spend to qualify or transfer if they did not repeat years or change courses, or leave higher education completely. (For this purpose, students who transfer to other institutions are considered to have succeeded if, for example, they transfer from year 1 in the first institution to year 2 in the second.) University Y, with a value of 73 per cent is not as efficient as university X, whose value is 98 per cent. This is not unexpected, given its higher non-continuation rate,. The adjusted sector benchmarks, which are 91 per cent for university X and 80 per cent for university Y, suggest that differences in the entry qualifications to, and subjects studied at, each university account for most of this difference in efficiency.

    Table E5 : Research indicators

    Institution

    Share of PhDs awarded per academic staff costs

    Share of PhDs awarded per research funding

    Total research funding from funding council

    Percent research funding from funding council
         

    £

    %

    University X

    2.27

    1.07

    10,200,000

    56

    University Y

    0.13

    4.50

    50,000

    1
  25. The final set of information on research indicators, two of which are shown in Table E5, comes from Table R1. It is clear from the final column of Table E5 that university X, with half of its funding council allocation being for research, is a major research university; while university Y has very little in the way of research funds from the funding council. Although both universities have similar numbers of undergraduate students, the academic staff costs at X are likely to be greater than those at Y because more of the staff time at X will be spent on research. Nevertheless, the staff costs will generally be closer than their research allocations for the two institutions. It is not surprising therefore that the share of PhDs for university X is twice as large as its share of academic staff costs, while it is only about the same as its share of the funding council allocation for research. University Y produces very few PhD students compared with its academic staff costs, but very much more than its share of the funding council allocation for research would suggest.
[ Go to: Contents | Tables | Annexes ]