Report 99/66

December 1999

Performance Indicators in Higher Education in the UK: Annexes

Annex A

General definitions

1. This annex provides definitions of the terms used in the performance indicator tables. The data used in constructing the indicators have been taken mainly from the Higher Education Statistics Agency (HESA) database, with some information from the database held by the Universities and Colleges Admissions Services (UCAS). In general, where information is available through HESA that database has been used. Some information is not currently held on the HESA database, and some institutions do not provide complete information to HESA. In those cases, data will be obtained from UCAS. More technical definitions, explaining which fields from the HESA and UCAS databases have been used, are provided in the electronic version of this document. Full details of fields included in HESA returns are available on their web-site,

   Coverage of tables

2. All students included in tables T1 to T5 are those whose normal residence is in the United Kingdom, excluding the Channel Isles and the Isle of Man. This information comes from the two HESA fields DOMICILE and POSTCODE. If there is valid information in only one of these fields, then that field is used. Otherwise, both fields are mapped to a country of residence; if these contradict each other, preference is given to the postcode field if both countries are within the United Kingdom, otherwise the DOMICILE field is used. Full details are included in the electronic version of this publication.

   Age

3. Many of the tables are split between young and mature students, defined as follows :
   1. Young students are those who enter an institution when they are aged less than 21; specifically, they must be under 21 at 30 September of the academic year in which they first enter the institution. So for students entering an institution in 1997-98, young students are those born after 30 September 1976.
   2. Mature students are those who are 21 or over, also at 30 September of the academic year in which they first enter the institution.
4. Note that students whose date of birth is not given, or whose date of birth suggests that they are 10 years or under, are not allocated to either of the age groups. For tables which provide information about young students, mature students, and all students, this means that the numbers under ‘All students’ are not necessarily the sum of ‘Young students’ and ‘Mature students’.
5. The above definitions are used to tie in with the UCAS definitions. Social class data (see below), in particular, are collected differently for young and mature students on the UCAS forms.

   Mode of study

6. Full-time students are those recorded as studying full-time at an institution, provided that the expected length of their course is at least 24 weeks. If the length of course is unknown, it is assumed to be 24 weeks or longer. Students on sandwich courses (thick or thin) are also included as full-time.
7. Part-time students are those not studying full-time.
8. Further details are available in the electronic version of this publication.

   Level of study

9. The level of study is taken from the qualification aim of the student. Only undergraduate students are considered at present. Degree students are those studying for any type of first degree; other undergraduates are those studying for diplomas, certificates, and other undergraduate courses. The qualification aims (HESA field QUALAIM, as coded for the 1997-98 return) with the following codes are used:

   	QUALAIM codes
   Degree	20, 21, 22, 23, 24
   Other undergraduate	25, 29, 30, 31 32, 41, 42, 43, 44, 51, 52, 61, 97

   Type of School

10. School type is one of the fields taken wholly from UCAS records. The field records school type on application. Only schools or colleges denoted independent are taken as non-state schools. This means that students from sixth form or further education colleges are included as being from state schools.

    Social Classes

11. This is the second field obtained from UCAS records. It is based on parental occupations, which are classified using the Standard Occupational Classification. The social classifications used are
    1. Social class I - Professional
    2. Social class II - Intermediate
    3. Social class III N - Skilled non-manual
    4. Social class III M - Skilled manual
    5. Social class IV - Semi-skilled
    6. Social class V - Unskilled

    The performance indicator is based on the proportion from social classes IIIM, IV and V, that is the proportions whose parents are in skilled manual, semi-skilled or unskilled occupations.

    Low participation neighbourhoods

12. This definition uses work carried out into the participation rates of young people in different areas of the country. Using postcodes, 160 different neighbourhood types (clusters) have been defined, and the proportions of young people (aged under 21) in each of these clusters who entered higher education in the academic years 1995-96, 1996-97, and 1997-98 have been calculated. Areas for which this participation rate was less than two thirds of the UK average over the whole period were defined as low participation neighbourhoods. It is intended that this method of defining areas as low participation will continue until the results from the next census become available for a new geodemographic analysis.
13. Students have been allocated to clusters on the basis of their postcode, using a commercial software package, Super Profiles, available through Claritas. This is one field where data from the UCAS file may be used if the HESA information is incomplete. If neither record for a student contains a complete valid postcode, then that student will not be included in either a ‘low participation neighbourhood’ or an ‘other neighbourhood’. Such a student will be included, where appropriate, in the total column. There are a few postcodes which have not been allocated to any of the clusters. Students from such areas will be treated in the same way as those whose postcodes are not valid.

    Entrants / Starters

14. Tables T1 to T4 provide information for entrants to an institution. These are defined as students who are returned as commencing a programme of study at that institution during the academic year of interest. This is based on the date of commencement of the student’s study (HESA field COMDATE). While most entrants go into the first year of a programme of study, some will start on the second, or later, year of programme, for example if they transfer from another institution. This definition is not the same as that used in HESA publications of first year students, which currently are defined by field YEARSTU. However, from 1998-99 HESA will use COMDATE to define such students.
15. Entrants who are recorded as leaving before 1 December have not been included in the calculations. When a student leaves very early in the academic year, there are different institutional practices about whether such students are recorded on the HESA student record, and excluding such students from the figures prevents such differences affecting the indicators.
16. Table T5 is based on starters at an institution. The concept of a starter has been used to be consistent with the method of projecting outcomes. Students at a particular institution are defined as starters if they are full-time degree students who have not been studying at that institution full-time for a degree in either of the two years prior to the academic year of interest. This means that students who were on a diploma course in the previous year at the current institution, and have transferred to a degree course in the current year, will count as starters (unless they had spent the year two years previously on a degree course there). Such students will not necessarily count as entrants. On the other hand, a student who has spent one year at an institution on a degree course, then spends a year out of studies but comes back onto a different degree course, will not be counted as a starter, but may be counted as an entrant.

    HESA statistics

17. Though the tables are largely based on data collected by HESA, the statistics differ from those that have been published by HESA in a number of respects. So, for example, the number of home first degree full-time entrants (table T1a) is shown as 290,542, whereas the HESA Reference volume, 'Students in Higher Education Institutions (1997-98)' reports the number of UK domiciled first degree first year full-time students as 290,449 (table 1b). The differences are as follows:-
    1. This publication uses HESA records that have been corrected by institutions following HEFCE monitoring exercises (HEFCE Circular 8/97, HEFCE Request 97/30 and HEFCE Request 99/03), as well as from the specific consultation about performance indicators.
    2. Where postcode or entry qualification data are missing or incomplete, data have been taken from files supplied by UCAS.
    3. HESA identified first year students by the 'year of student field'; tables 1 to 4 make use of the date of commencement of studies.
    4. The HESA data includes only those students active on the 1 December 1997; these tables also include students who entered after this date.
    5. The HESA definition of full-time includes only courses of more than 18 weeks, these tables include only courses of 24 weeks or more.
    6. The HESA reference date for age is 31 August, for these tables the reference date is the 30 September.
    7. The 'UK domiciled' students reported by HESA include students domiciled in the Channel Islands and the Isle of Man. These tables exclude these students.

Annex B

Adjusted sector benchmarks - technical notes and more detailed information

1. This annex contains details of the subject and entry qualifications breakdown used to obtain the adjusted sector benchmarks; tables showing both the numbers of students in each category and the proportion of students in each category with different characteristics; and technical details and assumptions made in producing both the adjusted sector benchmarks and the associated standard deviations.

   Adjusted sector benchmarks

2. The annex starts with a simple example to explain how the adjusted sector benchmark is calculated.

   A simple example

3. The adjusted sector benchmarks make allowance for the subject mix of an institution, and the entry qualifications of its students. In order to demonstrate the mechanism, this example uses 4 subjects and 5 entry qualifications. The indicator used in the example is the non-continuation rate, the proportion of entrants in one year that are not in higher education in the next year. Similar methods are used in calculating the adjusted sector benchmarks for the other indicators, (except for the indicators of projected outcomes and efficiency, which are dealt with in paragraph 22). Note that where an indicator is provided for different groups of students, for example young students from low participation neighbourhoods, or young students from other neighbourhoods, each adjusted sector benchmark would be calculated on the basis of the number of that type of student in the sector as a whole.
4. Suppose there are 100,000 students in the sector, and that they are split across four different subject categories, and with five groups of entry qualifications. The split might look like that given in table B1.

   Table B1 : Distribution of students across the sector (fictitious data)

   Entry Qualifications	Subject
   	A	B	C	D	TOTAL
   I	8,000	8,000	1,000	3,000	20,000
   II	6,000	8,000	3,000	3,000	20,000
   III	3,000	6,000	8,000	3,000	20,000
   IV	2,000	6,000	8,000	4,000	20,000
   V	1,000	2,000	5,000	12,000	20,000
   TOTAL	20,000	30,000	25,000	25,000	100,000

5. The non-continuation rates in the different categories across the sector are shown in table B2. These suggest that, across the sector as a whole, non-continuation rates are relatively high for subject C, and high among students in entry qualification category V. The overall non-continuation rate for the sector is 8%.

   Table B2 : Non-continuation rates for sector (fictitious data)

   Entry Qualifications	Subject
   	A	B	C	D
   I	2.5%	5.0%	10.0%	3.3%
   II	5.0%	5.6%	10.0%	5.0%
   III	5.0%	6.7%	12.5%	6.7%
   IV	10.0%	7.5%	15.0%	6.3%
   V	15.0%	15.0%	18.0%	6.7%

6. Suppose there are two institutions, each with 2000 students, but split rather differently among subjects and entry qualification groups. University X covers all subject areas, but has no students with entry qualifications V. University Y has students with all entry qualifications, but does not offer subject area A. The actual split is shown in table B3.

   Table B3 : Distribution of students at universities X and Y

   University X						University Y
   Entry Quals	Subject					Entry Quals	Subject
   	A	B	C	D	TOTAL		A	B	C	D	TOTAL
   I	300	300	100	100	800	I	0	50	100	50	200
   II	100	100	0	100	300	II	0	40	80	80	200
   III	200	100	50	50	400	III	0	200	200	100	500
   IV	0	300	50	150	500	IV	0	100	200	200	500
   V	0	0	0	0	0	V	0	210	220	170	600
   TOTAL	600	800	200	400	2,000	TOTAL	0	600	800	600	2,000

7. Of students at institution X, 150 did not continue after their first year, and of those at institution Y, 250 did not continue. So the percentage not continuing after their first year at the institution is 7.5% for institution X, and 12.5% for institution Y. On the face of it, this means institution X is doing better than the sector average of 8% not continuing, while institution Y is doing considerably worse.
8. Now look at the adjusted sector benchmark for each institution. This is calculated by taking the number of students in an institution who are in each subject / entry qualification category, and working out how many of these would not continue if the sector value in that category were to apply. For entry qualification I and subject A, the sector rate of non-continuation is 2.5%, and so university X, with 300 students in that category, would expect 2.5% of 300 = 7.5 students to leave after their first year. Similarly, for entry qualification V and subject C, where the sector rate is 18%, university Y would expect 18% of its 220 students, i.e. 39.6 students, to leave after a year. Table B4 shows the numbers that would be expected to leave for each institution in each category.

   Table B4 : Applying sector rates to institutions X and Y - numbers not continuing

   University X					University Y
   Entry Quals	Subject				Entry Quals	Subject
   	A	B	C	D		A	B	C	D
   I	7.5	15	10	3.3	I	0	2.5	10	1.7
   II	5	5.6	0	5	II	0	2.3	8	4
   III	10	6.7	6.3	3.3	III	0	13.3	25	6.7
   IV	0	22.5	7.5	9.4	IV	0	7.5	30	12.5
   V	0	0	0	0	V	0	31.5	39.6	11.3

9. At institution X, the total number of students who would not be expected to continue beyond their first year would be the sum of all these values, namely

   7.5+15+10+3.3 + 5+5.6+0+5 + 10+6.7+6.3+3.3 + 0+22.5+7.5+9.4 = 117

   The adjusted sector benchmark would therefore be 117 / 2000 = 5.9%

10. Similarly for institution Y, the numbers who would not continue if the sector values applied would be

    2.5+10+1.7 + 2.3+8+4 + 13.3+25+6.7 + 7.5+30+12.5 + 31.5+39.6+11.3 = 206

    This gives a total of 206 students not continuing, and an adjusted sector benchmark of

    206 / 2000 = 10.3%

11. Comparing the institutions’ own values to the adjusted sector benchmarks, we find that institution X now performs worse than might be expected. Looking at the profile of its students, its non-continuation rates should be lower than the sector average because its students come mainly from categories with low rates. Institution Y still does worse than expected, but not by as much as it seemed at first; again, many of its students are from categories where the non-continuation rate is high across the sector.
12. In addition, the fact that the two adjusted sector benchmarks are so different means that these two institutions are not really comparable. We should not make a direct comparison between the two for this particular indicator.

    Technical notes

13. In order to calculate the adjusted sector benchmark for a particular indicator and institution, the breakdown of the institution’s students by subject and entry qualification is required. However, the ‘sector population’ is not the same in all cases. Each indicator relates to a specific sub-set of the institution’s students, for example young full-time first degree students, or mature part-time undergraduates. The adjusted sector benchmark is based on the equivalent sub-set of the sector population.
14. We have used 13 subject groups and 21 entry qualification groups, which together provide 273 categories on which to base the adjusted sector benchmark. Assume they are numbered from 1 to 273. Let the number of students from the sector in category i be N_i, and in the institution be n_i, so that N = SUM(N_i) is the total number of students in the sector, and n = SUM(n_i) is the total number of students at the institution, where summation is over all categories. Let P_i be the proportion of students in the sector who are in category i who have the characteristic of interest, e.g. are from state schools, or have left HE after a year.
15. If the proportion of students with the characteristic at the institution in each subject/entry qualification category was the same as in each category in the sector, then the overall proportion with the characteristic would be

    [SUM(P_i * n_i )] / n

16. This is what we have called the ‘adjusted sector benchmark’.
17. Another way of interpreting this is to say it is the value that the sector average would be if the sector students were split across the 273 subject/entry qualification categories in the same proportions as the institution.

    Standard deviations

18. In general, small differences between an indicator and its benchmark are not very important. However, it is not always obvious what constitutes a small difference. We have therefore calculated the standard deviations that would be expected under certain simplifying assumptions.
19. We assume that the numbers of students in the sector in each subject/entry qualification category are fixed, and similarly that the numbers in each category at an institution are fixed. The students at a particular institution can then be thought of as a stratified random sample from the sector, taking the subject/entry qualification categories as strata. This means we are assuming that students within a category are selected at random from that category, and independently of students from the other categories. In statistical terms, we assume that the students at an institution form a stratified random sample of students in the sector, where the strata used are the subject/entry qualification categories.
20. Using these assumptions, the variance of the indicator can be found as

    var (PI) = SUM(n_i/n)² * ((1-f_i)/n_i) * P_i * (1 - P_i)
    where f_i = n_i / N_i

    The square root of this value, which is the standard deviation of the indicator, can then be used to test whether the difference between the indicator and its benchmark is small or not. A difference that is less than twice the size of the standard deviation can certainly be said to be small. In the tables, we have marked those institutions where the difference is greater than three times the standard deviation and is greater than three per cent. This is to draw attention to areas where the difference is not only statistically significant, under our assumptions, but is also of material importance.

21. If an institution is marked in this way, it should be taken as an invitation to investigate possible causes for the differences that have arisen, whether they provide an indicator that is better than the benchmark, or worse than the benchmark. Where the difference is not marked, then this implies that the indicator is either within the range that would be expected given random fluctuations, or is less than 3% away from the benchmark.

    Projected outcomes

22. The adjusted sector benchmarks for the projected outcomes indicators are obtained by adjusting the transition matrix rather than the actual indicators. The adjusted transition matrix of sector data is produced by weighting each student in the sector to give a distribution over the entry qualifications and subjects the same as for the institution. This weighting represents an approximation, since the same weighting is used for all progression paths. The effect is likely to be small. (Further details are provided in Annex C, paragraph 25.)
23. Because the adjustment is made to the transition matrix and not directly to the outcomes, the standard deviations have been obtained by assuming students have been selected at random from the outcome categories. Again, this is a simplification, but appears to give realistic results. But if the pattern of projected outcomes for an institution is very different from the adjusted sector outcomes, then the standard deviations cannot be relied on, and have been omitted from the table. The electronic version of the publication will include further details of how the difference in the pattern of outcomes has been identified, and the standard deviations of the outcome and efficiency measures.

    Subject and entry qualifications breakdown for the sector

24. The adjusted sector benchmarks have been produced by adjusting for subject studied and entry qualifications according to the proportions of students in each group within each institution. Subjects studied have been combined into 13 groups, while entry qualifications have been combined into 21 groups. The actual categories are described below.
25. The assumption behind the adjusted sector calculations is that each category defined here contains students who form a homogeneous group. Further work is being done to see how valid this assumption is, and whether there are other factors that might make a significant contribution to a benchmark.

    Subject categories

26. The subject categories have been obtained from the subject of qualification aim, HESA field SBJQA. Table B5 shows the codes used for the subject categories, and the subjects they represent. These differ from the categories normally used by HESA in that several have been merged, for example Biological Sciences with Physical Sciences, and Mathematics with Computing. Two categories have only been merged if the characteristics of the students in them appear to be broadly similar.
27. Note that any combined studies programme whose subjects lie completely within one category have been included in that category. So for example a student on a joint degree programme studying Mathematics and Computing will be included in category E.
28. The third column of the table shows the HESA subject codes which the category includes. The HESA subject codes consist of a letter followed by a number, and if all codes beginning with a particular letter are in the category, only the letter has been given. Any invalid subject codes, or combinations not wholly within one category, have been allocated to the combined studies category.

    Table B5 : Subject categories and their equivalent subject codes

    	Subject description	HESA subject codes
    A	Medicine, Dentistry, and Veterinary science	A, D1
    B	Subjects allied to medicine	B
    C	Biological sciences, and Physical sciences	C, F
    D	Agriculture and related subjects	D, except D1
    E	Mathematical sciences and Computer science	G
    F	Engineering and Technology	H, J
    G	Architecture, Building, and Planning	K
    H	Social studies and Law	L, M
    I	Business & administrative studies and Librarianship & information science	N, P
    J	Languages and Humanities	Q, R, T, V
    K	Creative arts and Design	W
    L	Education	X
    M	Combined subjects	Y

    Entry Qualifications

29. The majority of students in the UK still enter higher education with either A-levels or Scottish Highers, and this is recognised in the following groupings. As with subjects, the groupings have been chosen so that as far as possible the students within each group are relatively homogeneous.
30. Both A-level grades and Scottish Highers grades are converted to a number of points. In deciding on the category to which a student should be allocated, A-levels and Scottish Highers have been treated as equivalent as far as allocated points are concerned, so that a student with 20 A-level points is allocated to the same category as a student with 20 Scottish Higher points. If a student has both Scottish Highers and A-levels, the number of points used is the greater of the two. (Note that UCAS has recently announced a revision of the points scheme for particular levels of school qualifications and this may lead to a revision of the points used in due course.)
31. A few students now enter higher education with a Baccalaureate. It has been decided to allocate these students to the same category as students with A-level or Scottish Highers having up to 4 points. This may be reviewed in future.
32. Access and foundation courses have been given a separate category, as have BTEC and related qualifications.
33. Students’ entry qualifications have been taken from the HESA database if possible, but if they are not available from HESA then the UCAS database has been used. The HESA fields QUALENT2, ALEVPTS, and HIGHPTS have been used from HESA to determine the category (or the equivalent fields from UCAS).
34. Table B6 shows the categories used, their descriptions, and the values of the QUALENT2 field used in the definition. The values of QUALENT2 used in defining A-levels or Highers are : 31, 32, 33, 34, 35, 36, 39, 40.

    Tables

35. Tables B7 and B8 show the actual numbers of full-time degree entrants to higher education across the sector in each of the defined categories in 1997-98; B7 shows the number of young entrants, and B8 the number of mature entrants. Tables B9 to B14 provide the proportions in each category with various characteristics, again for the whole sector, split into young and mature entrants. Table B9 shows the proportion of young entrants in 1997-98 who come from state schools in each category, and Table B10 shows the proportions from social classes IIIM - V. Tables B11 and B12 give the proportions of 1997-98 entrants from low participation neighbourhoods, for young and mature respectively. Tables B13 and B14 show the proportion of 1996-97 entrants in each category who did not continue in higher education after their first year, for young and mature entrants.

    Table B6 : Entry qualifications and codes used in their definition

    	Entry qualification	Qualent2 code
    A pts NK	A-levels / AS levels / Scottish Highers, number of points unknown	31, 32, 33, 34, 35, 36, 39, 40
    A pts 4	A-levels / AS levels / Scottish Highers with 1 to 4 points, or Baccalaureate	31, 32, 33, 34, 35, 36, 39, 40, 42, 46
    A pts 8	A-levels / AS levels / Scottish Highers with 5 to 8 points	31, 32, 33, 34, 35, 36, 39, 40
    A pts 10	A-levels / AS levels / Scottish Highers with 9 to10 points	31, 32, 33, 34, 35, 36, 39, 40
    A pts 12	A-levels / AS levels / Scottish Highers with 11 to12 points	31, 32, 33, 34, 35, 36, 39, 40
    A pts 14	A-levels / AS levels / Scottish Highers with 13 or 14 points	31, 32, 33, 34, 35, 36, 39, 40
    A pts 16	A-levels / AS levels / Scottish Highers with 15 or 16 points	31, 32, 33, 34, 35, 36, 39, 40
    A pts 18	A-levels / AS levels / Scottish Highers with 17 or 18 points	31, 32, 33, 34, 35, 36, 39, 40
    A pts 20	A-levels / AS levels / Scottish Highers with 19 or 20 points	31, 32, 33, 34, 35, 36, 39, 40
    A pts 22	A-levels / AS levels / Scottish Highers with 21 or 22 points	31, 32, 33, 34, 35, 36, 39, 40
    A pts 24	A-levels / AS levels / Scottish Highers with 23 or 24 points	31, 32, 33, 34, 35, 36, 39, 40
    A pts 26	A-levels / AS levels / Scottish Highers with 25 or 26 points	31, 32, 33, 34, 35, 36, 39, 40
    A pts 28	A-levels / AS levels / Scottish Highers with 27 or 28 points	31, 32, 33, 34, 35, 36, 39, 40
    A pts 30	A-levels / AS levels / Scottish Highers with 29 or 30 points	31, 32, 33, 34, 35, 36, 39, 40
    ACCFND	Foundation or Access course	43, 44, 45
    GNVQ3	GNVQ or equivalent, level 3	37, 38
    BTEC/ONC	BTEC, ONC, SCOTVEC or equivalent	41
    HE	Higher education qualification	1, 2, 3, 11, 12, 13, 14, 15, 16, 21, 22, 23, 24, 25, 26, 27, 28
    NONE	No previous qualification	92, 93, 98
    OTHERS	Other qualifications not given elsewhere	51, 52, 53, 54, 55, 61, 62, 63
    UNKNOWN	Unknown qualification	99

    Table B7 : Young full time first degree entrants 1997-98 by subject and entry qualification

    Table B8 : Mature full time first degree entrants 1997-98 by subject and entry qualification

    Table B9 : Percentage from state schools or colleges

    Table B10 : Percentage from Social Class IIIM, IV, V

    Table B11 : Percentage from low participation neighbourhoods: Young full time first degree entrants 1997-98

    Table B12 : Percentage from low participation neighbourhoods: Mature full time first degree entrants 1997-98

    Table B13 : Non continuation following year of entry: Young full time first degree entrants 1997-98

    Table B14 : Non continuation following year of entry: Mature full time first degree entrants 1997-98

Annex C

Calculation of projected outcomes and efficiencies - technical notes

1. The projected outcomes and learning efficiencies were produced using a census cohort approach. The method involves identifying a group of starting students known as a cohort. Their future progression outcomes, whether they qualify, or discontinue etc., are then projected based on current progression patterns at their institution.
2. This annex provides details of the method used, gives a simple example to illustrate the calculations involved, and shows the complete transition matrix for the sector, of the type used in producing table T5. For each institution, a similar matrix was produced.

   Projecting outcomes

3. The progression behaviour of all the students at an institution is summarised into a matrix called the transition matrix. The rows of this matrix are possible starting ‘states’, the columns possible end states. The values in the cells of each row are the proportion of students who are in the starting state who make the transition to the end state. One starting state (row) might be ‘full-time first degree year of programme one’, a possible end state (column) could be ‘part-time first degree at the same institution’; the value at their intersection would be the proportion of students who are on the year of programme 1 of a full-time first degree who then move to a part-time first degree. By defining states such as recently absent students and those moving to other modes or levels of study we can allow for students returning to study or being awarded a degree from part-time study, for example.
4. It is assumed that the probability of moving from one state to another is the same, regardless of how the state has been reached. So for example, this means that the probability that a student who has gone from first year to second year to third year, and is now in the third year of a degree has the same probability of qualifying at the end of the year as a student who transferred in to the course from another institution, or one who has repeated a previous year. It also means that we allow the possibility of a student repeating a year more than once (which in practice is usually not accepted). Although the assumption is not completely accurate, any errors it introduces are generally very small. The effect of the errors will be to slightly under-estimate the proportion of students leaving with no qualification, and slightly over-estimate the time a student will spend in the system.
5. Some small institutions, and institutions which have made major changes in the format of their degree programmes, may find that the projections lead to some students reaching an ‘unknown’ state. This occurs if there were no students in a particular state during 1996-97, but there was movement into that state in 1997-98. For example, if an institution had no five year long programmes of study, but introduced such programmes where the first year of programme 5 was to be in 1997-98, we would have a record of students moving into year of programme 5, but no record of students moving out of it. This would lead to a number of unknown outcomes.

   Measuring efficiency

6. The efficiency measure is defined as the ratio of total efficient time for all starting students to the actual time taken by these students. The efficient time is the time that they would be expected to take to obtain a qualification if they did not repeat years or change course, etc. To produce the measure of efficiency, we need to know from which year of programme the student has qualified, or between which programme years they are transferring. A student obtaining a degree from year of programme 4, for example, is assumed to be on a degree course that should take 4 years, and is allowed 4 notional years as his or her ‘efficient time’. Similarly, a student who transfers from year of programme 1 at one institution to year of programme 2 at another institution is allowed 1 notional year for efficient time for the first institution; but a student transferring from year 1 at the first institution to year 1 at another institution gets no notional years. Students who leave higher education with no qualifications have an efficient time of zero.

   A simple example

7. To illustrate the method, assume an institution runs only three year degree courses, and two year diploma courses. Students who come in to a degree course may change to the diploma course, but not vice versa; and students may transfer to another institution or leave higher education completely. It is assumed that transfers or those leaving HE do not resume at the same institution later. It is also assumed, for this example, that students transferring to another institution go into the next year of study at their new institution.
8. Assume the following pattern of movement of students between consecutive academic years (for the actual indicators this year, this is between 1996-97 and 1997-98) :

   Of year 1 degree students, 5% repeat year 1, 70% move on to year 2, 5% stay in year 1 but on the diploma, 5% move to year 2 diploma, 5% transfer to other institutions, and 10% leave.

   Of year 2 degree students, 5% qualify with a diploma, 5% stay in year 2, 75% move to year 3, 5% move to year 2 of the diploma, 5% transfer, and 5% leave.

   Of year 3 degree students, 80% obtain a degree, 5% obtain a diploma, 5% repeat year 3, and 10% leave with no qualification.

   Of those who move to year 1 of the diploma, 50% go on to year 2 of the diploma, 30% transfer, and 20% leave.

   Of those going to the second year of the diploma, 90% obtain a diploma and 10% leave.

   This can be expressed in the following ‘transition matrix’.

   Table C1 : Transition matrix

   To : From :	Qualify degree	Qualify diploma	Year 1 degree	Year 2 degree	Year 3 degree	Year 1 diploma	Year 2 diploma	Transfer	Leave
   Qualify deg	100%	0	0	0	0	0	0	0	0
   Qualify dip	0	100%	0	0	0	0	0	0	0
   Year 1 deg	0	0	5%	70%	0	5%	5%	5%	10%
   Year 2 deg	0	5%	0	5%	75%	0	5%	5%	5%
   Year 3 deg	80%	5%	0	0	5%	0	0	0	10%
   Year 1 dip	0	0	0	0	0	0	50%	30%	20%
   Year 2 dip	0	90%	0	0	0	0	0	0	10%
   Transfer	0	0	0	0	0	0	0	100%	0
   Leave	0	0	0	0	0	0	0	0	100%

9. Now apply these movements to a set of 1000 starting degree students, 800 of whom start in the first year and 200 in the second year. (Note that all start on a degree course.)
10. The pattern of progression of the 800 students starting in the first year is shown in table C2. In the first year in the institution, all 800 are in year 1 of the degree programme. In the second year, there will be 40 in year 1 (5% of 800), 560 in year 2 (70%), 40 in each of years 1 and 2 of the diploma, 40 will transfer to another HEI, and 80 will leave higher education completely. To see where these students will be in the next year, we need to apply the progression rates for year 1 degree students to the 40 in that state, the rates for year 2 degree students to the 560 in that state, and the diploma rates to the students there. This will lead to the values given in the year 3 row; note that the 64 qualifying with a diploma are made up of 90% of the 40 who are on year 2 of the diploma plus 5% of those who are on the second year of the degree. This procedure is repeated until there are no changes to the numbers qualifying or transferring from one year to the next. In this case that takes seven years, by which time 392 will have obtained a degree, 140 a diploma, 86 will have transferred, and 182 will have left with no qualification.

    Table C2 : Progression of first year starters

    Year at institution	Qualify degree	Qualify diploma	Year 1 degree	Year 2 degree	Year 3 degree	Year 1 diploma	Year 2 diploma	Transfer	Leave
    Year 1	0	0	800	0	0	0	0	0	0
    Year 2	0	0	40	560	0	40	40	40	80
    Year 3	0	64	2	56	420	2	50	82	124
    Year 4	336	133	0	4	63	0	4	86	174
    Year 5	386	140	0	0	6	0	0	86	181
    Year 6	391	140	0	0	1	0	0	86	182
    Year 7	392	140	0	0	0	0	0	86	182
    Total time taken			842	620	490	42	94

11. Similarly, the 200 starting in the second year of the degree programme will progress as shown in table C3. The method assumes that students starting in the second year will have the same progression pattern as the students who started in the first year then moved into the second year. This group will all reach a final state within five years, when 133 will have obtained a degree, 28 a diploma, 11 will have transferred to another institution, and 28 will have left with no qualification.

    Table C3 : Progression of second year starters

    Year at institution	Qualify degree	Qualify diploma	Year 1 degree	Year 2 degree	Year 3 degree	Year 1 diploma	Year 2 diploma	Transfer	Leave
    Year 1	0	0	0	200	0	0	0	0	0
    Year 2	0	10	0	10	150	0	10	10	10
    Year 3	120	27	0	1	15	0	1	11	27
    Year 4	132	28	0	0	1	0	0	11	28
    Year 5	133	28	0	0	0	0	0	11	28
    Total time taken			0	211	166	0	11

12. So altogether of the 1000 students, there will be 525 with degrees, 168 with diplomas, 97 will have transferred to another institution, and 210 will have left with no qualification. The projected outcomes can be expressed as :

    	Degree	Diploma	Transfer	Neither award nor transfer
    All starters	52%	17%	10%	21%

13. In order to calculate the efficiency, we need to look at how long each student spends in the university. We assume that all changes (move to the next year, transfer, or leave) take place at the end of an academic year. This means that each entry in the columns labelled Year 1, etc, of tables C2 and C3 imply that number of ‘student-years’ has been used. The totals then tell us how many student-years have been spent by this cohort on year of programme 1 of the degree, etc, so we can obtain the total student-years spent at the institution. That gives us a total of 2476 student-years.
14. If the system was totally efficient, then all the students obtaining degrees would have taken 3 years, and those obtaining diplomas 2 years. Let us assume in addition that students who transfer elsewhere spend an average of 1.5 years at the institution (in practice, this average would be obtained from the pattern of transfers). Under this efficient system, the first year entrants would take

    392 x 3 + 140 x 2 + 86 x 1.5 = 1585 years,

    while the second year entrants would take

    133 x 2 + 28 x 1 + 11 x 0.5 = 299.5 years.

    The total time taken in an efficient system would therefore be 1884.5 years. The efficiency of this institution can then be calculated as :

    1884.5 / 2476 = 76%

    The average time spent in the institution, either actual or efficient, per student who goes out with some success can be found by dividing the total times by the number of successful students. The number of successful students in this example is the number who obtain a degree or a diploma, or who transfer, i.e. 525 + 168 + 97 = 790. So the average time actually taken is 2476 / 790 = 3.1 years, and efficient time is 1884.5 / 790 = 2.4 years.

    Technical details

    Projected outcomes

15. To ensure that the data used in the calculations are robust, it is important to define the stage which a student has reached by reference to four years of data. These are the ‘reference year’, which is the year to which all calculations relate; the two years prior to the reference year; and the year following the reference year. For this publication, the reference year is 1996-97.
16. At present, the method used is restricted to students who are residents of the United Kingdom, excluding the Channel Islands and the Isle of Man, who are studying full-time for a first degree. In what follows, the term ‘student’ is taken to refer to someone with all these characteristics, unless it is further qualified.
17. There are three relevant states in which a student can be in any one year. These are :
    • State A : Active at the institution as a full-time student with first degree qualification aim
    • State a : Active at the institution as an undergraduate student not in state A (could be full-time sub-degree, part-time degree, and so on.)
    • State D : Not active at the institution as an undergraduate student (could be at another institution)
18. The starting population consists of students who are currently studying full-time for a first degree, i.e. are in state A, and were not in this state in either of the previous two years, i.e. were in one of the states a or D in the two years preceding the reference year. Notice that no use is made of the date of commencement of studies in defining the starting population. The term 'entrants' is reserved for students reported as commencing studies in a given year, and 'starters' for members of the starting population as defined here.
19. As well as the starting population, we need to define a ‘transition population’, which will be used to determine the pattern of progression at the institution. This transition population is defined as all students who are in state A at the institution in the reference year, plus all those students who were in state A in the previous year and are in state a or state D in the reference year but have not obtained a degree. This is essentially all students who are currently, or were in the previous year, full-time first degree students. Students who on a sub-degree programme are only included if they have transferred from a degree programme.
20. For each student in the transition population, we need to know what year of programme they are on in the reference year, or what year of programme they were on previously if they are currently in state D. For the year following the reference year, we need to know if they have qualified with a degree or a sub-degree; which year of programme they are on if they are still at the institution, and whether they are still studying full-time for a first degree; if they have transferred to a sub-degree or part-time programme; or if they have transferred to another institution.
21. A student is assumed to have discontinued with ‘no award or transfer’ if there is no record of the student as a full-time first degree student in the reference year, and no record as an undergraduate student in the next year. A student who is in state A in the reference year, but for whom no record can be found the following year, is initially deemed to become ‘inactive’, with the possibility of returning to the institution in the following year. It is only if students cannot be traced for a second year that they are categorised as ‘neither award nor transfer’.
22. A student who is awarded a sub-degree qualification from state A is also initially placed in the inactive state. This allows for students who may return after a year to complete their degree programme. Students in state a in the reference year are categorised as failing to get an award if they cannot be traced in the following year.
23. To calculate the transition matrix, the states each student is in, both in the reference year and in the following year, need to be specified more fully. The specification includes mode of study, year of programme, and whether or not the student has obtained a qualification. The values in the matrix are then found as:

    t_ij = n_ij / n_i

    where n_ij is the number of students at the institution in state i in the reference year and state j in the following year, and n_i is the number of students at the institution in state i in the reference year.

24. Full details of the states used in the transition matrix are provided in table C4, which also shows whether or not the state is a ‘valid end’; the end value associated with each end state; and the load associated with the other states. The transition matrix for the sector is shown in table C6.
25. The matrix used in calculating the adjusted sector benchmarks is produced in a similar way, using all the students in the sector. However, each student is given a weight which reflects the relative importance of his or her subject of study and entry qualifications at the institution, and it is the sums of these weights, rather than the actual number of students, that are used to calculate the values in the matrix. The weights are defined as:

    (proportion of institution’s students in subject/entry qualification category) / (proportion of students in sector in subject/entry qualification category)

26. Several of the states are what are known as ‘sink states’. Such states are characterised by the fact that once a student enters such a state, that is the end of his or her progress through the system as far as the institution is concerned. So all the ‘qualifying with a degree’ states are sink states, as once a student has obtained a degree that completes their progress. Also, all the transfer states are sink states, as we assume that a student who transfers to another HEI will not return to the first institution. The state labelled ‘absent’ is also a sink state, from which students cannot return to the institution.
27. The data are then used to create a transition matrix T from the students in the transition population. Each row represents a state in the reference year, with each column representing a state in the following year. Each row of the matrix contains details of the percentage of students who were in the row state in the reference year, who are in the column state in the following year. Rows representing sink states contain only one non-zero value, on the diagonal, as all students in such a state will remain there.
28. The starters are described by the vector n which gives the number of starters in each of the states. All starters will be on a first degree programme, and most will be in year of programme 1, but at some institutions there may be considerable numbers of starters in later years.
29. Each state has an associated FTE, described by f, giving the amount of study time incurred by being in that state for a year. An FTE of 0 is assumed for all part-time study.
30. The progression of the starters can be computed by applying the transition matrix to the vector of starters. After i years the cohort will be in state n.T ^i-1. A total of ( n.T ^i-1 ) . f ^T study time will have been used in year i, and the 'actual' time used is given by the sum of these values for a large i. The value of i is considered ‘large’ when the values in n.T ^i-1 of the numbers of students qualifying or transferring no longer change as i increases. At that stage, the value of ( n.T ^i-1 ) . f ^T is also effectively zero.
31. There are some circumstances, for example when an institution is starting a new course, or changing the pattern of an existing course, when there are values in the state vector q = n.T ^i-1 for large i which are not an end point, i.e. a qualification, a transfer, or an absence. In such cases the outcome is recorded as unknown, and the projected outcomes given should be treated with caution.

    Efficiency

32. The efficiency of an institution has been defined in terms of how long students should take before reaching a valid end point compared to how long they actually do take. We have defined a number of outcomes which are considered as ‘valid ends’ for this purpose. They are shown in table C4.
33. Each state has a start and end value, described by vectors s and e. These are used to calculate the efficient time taken. Any state which is not a ‘valid end’ has an end value of 0. Any state which is a valid end has an end value which reflects the time that should be used in reaching it, its efficient time. The start value is only relevant to the year of degree programme states, which each have a start value one less than their year of programme value.
34. The efficient time is obtained by looking at the year of programme in which the end is reached. In general, each (non-repeated) year of study leads to one year for efficient time. So a student qualifying with a degree at the end of year of programme k is allotted k years for efficient time. A student obtaining a sub-degree qualification, from any year of programme after the first, is allotted 2 years for efficient time; a student obtaining a degree qualification from part-time study is allotted 3 years efficient time. A student transferring from year of programme x at the institution to year of programme (x+1) at another is allotted x notional years, while a student transferring from year of programme x to the same year of programme at another HEI is allotted (x-1) years. A student who is found on postgraduate study at the same institution but without a qualification is assumed to have obtained a degree, giving 3 years for efficient time. A student who is found on postgraduate study elsewhere, or in the FDS, but has not been given a qualification, is assumed to have obtained a sub-degree qualification, giving 2 years for efficient time.
35. The states in the transition matrix have been designed to take these ‘notional times’ into account. States ‘QFDk’ assume qualification with a degree from year of programme k, or with k years for efficient time. So all students who obtain a degree following part-time study are entered into state QFD3. Similarly, ‘Trans_k’ implies transferring with k years for efficient time. A student who transfers from year 2 at one institution to year 3 at another will be entered into state Trans_2.
36. The computations, using the above notation, are then fairly straightforward. The actual time taken by a cohort of students at an institution is the sum of all the values ( n.T ^i-1 ) . f ^T with summation over all values of i. The efficient time taken by the cohort is given by :

    q. e ^T - (q.c^T) / (SUM(n)) . ( n. s ^T )

    where c_j = 1 if e_j > 0
    c_j = 0 otherwise

    Summation is over all values in the vector. The first term would be the efficient time if all starters were on year of programme 1; the second term is an adjustment to allow for starters on other years of programme.

37. Until the 1998-99 HESA return, there was no way of recording foundation year students. There have therefore been differences in the way foundation year students have been returned by different institutions, and this can have a slight effect on the values taken for efficient time. This year we have therefore made an adjustment to the efficiency calculations to take this into account. For data from 1998-99, we shall be able to obtain information from the HESA record about foundation students.
38. The adjustment is needed because students on a foundation year are returned by many institutions as being on year of programme 1, followed by years of programme 1, 2, etc when they move onto the degree. They will therefore be recorded as qualifying from a year of programme k, giving k years of efficient time, when they should actually be allotted k+1 years for efficient time.
39. Where this has occurred, we have simply increased the efficient time for the institution by the number of foundation year students returned in this way. This errs on the generous side. (Note that the value without this adjustment will also be available in the electronic version of the tables.)

    Table C4 : States used in the transition matrix

    State code	Description	Valid end	End credit	Load value
    QFD1	Qualified with degree, from year of programme 1	yes	1	-
    QFD2	Qualified with degree, from year of programme 2	yes	2	-
    QFD3	Qualified with degree, from year of programme 3	yes	3	-
    QFD4	Qualified with degree, from year of programme 4	yes	4	-
    QFD5	Qualified with degree, from year of programme 5	yes	5	-
    QFD6	Qualified with degree, from year of programme 6 or above	yes	6	-
    FT FD 1	On full-time first degree, year 1		-	1
    FT FD 2	On full-time first degree, year 2		-	1
    FT FD 3	On full-time first degree, year 3		-	1
    FT FD 4	On full-time first degree, year 4		-	1
    FT FD 5	On full-time first degree, year 5		-	1
    FT FD 6+	On full-time first degree, year 6 or above		-	1
    FT SD 1	On full-time sub-degree, year 1		-	1
    FT SD 2	On full-time sub-degree, year 2		-	1
    PT FD	On part-time first degree		-	0
    PT SD	On part-time sub-degree		-	0
    QSD1	Qualified with sub-degree, from year 1	yes	1
    QSD2	Qualified with sub-degree, from year 2	yes	2
    absent	Absent from HE		0
    unk	Unknown (cannot be allocated to another state)		0
    Trans_0	Transferred to another HEI, with no efficient years		0
    Trans_1	Transferred to another HEI, with one efficient year	yes	1
    Trans_2	Transferred to another HEI, with two efficient years	yes	2
    Trans_3	Transferred to another HEI, with three efficient years	yes	3
    Trans_4	Transferred to another HEI, with four efficient years	yes	4
    if FT FD 1	Inactive, previously in year 1		-	0
    if FT FD 2	Inactive, previously in year 2		-	0
    if FT FD 3	Inactive, previously in year 3		-	0
    if FT FD 4	Inactive, previously in year 4		-	0
    if FT FD 5	Inactive, previously in year 5		-	0
    if FT FD 6+	Inactive, previously in year 6		-	0

    Transition matrix and starters

40. Table C5 shows the profile of starters on first degree courses by year of entry to the course. The majority of students start on the first year, but a few start in year 2 or later.

    Table C5 : Starters on full-time first degree programmes by start year of programme

    Year 1	Year 2	Year 3	Year 4	Year 5	Year 6	Total number of starters
    91%	5%	2%	1%	0%	0%	268,000

41. Table C6 shows the full transition matrix for the sector. As can be seen, the majority of students in a particular state either move to the next year of programme, or qualify. For example, of students who were in year of programme 1 in 1996-97, 78% moved to year of programme 2 in 1997-98; and of those on year of programme 2, 85% moved on to year of programme 3.

    Table C6: Sector transition matrix

Annex D

Research indicators - technical notes

1. The research indicators used here are based on two measures of input and two measures of output. All of these measures are currently available through the HESA databases.
2. The indicators used are standardised by cost centre, as explained below. If an institution uses the same proportion of input as it produces of output, then the indicator will take a value of 1.
3. The mappings of RAE units of assessment, and subjects, to cost centres is provided with the electronic version of this document.

   Measuring input to research

4. There are a number of problems in creating a measure of research input. These arise partly because data on expenditure are not split between research and teaching, and partly because the individual staff record is only returned for staff with a full-time equivalent (FTE) of 25 per cent or more. There is therefore no one ideal input measure, so it is proposed to use two complementary measures. These are:
   1. Academic staff costs. This measure is available for all institutions by cost centre from the HESA finance record table 4. It is therefore more complete than any based on the staff returns, though no separation of expenditure on research from that on teaching is possible.
   2. Research income from funding councils. The measure based on academic staff costs takes no account of the varying relative levels of resources spent on research and teaching. The measure proposed here, the research income from funding councils allocated for quality (QR funding), aims to provide an indicator that does take account of the varying level of resources available for research. It is based on the funding allocation model used to allocate the recurrent research funds. Such funds form part of the block grant, which institutions are free to distribute internally as they see fit. This measure assumes that this internal distribution will follow the funding allocation model used by the funding council. This, in general, will not be the case, so the measure will provide only a rough indication of what funds go where. The research funding is allocated by 69 RAE units of assessment which have been mapped to the 34 cost centres.

   Measuring research output

5. Similarly, two measures of research output based on HESA records are proposed:
   1. PhDs awarded. The number of doctoral degree completions provides a measure of the vitality of the institution in educating new researchers. The number of PhDs has been taken from the current HESA student record, summing records with a qualification obtained of ‘PhD mainly by research’. The cost centre has been identified through the same record, wherever possible, or from records for the student in previous years. In some cases, we have had to map the subject of study to cost centre.
   2. Research grants and contracts. Although this could be conceived as an input, it also provides a measure of the success of researchers in attracting funds over and above those allocated by the funding councils. The value of research grants and contracts comes from the current HESA finance record.

   Standardising for subject variation

6. There are often differences in the characteristics of research output between subjects. These can be adjusted for by making the measure specific to cost centre. If the output is, say, research contract income, then the indicator treats a pound of income as a different ‘currency’ of research output for each cost centre. This is done by looking not at the actual input and output, but at what proportions they form of the sector input and output. This means that institutions whose research is primarily in areas where the unit costs are low are not at a disadvantage compared with institutions whose research is mainly in areas with high unit costs.
7. The computation of these indicators is done as follows. Let:

   r_k = research output of institution in cost centre k

   R_k = research output of sector in cost centre k

   s_k = input to institution in cost centre k

   S_k = input to sector in cost centre k

   Then, t = SUM(s_k) is the total input to the institution for all subjects.

   For cost centre k the relative performance of an institution, p_k is given by:

   p_k = ( r_k / s_k ) / ( R_k / S_k )

   The overall performance of the institution, p, is then calculated by summing the cost centre ratios using the weighting ( s_k / t ):

   p = SUM( p_k . s_k / t ) = SUM( r_k . S_k / R_k ) / t

   Measures of coverage

8. To put these indicators into context, a number of measures of coverage were considered. The ones included in the table are amount of research funds from the funding councils, the percentage of funding from the funding councils allocated for research, and the total number of cost centres to which there is some input.