Ref 98/25

Mapping Education Research in England

An analysis of the 1996 Research Assessment Exercise

David Kerr with Anne Lines, Annette MacDonald, Lawrence Andrews
National Foundation for Educational Research
Higher Education Funding Council for England

May 1998

The electronic version of this document contains the Contents, Executive Summary and Introduction only. The complete printed document is available from HEFCE.

CONTENTS

ACKNOWLEDGEMENTS

ABBREVIATIONS

EXECUTIVE SUMMARY

CHAPTER 1. INTRODUCTION AND BACKGROUND

1. 1 HEFCE Programme of Pedagogic Research
1.2 Research Aims
1.3 Report Outline
1.4 The 1996 Research Assessment Exercise

CHAPTER 2. RESEARCH METHODS

2.1 Background
2.2 Quantitative Approaches
2.3 Qualitative Approaches
2.4 Conclusion

CHAPTER 3. ANALYSIS AND FINDINGS FROM RA2

3.1 Basic Typology of Themes
3.2 Key Background Variables
3.3 Cross­cutting of Themes by Variables

CHAPTER 4. ANALYSIS AND FINDINGS FROM RA4

4.1 Rating Awards
4.2 External Research Income

CHAPTER 5. ANALYSIS AND FINDINGS FROM RA5 AND RA6

5. 1 Sub­areas of Research (or Research Specialisms)
5.2 Research Themes
5.3 Future Research Plans
5.4 Funding

CHAPTER 6. DISCUSSION OF FINDINGS

6.1 1996 RAE Context
6.2 Lessons for the Future

TECHNICAL REPORT

REFERENCES

APPENDICES

EXECUTIVE SUMMARY

Purpose of report

This report, commissioned by the Higher Education Funding Council for England (HEFCE), provides a map of educational research in universities and higher education institutions in England. The purpose of the map is to help HEFCE to situate its major new three­year Programme of pedagogic research within the greater body of externally­funded research. The map also provides a basis for identifying research themes of strategic importance to the new Programme.

Background

The map is based on the 1996 Research Assessment Exercise (RAE). The research team at the National Foundation for Educational Research (NFER) analysed the RAE outcomes using both quantitative and qualitative approaches. The methods used were exploratory and innovative; this was necessitated by the nature of the RAE data. The team examined the following items from the RAE, submitted by 84 institutions:

• the list of research publications (10,008 separate items) (RA2)
• the amounts and sources of external funding (RA4)
• the research environment and plans (RA5)
• general observations and additional information (RA6).

The report does not comment on the quality of the educational research.

Key Research Findings

The research into the RAE submissions for Education shows the following:

• Research was concentrated around six basic themes: education policy; subject-based enquiry; teaching; education management; education; and learning.
• Grouping all the basic themes into six overarching themes showed that three predominated: education (encompassing teaching, learning and subject­based enquiry); education policy; and education management.
• 'Cross­cutting' the themes by a series of background variables confirmed the dominance of the three overarching themes.
• The predominant focus and organisation of research was through subject studies.
• The 'old' universities (i.e. those with university status prior to 1992) were awarded higher RAE ratings than the 'new' universities (i.e. the remainder, including former polytechnics and higher education institutions). No new university received a 5, 5* or a 4 and no old one a 1 or 2. The 34 old universities had a much higher research output per institution than the 50 new, carrying out almost 58% of the total research output. Institutions rated highly had a much greater research output; over 50% of the total was carried out by institutions rated 4, 5 or 5 *.
• The range of external funding varied enormously and unevenly across institutions and between years. Some two­thirds of institutions in receipt of external funding got up to £200,000 (in each year of the submission period). The remaining one third received between £200,00 and over £2 million. The largest external source of funding was UK central and local government bodies (46%), followed by charities (19%) and research councils (13%).
• The level of external funding was closely related to RAE rating, and increased in step with the rating achieved. Institutions achieving rating levels 3b and below received considerably less funding than those at 3a and above; thus the 'old' universities received considerably more funding than the 'new'.
• Future plans for research commonly referred to rapid projected expansion of research studentships.
• Research plans am being built from existing specialisms, with subject studies continuing to dominate.

Issues Arising

The focus on subject studies has implications for the proposed Programme of pedagogic research and the intention to fund research which is generic rather than subject specific.

The research team faced considerable challenges in analysing the RAE data. This raises issues about the format and storage of RAE data, and the pattern of completion of RAE submissions. There are several matters to consider concerning the organisation and analysis of future RAEs:

• the need for detailed advance planning of mapping exercises
• a thorough review of existing RAE data
• pre­coding of information submitted
• greater involvement of research­active staff in future submissions
• better advice and training for staff in institutions involved in the RAE process.

There is also a question concerning the extent to which this mapping exercise could be extended beyond English universities and higher education institutions to include all education departments in the UK, and other bodies involved in educational research.

CHAPTER 1. INTRODUCTION AND BACKGROUND

1.1 HEFCE Programme of Pedagogic Research

Higher Education Funding Council for England (HEFCE) has announced plans to fund a major Programme of pedagogic research from the academic year 1999­2000. Universities and other recognised higher education and independent institutes will be able to bid for funding for specific pedagogic research. The Programme is expected to fund projects in all sectors of education, with a particular emphasis on research which is 'generic' (i.e. not subject specific), innovative and has practical applications. A steering committee has been established in consultation with appropriate bodies and interests including the Teacher Training Agency (TTA) and the Economic and, Social Research Council (ESRC), to advise on the new Programme.

In order to proceed with the new Programme there was a need for HEFCE and the steering committee to have an assessment of research in education in the United Kingdom. In particular, there was a need to establish a map of educational research within England. Such a map would help HEFCE and the steering committee to situate the pedagogic Programme within the greater body of externally funded educational research. It would also provide a basis for identifying research themes of strategic importance to the new Programme.

There has been a growing debate in recent years about the scope and state of educational research in England, but there has been no systematic attempt to map and assess the type and nature of research carried out (Ranson, 1996; Hargreaves, 1996; Hammersley, 1997; Hegarty, 1997; McIntyre, 1997; Gipps, 1997; Barnard, 1997). This debate has been fuelled, in large measure, by the establishment of the Research Assessment Exercise (RAE). The RAE is administered by the higher education (HE) funding bodies in the United Kingdom and is currently organised on a four­year cycle. Although the primary purpose of the RAE is to produce ratings of research quality in every subject area (including Education), it also offers, through the data collected, a wealth of secondary information on the organisation, planning, funding and foci of research within universities and other higher education institutions who participate in the exercise. The first RAE took place in 1989 and the second in 1992. The most recent RAE for Education was carried out in 1996 and covers research reported for the period 1 January 1992 to 31 March 1996. The data collected in relation to universities and higher education institutions in England, is held by HEFCE.

Accordingly, HEFCE commissioned the National Foundation for Educational Research (NFER) to conduct an analysis of the 1996 Research Assessment Exercise data submitted by education departments in England. This report details the outcomes of the research team's analysis of the 1996 RAE for Education.

1.2 Research Aims

The specific aims of the research project were to:

a) interrogate the 1996 RAE data submitted by universities and higher education institutions in England by both quantitative and qualitative mews;

b) construct, as a necessary precursor to the analytical work on the RAE submissions, an appropriate basic typology of themes to be used to classify the RAE data and key background variables (or criteria) to cross­cut the classifications;

c) discuss and agree the basic typology of. themes and key background variables with HEFCE before beginning the analytical work;

d) undertake basic analyses of the RAE data using a range of means including the basic typology of themes, key background variables, cross­tabulations, key word searches and manual sifting of information;

e) classify and map the research undertaken by education departments in England from the results of the basic analyses and identify the concentrations or gaps in the research effort relating to particular themes;

f) analyse and describe the characteristics and distinguishing features of the research being conducted in universities and higher education institutions under major themes

g) make reference, where possible , to research initiatives which have begun since the completion of the 1996 RAE and to future plans for educational research in universities and higher education institutions;

h) prepare a concise report of the findings to HEFCE suitable for presentation to the steering committee to assist its discussion of the shape and nature of the Programme of pedagogic research.

1.3 Report Outline

This report as agreed, with HEFCE, contains only the main findings of the analysis of the 1996 RAE for Education. The remainder of the analysis is included in the Appendices. The report is divided into six chapters. This chapter, Chapter One sets the report in its wider context. In particular it explores the reasons why this report has been commissioned and provides important background information concerning the conduct and outcomes of the 1996 Research Assessment Exercise (RAE) for Education. Such background is essential in understanding the chapters that follow.

Chapter Two explains the research methods used in the analysis of the 1996 RAE for Education, since the research methods employed by the research team were largely innovative and experimental. There were two reasons for this. The first was because the Education RAE was not specifically set up for the type of analysis requested by HEFCE. The second reason, and a consequence of the first, was that the research project had to break new ground. Though cursory analysis of the information from Form RA2 Publications and Other Public Output had been attempted by the British Educational Research Association (BERA) (Bassey and Constable, 1997), and HEFCE had carried out a similar limited analysis of ­Form­, RAO Overall Staff Summary, no one had previously attempted to carry out a systematic analysis of the complete RAE submission for Education. This meant that the research team had to construct a suitable method for analysing the RAE data virtually from scratch. This chapter provides the background to the range of quantitative and qualitative approaches that were devised in order to analyse the various sections of the 1996 RAE submission.

Chapter Three describes the results of the analysis of Form RA2 Publications and Other Public Output. This data set formed the largest portion of the RAE submission from participating institutions. It provided the clearest information on the type and nature of the educational research being conducted in universities and higher education institutions in England. The chapter identifies key themes in the research and maps the concentration of research effort within and across those themes. This is achieved through cross­cutting the themes by a number of key background variables.

Chapter Four moves from research themes to the results of the analysis of Form RA4 External Research Income. It provides a fascinating overview of the external funding that higher education in England receive to support educational research. The chapter shows the total amount of external funding, received from 1992 to 1996 (i.e. over the period of the RAE submission) and the sources of that funding. It explores the level of funding and its distribution per annum, as well as its distribution by RAE rating awarded and type of institution, (i.e. whether 'old' or 'new' university).

It should be noted that the research team defined 'old' universities as those institutions that already had university status prior to the 1992 Further and Higher Education Act.

The rest were classified as 'new' universities, and included former polytechnics and a range of higher education colleges (the full list is given in Appendix IV).

Chapter Five describes the results of the analysis of Forms RA5 Research Environment and Plans and RA6 General Observations and Additional Information. The chapter provides important contextual information on the organisation of educational research in English higher education institutions, which needs to be set alongside the findings from the analysis of RA2. In particular, it offers clearer indications of the current focus of educational research since the 1996 RAE, and of its future direction in the run up to the next RAE.

Chapter Six is a reminder of the contextual factors which need to be taken into account when reading this report. These factors relate­both to the conduct of the 1996 RAE for Education and to the parameters within which this research project was carried out. The chapter also draws together the main methodological lessons learnt by the research team. These lessons am hinted at throughout the preceding five chapters and also in the Technical Report. They are spelt out here to guide further research on the RAE, perhaps as part of a more comprehensive review of educational research in the UK.

The Technical Report and Appendices are integral to understanding the report and its findings. They provide invaluable background on the nature and conduct of the 1996 RAE. They also detail the main research tools employed in the analysis of the Education RAE.

1.4 The 1996 Research Assessment Exercise

The purpose of the 1996 RAE was to produce ratings of research quality for all submitting institutions in the UK. These ratings were then used by the various HE funding bodies (including HEFCE) to determine the size of the research grant for institutionthe financial year 1997­98 onward. The 1996 approach was broadly similar to that in 1992, HEFCE published guidance for institutions concerning the criteria for assessment and how to make a submission (HEFCE, 1995a; 1995b).

Each institution made a written submission under eight headings. In practice, this meant the completion of a range of forms from RAO to RA6 using a standard format (see Appendix 1). The information provided was a mixture of quantitative and descriptive elements. As well as providing information specifically requested by the funding bodies, institutions also had the opportunity in their submissions to draw attention to any matters which they considered particularly significant in relation to judgements of research quality. The whole RAE submission process was carried out to a very strict timetable, which meant that all submissions had to reach the appropriate funding body by 30 April 1996 (see Appendix II).

When the written submissions were received by the HE funding bodies they were passed to a specialist Education panel. Panel members were selected against set criteria from within the body of educational researchers in UK higher education institutes. The role of the panel was to carry out peer review assessment of research quality on the basis of the written submissions. This entailed forming a view about the quality of research described in each submission.

The panel's judgement about the quality of research in each submission was then expressed in terms of a standard numerical rating scale. This rating scale applied not only to Education, but to all subjects involved in the RAE process. It comprised a common scale with common definitions of the points on the scale (see Appendix III). The two important adjustments to the 1996 rating scale, compared to that used in 1992, were the division of rating point 3 into points 3a and 3b, and the introduction of a top 5* rating point This increased the breadth of the rating scale (from five to seven points) and gave the panel a wider choice in terms of the rating awarded to each institution. The complete list of rating awards for Education was published in January 1997 (see Appendix IV). This showed the rating awarded to each submission as well as the number and proportion of research staff submitted for assessment in each institution.