September 2009/39 (web only)
Issues paper

This report is for information

---

Report on the pilot exercise to develop bibliometric indicators for the Research Excellence Framework

This report presents the work undertaken to develop the bibliometrics element of the Research Excellence Framework, the key part of which has been a pilot exercise to develop bibliometric indicators.

---

To:	Heads of HEFCE-funded institution Heads of universities in Northern Ireland
Of interest to those responsible for:	Research, Planning
Reference:	2009/39
Publication date:	September 2009
Enquiries to:	Pam Macpherson Barrett tel 0117 9317471 e-mail p.macpherson-barrett@hefce.ac.uk

---

Table of contents and executive summary (read online)

---

---

Contents

• Executive summary
• Introduction
• Aims and scope of the pilot
• Compilation of databases
• Normalisation
• Indicators and outcomes
• Interpretation
• Further analysis
• Work in progress
• List of abbreviations

• Annex A Pilot data specification
• Annex B Compilation of the Scopus databases
• Annex C Field categories and normalisation
• Annex D Other variants of staff-based models
• Annex E Main outcomes graphs
• Annex F Outcomes tables
• Annex G Discussion of bibliometrics by the Expert Advisory Groups
• Annex H Availability of citation data by subject
• Annex I Development of indicators
• Annex J Stability of indicators
• Annex K Normalisation factors for Web of Science and Scopus
• Annex L Conference proceedings in Scopus

---

Executive summary

Purpose

1.   This report presents the work undertaken to develop the bibliometrics element of the Research Excellence Framework (REF), the key part of which has been a pilot exercise to develop bibliometric indicators.

2.   The report forms part of the evidence base for our consultation on the REF (HEFCE 2009/38).

Content

3.   The report outlines the aims and process of the pilot and summarises the outcomes and feedback received from participating HEIs and our REF Expert Advisory Groups. We also present the results of further bibliometric analysis that arose out of the pilot exercise and discuss analysis planned for the future.

Coverage

4.   The pilot exercise covered 35 Units of Assessment (UOAs) from the 2008 Research Assessment Exercise (RAE 2008) where coverage of research outputs was 40 per cent or greater in the commercial bibliometric databases. Twenty-two higher education institutions (HEIs) were selected to participate in the exercise. They covered a range of HEI types and sizes.

5.   Publications between 2001 and 2006 inclusive were analysed; including citations up to 2007. We used the two main commercially available citation databases for the pilot exercise: Web of Science (WoS) and Scopus.

Main models

6.   We tested three models in the pilot exercise:

1. Model 1, based on institutional address. Data were harvested from the citation databases from publications recorded between 2001 and 2006 according to the inclusion of an address associated with a participating HEI. These data were then mapped to UOAs.
2. Model 2, based on staff, all papers. Data sets were constructed to include all publications produced by staff submitted by the HEI to RAE 2008 between 2001 and 2006.
3. Model 3, based on staff, selected papers. Data sets were constructed to include the six most highly cited papers of staff submitted by HEIs to RAE 2008.

Outcomes

7.   The main outcomes are presented graphically and in tabular form in the annexes to this report. We took extensive advice on the interpretation of the outcomes from the REF Expert Advisory Groups and participating pilot institutions.

Key points

8.   Bibliometrics are not sufficiently robust at this stage to be used formulaically or to replace expert review in the REF. However there is considerable scope for citation information to be used to inform expert review.

9.   The robustness of the bibliometrics varies across the fields of research covered by the pilot, lower levels of coverage decreasing the representativeness of the citation information. In areas where publication in journals is the main method of scholarly communication, bibliometrics are more representative of the research undertaken.

Action required

10.   No action is required in response to this document.