Bibliometrics
We are proposing to use data on citations to inform and supplement the Research Excellence Framework's (REF) process of expert review, where robust data is available.
Those Units of Assessment (UOAs) for which robust data are available will make use of citation information. Sub-panels will decide this in advance. We expect that medicine, science and engineering panels will do so, but that the arts, humanities and a number of other panels will not.
We will provide the relevant panels with citation information. These panels will use the information to inform and supplement their review of the outputs, to assist with achieving consistency, international benchmarking, and where possible reducing workloads.
There will be clear guidelines on using the data robustly to take account of the known limitations and avoid bias. Panels will not make judgements about the quality of outputs solely on the basis of citation information; expert judgement must be applied. All submitted outputs will be treated equally, whether or not there is citation information available for them.
Bibliometrics pilot exercise
Our current proposals are based on a pilot exercise to test bibliometric indicators of research quality that we ran in 2008-09. The conclusions of the pilot exercise drew on feedback from participating institutions, our Expert Advisory Groups, and the wider sector.
The pilot exercise showed that citation information is not sufficiently robust to be used formulaically or as a primary indicator of quality; but there is considerable scope for it to inform and enhance the process of expert review.
The bibliometrics pilot exercise was conducted with 22 HEIs and covered 35 UOAs from the 2008 Research Assessment Exercise. We continue to conduct further analysis on the behaviour of the commercial citations databases, normalisation and equalities issues. These will be published in 2009.
We used two commercially available citation databases for the bibliometrics pilot exercise: Thomson Reuters Web of Science and Elsevier's Scopus. At this stage we have no preferred database for the REF.
We are investigating the implications for higher education institutions (HEIs) of using citation data in the REF, and are exploring licensing issues with JISC Collections. Decisions on citation databases will be taken after the 2009-10 consultation has concluded.
Further information
- Report on the pilot exercise to develop bibliometric indicators (HEFCE 2009/39)
- Interim report on the bibliometrics pilot exercise
- Report on data collection for the bibliometrics pilot exercise
- First report on 'lessons learned' from the bibliometrics pilot exercise: data collection
- Second report on 'lessons learned' from the bibliometrics pilot exercise: outcomes
- Full list of bibliometrics resources and related publications
Last updated 23 September 2009