Bibliometrics
Responses to the 2009 consultation on the REF exercise showed support for the use of citation information to inform the review of output quality by some panels. Some raised concerns about the costs involved and the potential equalities implications.
Using citation information
We will proceed in making citation information available to some REF panels, as follows:
- Each sub-panel will be invited to decide whether it wishes to use citation information to inform its review of outputs. Panels will set out in their criteria statements whether or not they will use such data, and if so how. This will be within a framework of central guidance, to ensure appropriate use of the data and to avoid any potential disadvantage to outputs for which citation data are unavailable.
- We will provide citation data to those panels using them in a standardised and simplified format. We will aim to procure the data and make them available to panels in a way that is transparent and available in an appropriate form to institutions – should they wish to use them to inform their selection of outputs – at minimal additional cost to institutions.
- We will reconsider whether the benefits of incorporating citation information into the REF outweigh the costs if only a small minority of panels request citation information, the costs are high, or if the equalities implications cannot be effectively mitigated.
Bibliometrics pilot exercise
Our proposals were based on a pilot exercise during 2008-09 to test bibliometric indicators of research quality.
The bibliometrics pilot exercise was conducted with 22 higher education institutions and covered 35 units of assessment from the 2008 Research Assessment Exercise. We used two commercially available citation databases for the bibliometrics pilot exercise: Thomson Reuters Web of Science and Elsevier's Scopus. At this stage we have no preferred database for the REF.The conclusions of the pilot exercise drew on feedback from participating institutions, our Expert Advisory Groups, and the wider sector.
The pilot exercise showed that citation information is not sufficiently robust to be used formulaically or as a primary indicator of quality; but there is considerable scope for it to inform and enhance the process of expert review.
Further information
- Report on the pilot exercise to develop bibliometric indicators (HEFCE 2009/39)
- Interim report on the bibliometrics pilot exercise
- Report on data collection for the bibliometrics pilot exercise
- First report on 'lessons learned' from the bibliometrics pilot exercise: data collection
- Second report on 'lessons learned' from the bibliometrics pilot exercise: outcomes
- Full list of bibliometrics resources and related publications
Last updated 26 March 2010