Home > Publications & reports > Publications

Analysis of data from the pilot exercise to develop bibliometric indicators for the REF: The effect of using normalised citation scores for particular staff characteristics

February 2011 | ref: 2011/03

To:

Heads of HEFCE-funded higher education institutions, Heads of universities in Northern Ireland

Of interest to those responsible for:

Equality and diversity management, Human resources management, Institutional strategic planning, Research management

This analysis aimed to evaluate what the effect would be of using citation scores in the Research Excellence Framework (REF) for staff with particular attributes, such as the protected characteristics and early career researchers. It drew on the data collected for the pilot exercise to develop bibliometric indicators for the REF.

Contents

  • Executive summary
  • Introduction
  • Data comparison
  • Early career researchers
  • Age group
  • Sex
  • Age and sex
  • Disability
  • Ethnicity
  • Nationality
  • Mode of employment
  • Discussion
  • Annex A Statistical models
  • Annex B Data definitons and groupings
  • Annex C Complete set of data tables and figures
  • Annex D Average number of papers per person by characteristics
  • Annex E Glossary and abbreviations

Executive summary

Purpose

1. This analysis aimed to evaluate what the effect would be of using citation scores in the Research Excellence Framework (REF) for staff with particular attributes, such as the protected characteristics and early career researchers. It drew on the data collected for the pilot exercise to develop bibliometric indicators for the REF.

2. This analysis follows the publication of two HEFCE documents in 2009: 'Second consultation on the assessment and funding of research' (HEFCE 2009/38) and 'Report on the pilot exercise to develop bibliometric indicators for the Research Excellence Framework' (HEFCE 2009/39). It aims to supplement the findings, informing the development and interpretation of bibliometric indicators with respect to particular characteristics of staff.

Key points

Methodology

3. The data used in this analysis were collected as part of the pilot exercise to develop bibliometric indicators for the Research Excellence Framework. The population was a subset of those staff put forward for the RAE2008.

4. The methodology of this analysis is similar to that used in 'Selection of staff for inclusion in RAE2008' (HEFCE 2009/34). Using the dataset from the REF bibliometrics pilot exercise, we considered the proportions of staff who achieved a normalised citation scoresee note 1 of greater than or equal to (≥) 2 or ≥ 4 by seven staff characteristics. We then used statistical models to compare the proportion of staff in each characteristic group that achieve these two citation score thresholds on a like-for-like basis.

5. A direct comparison between the two citation databases reviewed showed that Web of Science (WoS) was found to have less coverage for staff included in the pilot than Scopus. WoS covered 90 per cent of the records and 94 per cent of the staff members that Scopus didsee note 2.

6. Seven staff characteristics were considered; each is outlined below. The report focuses on staff achieving the ≥ 2 citation threshold from the Scopus database except where these results differed significantly from either those observed from the WoS database or with the ≥ 4 threshold.

Early career researchers

7. As might be expected, evidence suggested that staff in the early part of their academic careers (referred to here as early career researchers or ECRs) were less likely to be highly cited than those with longer careers. According to the Scopus database, 56 per cent of non-ECR records achieved a citation score of ≥ 2 compared to 51 per cent of ECR records. This suggests that using citation scores does not level out the advantage enjoyed by researchers with more years of experience. At the ≥ 2 citation score threshold this result was statistically significant for both citation databases, but not at the ≥ 4 threshold.

Age and sex

8. Age and sex were both significant factors in the modelling of this data however age effects were dependent on sex, so we considered the factors both separately and together.

9. Data from both citation databases indicated that those staff aged 50 or over were more likely to be highly cited than younger staff. The Scopus data showed that in the 50 to 54 age group 58 per cent of records achieved citation scores ≥ 2 compared to 43 per cent of records in the under 30 age group.

10. Men were consistently more likely to be highly cited than women. The Scopus data showed that 56 per cent of male records achieved citation scores ≥ 2 compared to 49 per cent of female records. However, this summary does not account for additional factors included in the models, such as age of staff member or subject area under which they were grouped.

11. The data showed that the proportion of men achieving a citation score of ≥ 2 was larger than women for many age groups, but that this was not always true for the age groups under 30 and 50 to 54. After other staff attributes were taken into account, the difference between men and women was found to be statistically significant in the age range 32 to 63.

Disability

12. Staff with a disabilitysee note 3 were less likely to achieve the ≥ 2 citation score threshold. According to the Scopus database, non-disabled staff had 54 per cent of records achieving citation scores ≥ 2 compared to 46 per cent of disabled staff records. However, this difference was explained in all cases by other staff attributes included in the models, such as their institution and subject area.

Ethnicity

13. The reported results, for ≥ 2 and ≥ 4 thresholds and WoS and Scopus citation databases, were mixed and did not agree on any statistically significant effects of using citation scores by ethnic group. This is thought to be due to there being only small numbers of staff within each ethnic minority group. However, there was some indication that staff from a Black ethnic background were less likely to achieve the ≥ 2 citation score threshold than other ethnic groups. In the Scopus database, 36 per cent of records from the Black ethnic group achieved a citation score of ≥ 2 compared to 55 per cent and 59 per cent of records from the White and Mixed ethnic groups respectively.

Nationality

14. The proportion of staff with UK nationality achieving the ≥ 2 citation score threshold was not significantly different from the proportion of staff with non-UK nationality. The Scopus data showed that 55 per cent of records from UK nationality staff achieved citation scores ≥ 2 compared to 52 per cent of records from non-UK nationality staff.

Mode of employment

15. There was some evidence to suggest that part-time staff were more likely to achieve the ≥ 2 citation score threshold than full-time, however this was only statistically significant using the Scopus citation database.

Discussion

16. The population was a subset of those staff put forward for RAE2008, and we found that the trends in citation score achievement often reflected trends previously noted in the HEFCE publications 'Selection of staff for inclusion in RAE2008' (HEFCE 2009/34) and 'Selection of staff for inclusion in RAE2001' (HEFCE 2006/32).

17. We take the differences observed in citation scores seriously and are considering ways to address the inequalities that they imply. However, the potential role of citation information is still undecided. Panels, once constituted, will be asked for their view on the use of citation data within the assessment process, and standardised data will be provided if there is sufficient interest.

18. If citation data are used then the four UK higher education funding bodies will need to ensure that institutions planning to make submissions to the REF are aware of the results of this analysis so that they can take them into account when selecting staff for inclusion. Further, panels will also need to account for the differences found and will require guidance as part of their equality briefing.

19. The four UK higher education funding bodies have set up the REF Equality and Diversity Advisory Group to advise on promoting and supporting equality in the REF process. The advisory group felt that this report should serve as a health warning to the sector when using citation information in their business processes; indeed there are particular circumstances where individuals have low publication rates for good reasons.


Notes

  1. The normalised citation score was calculated as part of HEFCE 2009/39 and adjusted the citation score to account for field of research, type of publication and year of publication. For further details see 'Report on the pilot exercise to develop bibliometric indicators for the Research Excellence Framework' (HEFCE 2009/39), paragraphs 54-56.
  2. Percentages are calculated as a proportion of the whole population. The WoS coverage was not a subset of the Scopus coverage.
  3. Disability is recorded on the basis of the staff member's own self-assessment; evidence suggests that this method of collection leads to an element of under-disclosure.
Enquiries should be directed to:

Hannah White, tel 0117 931 7063, e-mail h.white@hefce.ac.uk

REF team, tel 0117 931 7445, e-mail info@ref.ac.uk

Page last updated 7 December 2011

Share this: