Consultation 98/54Research funding: introduction of a policy factorRespond by 18 December 1998
Executive summaryPurpose 1. This document seeks views on the value of a policy factor in calculating the HEFCE's allocation of funding for research. Key points 2. Such a policy factor would relate only to the allocation of total research funds between subjects (Units of Assessment), not to the distribution of funds between higher education institutions. This consultation draws on a study of the international standing of higher education research in England, undertaken by the Centre for Policy Studies in Education at the University of Leeds. The study's summary report is at Annex A. 4. At Annex B is the list of HEFCE Units of Assessment for the 1996 Research Assessment Exercise. Action required 5. Replies are invited by 18 December 1998, to enable us to decide whether, and on what basis, to introduce a policy factor in the research funding method from 1999-2000. Background6. We announced our new funding method for research in February 1997 (HEFCE Circular 4/97 'Funding Method for Research from 1997-98'). Most of our research funds (£804 million in 1998-99) are distributed according to ratings of quality in the Research Assessment Exercise (RAE). In allocating these quality-related (QR) grants for research, we distribute the total sum provided between the 69 Units of Assessment (UOAs) used in the RAE, on the basis of the relative costs of the subjects and the volume of research carried out in each. 7. Circular 4/97 indicated that we would consult on the introduction of a policy factor into the method. This would allow the amount provided for each subject (the subject quanta) to be weighted, if appropriate, to reflect the nation's relative international strength, or national need, in different subjects. 8. This consultation paper seeks views on the proposed policy factor. It also reports the results of a study commissioned to inform discussion on the relative international standing of research in different subjects, and seeks views on whether this should influence the distribution of resources between subjects. National need 9. The Government's Foresight programme is the most concerted activity to date which might contribute to a view about national need. In 1996 we explored how the Technology Foresight priorities might map onto the UOAs, but this work proved of only limited value. In both formal and informal consultations, strong views have been expressed that it would be wrong to apply the results of Technology Foresight to the funding method in a formulaic way. 10. The main role of our funding is to build research capability in institutions. In practice this means expenditure on academic salaries, and support for research students and major infrastructure items such as buildings, library materials, and large pieces of equipment. Council funding is also expected to support young members of staff in the early stages of their research careers. 11. Bearing in mind that the HEFCE's grant represents about one-third of the total funding for research in higher education, there is a view that the Research Councils, charities, Government and industrial sponsors of research, with their missions as funders of projects, are better able to reflect national need. Leaving aside general questions about the funding of the dual support system, the key question is the inter-relationship between institutional block grant from the HEFCE, and funding provided by the sponsors of projects. If research capability is effectively matched to the demands of project funders, the introduction of a policy factor may be unnecessary. However, if there are areas where the capability of the higher education sector is significantly out of line with demand from project funders, then the introduction of a policy factor might be thought desirable. 12. This consultation document does not develop further the question of a policy factor based on national need. However, we seek comments on whether there are subject areas whose funding should be increased on grounds of national need. International standing 13. The work on defining the international standing of the nation's research activity has been carried forward primarily through three studies: a. A bibliometric study, undertaken by a team at the Centre for Policy Studies in Education (CPSE), University of Leeds, based on citation analysis of published journals. b. An initial study of the scope for using information about patenting activity and about the published research underlying patents. c. An examination of submissions from higher education institutions (HEIs) to the 1996 RAE for evidence of indicators of international standing and peer esteem. 14. A steering group, including representatives of the Office of Science and Technology (OST), the Research Councils and learned societies, guided the work in 1997. These bodies have also been invited to comment on how far the research findings provide a robust basis for funding decisions, and whether there are other factors, including considerations of national need, which should be taken into account. Patents and information from RAE submissions 15. On the advice of the steering group, we have decided not to pursue further the analysis of patents at present, on grounds of technical difficulty; it would be necessary to scrutinise complete patent applications, reducing the scope for computerised analysis. The pattern of patenting activity varies considerably between UOAs, so the most useful information provided would be an indication of activity over time within individual UOAs, rather than a primary indication of relative strength in an international context. Moreover, the findings of such a study would apply to only a proportion of UOAs. 16. Within the HEFCE we undertook a sample analysis of references in RAE submissions to indicators of international peer esteem. We examined what indicators were cited by HEIs, and how far panels had identified such indicators, either in their published criteria or later in considering the submissions. We concluded that certain indicators (for example, invitations to address international conferences) occurred with some frequency, and had been accepted by the assessment panels as evidence of peer esteem on the international stage. However, this was not sufficient evidence from which to draw conclusions about the relative standing of particular subjects and would not be robust enough to inform funding allocations. Bibliometric analysis 17. A summary of the findings of the CPSE study is at Annex A. The complete final report is a substantial document, copies of which can be downloaded from the CPSE website at http://www.leeds.ac.uk. The study sought to determine whether journal publication and citation data, collected by the Institute for Scientific Information (ISI) in Philadelphia, could be used to index the international standing of institutions' research effort with sufficient robustness to support the implementation of a policy factor. It also sought to map the research activity defined by the ISI's standard subject definitions onto the subject UOAs in the UK institutions' RAE submissions. 18. The study appears to show variations in the quality of research in English HEIs compared with international benchmarks. There may be an argument that limited resources should be concentrated where we have particular strengths, but there is an equally compelling argument for targeting resources at areas of weakness, particularly if such areas are thought to be important. 19. We seek views on whether the distribution of HEFCE funds between subjects should be influenced by judgements about their relative international standing; and if so, whether the findings of the CPSE study provide sufficient grounds for such judgements. Annex ABenchmarking of the international standing of research in England: a consultancy study based on bibliometric analysisSummary report
ContentsPreface I am pleased to introduce this bibliometric analysis of the research in English higher education institutions (HEIs), as compared with other countries. Despite their many and well-recognised biases and other problems, bibliometrics – counts of papers, citations, and other objective measures of published outputs and impacts – are a long-standing area of interest in science policy. They have provided a focus for a significant body of work, including both grant-funded research and Government-funded commissions in this country and overseas. This new HEFCE study continues that interest with a specifically English focus, and complements work on international comparisons completed by the Office of Science and Technology (OST) in March 1997 and later published in the journal, Science. (Office of Science and Technology 'The Quality of the UK Science Base', 1997, Department of Trade and Industry.) In my article on 'The Scientific Wealth of Nations' (May, R. M. 'The Scientific Wealth of Nations', 1997, Science, vol 275, pages 793-796.), I took a broad comparative overview of more than 8 million scientific research papers (broadly defined) from 79 countries, over 14 years. And I risked some speculations, grounded on tentative conclusions arising from this work. First, the USA, UK and Canada did much better – on a number of performance indicators – than France, Germany and Japan. Second, there are particular UK strengths in biological and biomedical research, and a significantly greater evenness in UK scientific impact across disciplines, than for many comparator countries. I suggested that institutional settings, and in particular the degree of hierarchical structure within them, seemed likely to play an important role in explaining these international differences. The HEFCE study has taken a more detailed approach. It is concerned solely with England, and has a particular interest in the place of HEIs within the research 'economy'. It focuses on a smaller number of those comparator countries which have the greatest concentrations of research activity; and it looks at a more recent time period. The study also takes a novel approach to bibliometric classification, using journal lists defined by researchers, to provide a specific analysis for the 69 Units of Assessment of the 1996 Research Assessment Exercise. As an ecologist, I am pleased to see familiar tools for analysing diversity and similarity being applied to the 'species' of disciplines and institutions in these research community lists. The key findings of the study seem in accord with OST's earlier conclusions. Many of the results tell us about an English success story in research, but there are also areas which appear less competitive internationally. It is inevitable that many readers of this report, particularly those relatively unfamiliar with bibliometrics (and, even more, those with a visceral dislike of them), will find reason to reject either particular findings or the general approach to 'measuring' research. I would urge you, before coming to that conclusion, to consider why two rather different studies have delivered similar results. Bibliometrics are only a tool: they are not an end in themselves. They can stimulate many questions, but they will never provide final answers. There is much food for thought here, and the HEFCE's consultation on this report provides a valuable opportunity for the research community in England to look at its international standing. This is an important debate and I commend the reader to undertake a considered review of the evidence in this report. Sir Robert May FRS 1. This report describes the results of a bibliometric analysis commissioned by the HEFCE to develop indicators of the international standing of English higher education institutions (HEIs) in research, across the 69 Units of Assessment (UOAs) established for the 1996 Research Assessment Exercise (RAE). 2. The study is based on the published outputs (books, journals, conference proceedings, and so on) submitted by academic staff to the RAE. Since these published outputs (up to four items per individual) were a key indicator by which peer review panels made their judgements, they are assumed to represent the highest level of performance achieved in the assessment period. Therefore, collectively, these outputs offer a basis for a comparative measure of the broad international quality of the community. In practice, the study focused solely on Category C items (peer-reviewed journal articles) for which databases providing comparative international material already exist. Approach 3. The RAE journal dataset compiled from Category C publications submitted by researchers to a given UOA provides the basis for a publication-based map determined by the community itself. This is the first dataset derived in this way to be analysed for international research comparisons. As a reference set, a second dataset was derived by combining journal sets grouped in the established subject categories of the Institute for Scientific Information (ISI) in Philadelphia. The ISI's 94 subject categories were mapped to the 69 UOAs by seeking maximum overlap with the RAE-derived journal lists. For each of the RAE and ISI journal lists, a dataset of papers with citation records was abstracted from the ISI database. 4. The RAE-derived and ISI maps address different aspects of the research community. The RAE map analyses the topology of England-specific activity in UOAs, which tend to map to structural entities (departments) in HEIs. The ISI map analyses the topology of a disciplinary community which spans country, system (frequently HEFCE UOAs) and institutional (departmental) boundaries. 5. The datasets used for the analyses covered refereed journal papers for the period 1988-1996, which produced about 500,000 papers for England. Statistically significant correlations were established between sample performance indicators derived from the RAE and ISI datasets. The greatest overlap between RAE and ISI fields is in predictable field-specific sets where departmental disciplines are consonant with internationally recognised disciplinary boundaries, for example UOA Physics maps well onto ISI Physics sub-categories. 6. The RAE dataset included a subset of papers with 'England HEI' addresses. This was used for analyses within England, such as England-HEIs against England-total. The ISI dataset was used for international comparisons of the relative research performance of England and six other countries: USA, Canada, France, Germany, Australia and Japan. These comparisons included primary analyses of total activity (publications and citations) and secondary indicators of performance (such as citation rates, re-based to world and disciplinary averages as appropriate). 7. England ranks first or second in the world, based on research impact (citations per paper within discipline), for half of the 47 subjects analysed at UOA level. (For 22 UOAs, Category C output data were inadequate or inappropriate for full analysis). The USA ranks first where England ranks second. 8. England continues to perform well when UOAs are grouped into 13 clusters of similar disciplines (Super-UOAs, or SUOAs). By comparison with world baseline impact (1.0) , England's performance ranged from 0.8 to 1.25. England is ranked second or third in 10 of the 13 SUOAs, and is second overall to the USA. Canada is a close third overall, but its research volume (and hence depth) is markedly less, and its average impact is lower. This general performance must be considered an excellent mark of research activity in England. 9. At UOA and SUOA level, England is particularly strong in Pre-clinical and Biological Sciences, and strong in Mathematical and Physical Sciences. Its performance is poor compared with the world average in Engineering. Performance in the Social Sciences is more variable, being strong in Social Policy and Economics but weak in Business Studies. Datasets are weaker in these disciplines, however, and data are inadequate for any analysis of the arts and humanities. 10. England-HEIs and England-total performance are similar for most UOAs. In the Clinical Medical UOAs, England-HEIs outperforms England-total; while in the Pre-clinical Sciences there is a significant non-HEI contribution. 11. There is a (non-significant) inverse relationship between world share of outputs and relative world impact: that is, in broad areas where England's output share is higher, the relative impact is lower. By contrast, for France, Germany and Japan there is a positive relationship between subjects with high output share and high relative world impact. The USA and Canada have a relatively even research performance across disciplines. England: performance by discipline 12. The performance of individual UOAs was analysed relative to a dataset derived from an RAE-based journal set, and relative to an ISI-based dataset. Data were also considered at the level of the following SUOAs. The clustering into SUOAs was determined by reference to a statistical analysis of journal similarity among the publications submitted to the RAE. Although some detail is obscured by this process, underlying and general patterns emerge. Clinical Medicine 13. England is in the second rank with Canada. Within clinical UOAs, England-total performance was low compared with England-HEIs, probably reflecting the large volume of non-HE activity within the NHS. England nonetheless stands well ahead of other European countries in this area. The clinical sciences include an exceptional diversity of medically-related ISI categories, and there is significant variation in England's performance among these categories, suggesting that over-aggregation in these UOAs may be an issue. Clinical Dentistry 14. England is well behind the leaders and below the world average, despite an exceptional world share. Generally, in an area of relatively poor performance and low impact for a science-based subject, England does poorly against that low benchmark. Pre-clinical Sciences 15. England is only fractionally second behind the world leader, the USA, and far ahead of Germany in third place. A high citation share means that England's moderate share of world publications in this group has a high impact. This area also has the greatest improvement in impact when non-HE activity is factored in. There is a particular non-HE bibliometric impact in the area of Pharmacology/Pharmacy, possibly due to the pharmaceutical and biotechnology industries as well as units funded by the Medical Research Council (MRC) and charities. There is a related improvement in Biochemistry. This non-HE activity is primarily due to Molecular Biology (which is also returned within Biological Sciences and probably also within Clinical Medicine), perhaps the most intensively supported and research active area within the global research base. Against a background of high international investment – in the public and private sectors – it may be difficult, but at the same time essential, to maintain a high research profile in order to sustain long-term research outcomes. Biological Sciences 16. England is second to the USA and far ahead of other countries. The effect was noted above of a link to a strong and research-based industrial sector which contributes to a strong performance, both through its financial support of HE and its own outputs. This has a broad impact in this area. England's research performance is strong internationally across all the disciplines, and relative impact is high despite a modest share of world publications. Biology contains mature fields in the core animal and plant sciences and in agriculture, where England has a strong historical reputation and world standing. A small world share of publications but large share of citations in biological subjects produces a strong relative impact, but indicates that the discipline is sustained on a relatively small base of activity – particularly in UOA 12 (Biochemistry). Mathematics 17. England is in a good second rank with Germany. In individual UOAs, England is ranked first for Pure Mathematics and second for Statistics. In Applied Mathematics, where performance is linked to its applications, England falls behind France and Germany which have strengths in physical and applied sciences. Mathematics produces the largest positive difference between publication share and citation share in both the RAE and ISI datasets. It is, however, important to note that this performance is based on a very small share of world outputs, suggesting that the field is relatively small. Physical Sciences 18. England ranks consistently third, at a grouped and individual UOA level. Despite a low world share, England is essentially part of the second rank of nations which includes Canada and Germany. This major area of research activity has significant links to areas underpinning the economic base, and to manufacturing and process industries. As a core part of the natural and applied sciences, any fragility in performance may be of concern. Engineering 19. England is in a trailing group with Canada and Australia. Despite a good world share of activity, England is below world average, ranks poorly, and is well behind its European competitors. This may indicate core strengths in other nations, particularly in the German research base where there is concentrated activity and impact in Engineering and in Physical Sciences. For England, on all analyses, there is a consistent dip in relative citation share across Engineering, seen most strongly in the ISI data. Performance measures are also consistently low within UOAs: the output has a low relative impact, and secondary impact measures dip below 1.0. Given the strong performance of otherwise weaker nations in these subjects, this result may suggest that Engineering performance is of significant concern. The fact that linked UOAs 28 and 33 (Civil Engineering and Built Environment) were ranked so low internationally (sixth of seven) would seem to reinforce this impression. The management of engineering research, a weak non-HE R&D base, and the decline of the industrial sector may be contributory factors. Engineering contributions to the RAE were notable for the common inclusion of conference proceedings, for which no impact measures could be calculated, but the analysis accounts for this by making like-for-like comparisons with engineering research in other countries. It should be noted that the performance of the UK as a whole differs from England alone: Scotland is particularly strong in Engineering. Earth and Environment 20. England has a very good share of activity in this area, where there are few differences in performance between countries. The general level of performance is in line with that in the Physical Sciences, and research performance in the Earth Sciences and Physical Sciences may indeed be interdependant. A surprisingly poorly ranked performance in Geography is balanced by better performance on secondary impact measures, and overall Geography performance benefits from a high relative world share of outputs and citations. Social, Health & Community 21. England is second to the USA and far ahead of other countries. This grouping brings together UOA 2 (Community-based Medicine), UOA 10 (Nursing) and UOA 11 (Other Studies and Professions Allied to Medicine). This was a clear grouping which emerged from the bibliometric similarity analysis, and it is linked to Psychology. There is an exceptional world share in UOA 2, and England is well ahead of other non-USA countries across the broader area. This suggests that there is a strong lead in an area which is weak globally, confirming other studies which have found strengths in Epidemiology. Nursing, by itself, is a relatively weak and emerging academic discipline: publication behaviour in England may not be typical. Overall, however, this area is likely to be of increasing economic and social significance. Social Sciences, including Law and Politics 22. (Caveat: social science data quality is not as well founded as in the natural sciences). England is well behind the USA and is in second place with Canada. There were many social science UOAs which had journals with titles which could not be matched on the ISI database. A significant volume of RAE material is therefore lost to the subsequent analysis. On the material which was available, there were very high degrees of similarity or sub-setting between UOAs 40 (Social Policy and Administration) and 41 (Social Work), and UOAs 40 and 42 (Sociology). Distinctiveness of these analyses is therefore suspect. Across this diverse area of 'social and related sciences' there is an exceptional world share, and a first ranking in Social Policy and in Social Work. In general, however, the share of world citations is less than the share of world publications; overall there is a disappointing level of performance with poor relative citation index. 23. The performance of UOA 34 (Town & Country Planning) is exceptional in terms of volume and impact. Most of England's input to this UOA is conventional planning, but some work in transport studies is included, although that sub-discipline is more widely returned. The performance cannot be attributed solely to the possible effect of the ISI grouping, as the RAE dataset based on the UOA's own journals produced a similar peak performance. 24. The performance of UOA 36 (Law) is also noteworthy: this subject is so dominated by the USA that any international analysis is distorted beyond normal and sensible limits. Business and Management 25. England is in third place, and well behind the USA. Although relative citation index is very low, the levels of performance in Accountancy and in Economics, which has a strong historical base and is consistently the strongest discipline, enhance the general position despite the low impact of Business Studies. There is a significant disparity between the volume of papers in the RAE and ISI datasets, particularly in Accountancy. The similarity measures strongly suggest that there is very substantial overlap in returns to these UOAs, despite the HEFCE definitions, and it may be impossible to distinguish the included subjects. Psychology 26. England is only marginally behind the world lead in the USA. There is a good world share and strong level of performance for UOA 13 (Psychology). It is worth noting that this SUOA is allied to Social, Health and Community in the similarity analyses and this may confirm an impression of potential strength in breadth and, with a total output equal to Engineering subjects, in depth. Other Social Science 27. This is not a coherent grouping, but catches those units for which a sound bibliometric analysis could be performed and which did not cluster elsewhere. UOA 61 (Library and Information Studies) performed rather strongly. UOA 65 (Communication and Media Studies) is weak. It is surprising that the international level of performance in UOA 68 (Education) is not better, given traditional strengths within the UK. Arts and Humanities UOAs 45 through 64 submitted relatively fewer Category C journal articles to the RAE, did not map well to the ISI system, and bibliometric impact measures benchmarked poorly against RAE panel grades. No detailed bibliometric analysis of these UOAs was pursued. 28. From our dataset based on the ISI categories grouped to UOAs, volume output and impact relative to the rest of the world (that is, re-based to a world average of 1.0) for the period 1988-1996 is shown in the table below.
29. The following commentary concentrates largely on areas of high relative output, where research performance is usually supported in depth. USA 30. The USA has a high and consistent (very low variance) impact across UOAs, especially in the natural sciences. Its average impact (1.32) and international ranking (1.38) put it clearly ahead of other countries, and it ranked first in 35 of the 47 UOAs and second in nine others. 31. The USA performance statistics are partly a consequence of its high relative volume (greater than the other six countries combined), which has a direct effect on the comparator group average. Among the comparator group, the USA has a greater than 40 per cent share of outputs in all but one of the UOAs and a greater than 50 per cent share of citations for all but two. This volume effect can be significant: the USA has a low input to Mining Engineering, and an exceptionally low impact, ranked sixth internationally. The effect of this is to produce a significant rise in relative impact in all other countries, and superficially there appears to be a global peak of achievement. Due to the relatively parochial significance of most research in Law, by contrast, the USA has a predominance which dwarfs research activity in all other countries (92 per cent of comparator group outputs). 32. There is also a more general trend, with the USA having a high share (typically 55-90 per cent of outputs and 60-90 per cent of citations) among the Clinical, Pre-clinical and Social Sciences, but a low share (40-53 per cent of outputs and 43-59 per cent of citations) among the Biological and Physical Sciences and Engineering. The effect of this is that the database has a relatively higher international 'richness' in this latter group, and the statistical comparisons should therefore be considered to be most reliable in these subjects and least reliable in the Social Sciences. 33. For the USA, the UOA league table indicates that its performance in the Physical Sciences and Applied Mathematics is unusual in being uniformly high, thereby creating an outstanding world lead. On the other hand, USA Engineering is bi-modal with Electrical, Chemical and Mechanical Engineering at the top, and Civil Engineering (and related subjects) and Mining Engineering much lower ranked. Variance in performance is higher among the Biological and Clinical Sciences, but the performances of Pharmacology, Pharmacy and Physiology are almost indistinguishable. Japan 34. Although discussion of Japan's performance should be in the context of a language and script which render much of the literature impenetrable to those without access to translation services, Japan's relative volume within the ISI database (more outputs than any other country except the USA) indicates a significant contribution to widely used international journals. Overall, however, Japan's average rank of 6.17 and average impact of 0.61 compared with this group average (0.70 compared with the world) suggest that its contribution to basic research has yet to be fully realised. Its low variance in impact is primarily a consequence of this low overall performance in relation to volume. 35. Japan's research strengths lie in Engineering (where it picks up to world average ahead of England and Canada) and in the Physical Sciences, particularly related to Electrical Engineering and to Materials Sciences. In the biosciences it is rather weaker, despite significant investment in biomolecular research in recent years. This suggests that basic research in this area still has some way to develop. What is difficult to pick out from a whole-country analysis is the contribution made by particular sectors, and the performance of some institutes may be highly competitive. EU nations 36. By plotting relative impact against UOA number, we have shown that there is some general correlation in disciplinary strength for England, France and Germany. This may be interpreted as an indication of a common European research culture, but there are important differences in absolute strength between the core and applied sciences and commonality does not extend to the Social Sciences. 37. The comparative overall performance of France and Germany is also very similar to that of Australia. The average ranks are 4.53, 5.06 and 4.70 respectively and the overall impacts relative to the comparator baseline across the 47 UOAs are 0.74, 0.70 and 0.77, with rather higher variance in France and Germany. Germany 38. The German pattern is much less even than that of the USA or England. It is one of concentrated output in selected core fields – particularly within the Physical and Mathematical Sciences and Engineering – and higher impact performance in many of those fields. For example, there is a particular interdisciplinary strength, with high output and impact, across Applied Mathematics, Physics and Electronic Engineering. There is also exceptional output and sound impact in Metallurgy and Materials and in Chemistry. 39. These strengths are not always allied to consistency across broader fields, and Mechanical and Chemical Engineering (ranked six and seven internationally) perform rather less well. The SUOA for Engineering slips to third overall behind France, and the quality of research in Physical Sciences is diluted by the volume of material. For example, Biochemistry has more output than England but lower impact, and is below world average. 40. Across the social and economic sciences, and related fields, Germany appears to have a consistently poor performance, with many subjects' research impact rated around half the world average. It is important to emphasise that, in these disciplines, this may be a consequence of different national publishing patterns and the extent to which these disciplines interact or are recognised at an international level. National culture may also be significant: theoretical approaches may also be of greater significance but less well marked internationally; subjects may be categorised differently, with Sociology orientated to and published within Philosophy. This does, however, raise important questions about the coherency of European social science research programmes. 41. Across the sciences, it may be argued that German research is effectively concentrated in more managed programmes than in England. For example, the mission-orientated Max-Planck Institutes should have a significant effect on performance in specific areas. France 42. Despite a relatively modest output volume, the performance of France in Engineering is notable. Most engineering subjects are in the UOA upper quartile and all in the top half of the list. Mining and Chemical Engineering have a particularly high impact on a relatively small output, but Civil and Mechanical engineering are also strong . This strength spreads across related areas, including Applied Mathematics and Physics. 43. This may be a 'management' effect, as suggested for Germany. The effect of the Centre Nationale de Recherches Scientifique (CNRS) may be to direct a more targeted research programme than would be typical in universities. On the other hand, biological subjects – which are well represented within the CNRS – are rather weaker. Furthermore, both Biochemistry and the Clinical Sciences perform rather poorly, both within the French league table and by international standards. It is interesting to note that France is also weak in the health and community area, suggesting that this may be an area of surprising general under-achievement, although Psychology is stronger. 44. Business Studies is stronger in France than in Germany, suggesting that not all the weaknesses in the latter are non-anglophone based, but the general level of performance in the Social Sciences is equally poor. Canada 45. Canada broadly ranks third behind the USA and England, with an average rank across UOAs of 3.19, and an overall relative impact of 0.74 compared with this group and 1.02 compared with the world average. There are no particular peaks or concentrations of research excellence: variance in impact is less than for other countries, and impact is maintained across a broad front despite a low relative volume. 46. At the level of SUOAs, Canada's average performance is on a par with that of England. Unlike England, however, it is difficult to pick out areas of consistent strength in depth. On the whole, within its own UOA league table, Canada is stronger in Physical Sciences than Biological Sciences, while Engineering tends to be rather mixed in quality. 47. Business and management is relatively strong, perhaps influenced by the level of the USA in this area. The performance of Canada in the Social Sciences is also notable, and in some environmental subjects (Agriculture, Environmental Sciences, Built Environment – the last with relatively high output and impact). On the other hand, its performance in Biological Sciences is marked more by high output but modest strength. Australia 48. Australia has a comparable average impact to France and Germany despite an output much less than half of either of theirs. At an aggregated level, it also has a more consistent performance, rather like that of Canada. Within most of the SUOAs, Australia has both strong and weak areas: for example, real strength in Mining and Civil Engineering is offset by much weaker performances in Electrical Engineering and Computer Science. 49. It performs well in the Earth and Environmental Sciences, which accords with its strengths in Mining and also in Agriculture and is perhaps culturally related to a successful history of natural resource exploitation. 50. For Australia generally, the relatively low volume makes further and more detailed analysis less appropriate. Methodology and datasets Background to methodology 51. The criteria set by the RAE require HEIs to submit data which enables peer review panels to make judgements on the research achievements (performance) and potential of academic individuals and groups, in terms of national and international standards within their disciplines. Since the outcome of these judgements influences future funding, it is assumed that the published outputs submitted to the RAE (up to four items per individual) represent the highest level of performance achieved in the assessment period. 52. Collectively, these outputs offer the basis for a comparative measure of the broad international quality of the community. The journal list for a UOA is a publication-based map determined by the community itself. Very few such maps exist, and none which have been collected and tested under criteria with funding consequences. This approach also ensures broad consistency in the mapping of individual academic departments or research groups to UOAs in both the bibliometric study and the RAE. Thus the research strengths of a particular UOA identified from the bibliometric study relate to the same groups submitted to that UOA in the RAE. Initial methodology and benchmarking 53. The outputs (books, journals, conference proceedings and so on) submitted by academic staff to the RAE were analysed and, where the Category C (refereed journal article) material was sufficient, were used to create journal sets for each UOA. The given title for journals appeared in the database in a number of variant forms which nonetheless had the same ISSN. To unify these variants, the journal titles were synonym-unified to the bibliographic database developed by the Institute for Scientific Information (ISI) in Philadelphia. The matched set of 6,146 unique journal titles covered 106,257 of the 131,091 journal articles submitted, accounting for 81 per cent of the full RAE Category C dataset. 54. The similarity between the RAE-derived journal lists and those in established ISI product categories was analysed. This analysis showed that: • core disciplines were well defined • the extent of non-specific overlap reflected the diversity of disciplines within UOAs • there was a higher article/journal ratio in overlap areas than in journals not overlapping the specific ISI category, confirming that the core area was that most intensively used for publication • a discrete part of the ISI set, with low RAE overlap, was journals • high impact factor journals (such as Nature) were among journals occurring in the greatest number of UOAs: these are fractionally attributed to UOAs in later analyses. 55. A linkage analysis of the bibliometric association between UOAs and the ISI journal categories captured 72 per cent of the RAE data in 2.5 per cent of possible category combinations (163 of 6486). Mapping quality was very poor for the arts and humanities, and some other UOAs were either marginal or appeared to be sub-sets of other disciplines. These UOAs were excluded from later analyses. 56. Benchmarking of impact for the journal data against RAE grades: • varied between UOAs and was weaker in the social sciences and absent in the arts • was very good in the natural sciences • within a UOA, the HEIs which achieved high grades also had high average bibliometric impact scores for these datasets • was not time-frame dependent, but the general trend of UK bibliometric performance is for publications and citations to increase with time. 57. Analysing similarity and diversity: • diversity within journal datasets varies continuously across most UOAs, that is, there are few marked discontinuities, and the evenness is fairly constant • there is little evidence of lack of homogeneity within UOAs except in UOAs 33 (Built Environment) and 34 (Town & Country Planning) • there is a relatively low level of average bibliometric similarity among UOAs compared with a random dataset of journal lists • similarity is further decreased marginally by taking frequency of journal occurrence into account, and there are many zeros in the RAE matrix because of journal specificity • there is some pair-wise similarity clustering of UOAs • there is evidence for more general clustering in cognate groups - these 'Super-UOAs' are used in later analyses • the level and nature of UOA clustering may be affected by restricting the HEIs contributing to the total and to the UOA journal lists • within UOAs, HEIs that are highly rated in the RAE (with grades of 4, 5 and 5*) tend to show a relatively high level of similarity, which may suggest a greater commonality in high impact research activity than across the field as a whole. Datasets 58. The ISI address-corrected database of UK publications and citations was used to create two main datasets of England-address publications. These were used to analyse the 47 UOAs for which sufficient high quality Category C data were available. Dataset I grouped data on the journal lists derived from the RAE submissions, with fractional assignment of commonly occurring journals, and a full extract was carried of all publications during the period 1988-1996 for each UOA. Dataset II grouped data in the established ISI disciplinary categories. This provided a total of 94 categories, which were then mapped to the UOAs by reference to the methodology established in the first phase. The full datasets included all publications with an address in England. 59. National research is partly driven by research in HEIs, including teaching hospitals and allied medical units, but is also supported by work in other public sector institutions such as Research Council institutes, and by charities and private sector laboratories. To investigate this issue, a subset of Dataset I was created for England-HEI addresses. Variations are most evident in the bio-medical area where units outside HEIs enhance England's performance – notably in the pre-clinical subjects (UOAs 5-8) and Biochemistry (UOA 12). Possible sources of high-impact input are Research Council institutes, charitable laboratories and the UK pharmaceutical/biotechnology industry. In the clinical UOAs, the effect is reversed: performance is above world average within HEIs but lower for England as a whole. This may reflect the quality of the volume of research supported by NHS funding. 60. The correlation between the UOA impact scores for the two datasets was statistically significant (r = 0.43, 46 d.f., P< 0.005). Significant outliers include UOA 33 (Built Environment – RAE = 1.4 impact cf world baseline, ISI = 0.62) and UOA 41 (Social Work – RAE = 0.54, ISI = 1.2). The overall pattern is for the ISI dataset to accentuate differences between units compared with the RAE-derived dataset. Thus the gradient of the core data-plot is somewhat steeper than parity: high-impact subjects improve and low-impact subjects decline. Although the absolute number of papers varies between the datasets, England's share of the world total is broadly correlated and is not biased consistently between the datasets. World share is very broadly 7 per cent of publications and citations for the RAE dataset, and about 7.5 per cent of publications and 9 per cent of citations for the ISI dataset. 61. There are well characterised fields such as Dentistry, Biology, Chemistry, Physics, and Civil, Mechanical and Chemical Engineering where the ISI datasets are very much smaller than the dataset derived from the RAE journal lists. This is almost certainly a consequence of the diversity of the research activity in academic departments in these large and well established areas, and the inclusive way the journal list has been created. This emphasises the difference between the discipline defined narrowly for standard bibliometric purposes and the nature of practical research management. 62. The overall effect seems to be for the RAE dataset to depress England's performance, possibly due to the multiple inclusion of more marginal or low ranking journals with low impact scores. This may be further evidence that the cognate RAE and ISI core categories correspond to the relatively high performance areas in a discipline. 63. For both datasets, impact scores are above world average for most of the natural sciences, but they decline in engineering and they are poor and variable in the social sciences; this variability may be due to the paucity of relevant data in these subjects. 64. The RAE based and ISI-based relative impact measures (impact = citations/outputs; relative impact = England impact/world impact) are in broad agreement. England scores better than 1.0 (higher proportion of cites than papers) for both datasets in 24 units (mostly broad-based sciences), and worse than 1.0 for both datasets in 12 units (mostly social sciences and arts). England's performance improves between the RAE and ISI measures in 18 of the 24 units, where both measure England's impact above world average, and declines in 8 of the 12 units where both are below world average. Both datasets produce low secondary measures of performance in the engineering and social science UOAs. Annex BUnits of Assessment in the RAE 19961. Clinical Laboratory Sciences Contact: Dr Jonathan Adams |
Skip to: main content | section navigation
You are in :
- Publications
- > 1998
- > Consultation 98/54
- > 1998