Higher Education Funding Council for England
CHEMS
The Higher Education Consultancy Group
Department for Education Northern Ireland

Report 98/68

Evaluation of the Fund for the Development of Teaching and Learning

November 1998

This version of the document contains the Contents and Executive Summary only. The full document is available in Word or RTF formats.

Contents

1. Executive Summary

This report is the first evaluation of the Fund for the Development of Teaching and Learning (FDTL). The evaluation was commissioned by Higher Education Funding Council for England (HEFCE/Department of Education for Northern Ireland (DENI) and undertaken by The Higher Education Consultancy Group (THECG) and the Commonwealth Higher Education Management Service (CHEMS) working together as a partnership. It was undertaken in April and May 1998 and involved a wide range of evidence being collected from FDTL projects and other interested parties.

FDTL Objectives and Projects

FDTL was launched by the HEFCE and DENI in December 1995 (Circular 29/95) with the specific aims being "to stimulate developments in teaching and learning; and to secure the widest possible involvement of institutions in the take-up and implementation of good teaching and learning practice." This Circular invited institutions to bid for funds under Phase One of FDTL for the 15 units of assessment considered by the HEFCE Quality Assurance Division between February 1993 and June 1995.

Phase One funding was only open to those institutions where the unit of teaching quality assessment in the proposed host institution was either in receipt of an excellent TQA grade, or in receipt of a satisfactory grade accompanied by a commendation in the published quality assessment report for the specific area of development proposed in the application. Bids were sought from either individual institutions or within consortium arrangements, and only two areas of potential activity were explicitly ruled out: that "the FDTL programme is not intended to fund dissemination strategies arising from other Council funded initiatives such as TLTP", and that "the Council is not seeking to fund research on teaching and learning" (Circular 29/95). In the event 44 proposed projects were approved and received funding.

Phase Two of FDTL was announced in Circular 22/96 in December 1996, and extended funding to eight further subject areas assessed between spring 1995 and autumn 1996. The relevant TQA criteria had also been updated in line with the new methodology in use, and now required applicants to have a grade four rating in the aspect of provision for which FDTL funding was sought. Whilst FDTL objectives remained constant, in the transition between the two phases it is clear that greater attention was paid in Phase Two to dissemination and implementation issues and less to development per se, and also more encouragement was provided for consortium projects organised on a subject discipline basis. In the event 19 projects received funding under Phase Two, making 63 projects in total, of which two are in Northern Ireland funded directly by DENI.

The 63 funded projects represent considerable diversity in terms of both the proposed outcomes and the institutions in which they are based, with 52% being located within former UFC institutions (the pre-1992 universities), 40% in post-1992 universities, and 8% within the colleges of higher education and further education sectors. However, in view of the fact that there were approximately four times as many "excellent" assessment grades allocated to the pre-1992 universities, successful bids show a higher proportional take-up of funding in the former PCFC sector. When analysing projects by the main theme with which they are concerned it is clear that the majority of projects fall into three main areas: developing and using learning resources; personal, professional and transferable skills; and identifying and sharing good practice to improve teaching and learning. Projects also vary in terms of size, although in practice there appears to be a tendency for institutions to bid up towards the maximum funding permitted, and in future a higher proportion of smaller projects should be encouraged.

The FDTL programme is supported by a small HEFCE secretariat based in Bristol, and a National Co-ordination Team (NCT) of 2.3 full-time people plus administrative support based in the Centre for Higher Education Practice (CEHEP) at the Open University. This evaluation assesses the work of both the secretariat and the NCT and notes the general praise for their activities provided by the majority of projects.

The Effectiveness of FDTL and Achievements to Date

So far as the effectiveness of the management of FDTL is concerned, the review considers this both in its own terms and also compares outcomes with a number of other centrally funded initiatives, including the Teaching and Learning Technology Programme (TLTP). It concludes that, in general, within existing resources central management is both effective and responsive to the needs of projects. However, within individual projects and host institutions the quality of management varies greatly, from excellent to poor, and particular difficulties are noted with the operation of some - not all - large consortia. The need for effective project management is a consistent finding from many Council evaluations, and that conclusion also applies to FDTL.

The report reviews the selection process used to consider applications for funding, and notes changes between Phase One and Phase Two. It concludes that the selection process is demonstrably fair and equitable, and more than meets any reasonable test of public scrutiny. Where bids were rejected and have challenged the grounds for such rejection, we have been unable to find any information that would suggest that these proposals were unfairly treated. However, some weaknesses remain in the selection process and recommendations are made to address them. The fact that NCT was unfortunately not in place at the start of Phase One to strengthen the initial selection process has led to a situation "where projects which would ideally have been recast somewhat before funding are now somewhat reluctant to depart from the agreed project plan" (NCT’s first annual report to the FDTL Advisory Group).

The evaluation concludes that the value for money associated with Phase Two projects is, in general, likely to be greater than for Phase One, because of the more rigorous selection procedures subsequently introduced, although it is too early to reach firm conclusions in this area. Certainly, the report concludes that the Phase One selection procedures were relatively generous in the interpretation of FDTL criteria, and that a number of projects were approved which might not have been chosen under Phase Two. Although the initial phase of any such innovation is likely to be somewhat experimental, this does suggest the need to maintain - and ideally enhance - considerable rigour in the selection process if the objectives of FDTL are to be met.

So far as the overall achievements of both the Fund as a whole and the individual projects are concerned, despite a number of operational problems - including the late start of many projects in Phase One - FDTL appears to be achieving its objectives so far as they are defined, and are also clear, to the higher education community. In general, the rationale for FDTL is sound given the learning and teaching and funding strategies of HEFCE, although it needs to be acknowledged that the effectiveness of a disciplinary basis for spreading good practice in teaching and learning remains unproven, despite some encouraging evidence from a recent review of subject networks supported by a DfEE initiative. However, some aspects of the objectives of FDTL and associated policy (in particular the requirement for dissemination) need to be clarified, and recommendations are made to achieve this.

The Final Report of the Evaluation of TLTP referred to "little more than the ‘leap of faith’ taken at the start of TLTP" (about the role of IT in teaching and learning). The origins of FDTL were slightly firmer but a number of issues still remain about the clarification of its role, and the assumptions on which it is based are still unproven. That evaluation of TLTP commented critically on "an overwhelming naivety as regards the complexity of the educational task which projects faced." In general, FDTL projects - although often under pressure - do not demonstrate such naivety, partly because of the strong and effective support of NCT, and partly because most projects involve staff interested in the process of teaching and learning rather than in the application of technology. Moreover, the Council and NCT appear to have learned a number of lessons from previous initiatives which have ensured a higher level of central guidance and co-ordination of projects.

Therefore we conclude that, in general, FDTL has so far achieved what it set out to do, a particularly laudable outcome in the context of NCT having been established well after the Fund started operating. Nonetheless a number of serious questions remain to be answered in future evaluations, and the clarification of the role and purpose of the Fund in the context of the Council’s new learning and teaching strategy will be an important starting point for any future analysis.

However, the real impact of FDTL can only be judged in the future on the basis of the effectiveness of the dissemination and implementation of the funded projects, and, as is clear from other initiatives both in the UK and overseas, such dissemination is fraught with difficulty. The effectiveness of dissemination to date is reviewed, and evidence produced of widely varying practice, and the report concludes that, if dissemination is to be successful, considerable assistance to projects will be required from NCT together with clearer guidance from the Council about what constitutes effective dissemination and transfer and how to achieve it.

The report notes a variety of support within institutions for projects in that, although most host universities have expressed public commitment for projects to the Council in the preparation of the bid document, in some cases significant institutional barriers to successful implementation exist. One particularly common problem identified by projects is the effect of the research assessment exercise which is widely perceived to be having a detrimental effect on the willingness of some institutions to extend good practice in teaching and learning and to encourage innovation. However, this is not true of all projects, and, encouragingly, a majority are reported as having some plans for continuation of activities after Council funding stops, although the realism attached to such plans varies widely, from the well considered to the very optimistic.

In Chapter 8 we review briefly the funding strategy adopted by the Council in support of FDTL, and consider whether value for money (VFM) is being achieved. The strategy is considered appropriate given the intended funding methodology of HEFCE to support its overall learning and teaching strategy, and, although judgements about VFM must be tentative at such an early stage in the initiative, it appears to be appropriate in relation to the level of expenditure committed. However, it is clear that the VFM achieved by projects is likely to vary widely, and greater clarity in funding arrangements is proposed for Phase Three.

The Future and Phase Three

In Chapter 9 the report looks forward and recommends the adoption of Phase Three of FDTL, and makes a number of detailed proposals for the clarification of various policy issues facing the Fund. Other recommendations concern modest changes to the system for selecting projects; the adoption of slightly more flexibility in the types of projects considered for funding, including encouraging smaller projects; a tightening of the criteria concerning the number of projects funded to produce learning materials; the adoption of a small and very selective continuation fund for existing projects; and changes to financial procedures, including the introduction of a small management fee for host institutions, together with a provision for the Council to implement a modest financial retention to encourage project completion on time. The report also notes the importance of future evaluation, and identifies key issues for evaluations in the next two years.

Finally, in Appendix C the experience of some countries overseas is reviewed in undertaking initiatives similar to FDTL, and it is noted that this work might usefully be expanded by NCT during the coming year better to inform UK activity.