Evaluation of the Teaching and Learning Technology Programme
Coopers & Lybrand
Institute of Education
Tavistock Institute
June 1996
Reference M 21/96
The electronic version of this document contains the executive summary only. The complete printed document is available from the HEFCE, price £7.00.
Executive Summary
1. Coopers & Lybrand, the University of London's Institute of Education and the Tavistock Institute were jointly commissioned in January 1996 to carry out an evaluation of the Teaching and Learning Technology Programme (TLTP).
2. Key elements of our approach have included a detailed review of the software produced, a series of indepth case studies of a representative sample of projects, a number of workshops with TLTP participants and a programme of interviews with most of those prominently involved in the steering and management of the programme.
3. A number of factors have constrained our ability to make a full assessment of TLTP's impact. These include the fact that many TLTP materials are only just coming on stream (with others still in the developmental stage), the tight timescale and budget within which this evaluation has had to be completed, and our necessary dependence in many cases on weak secondary data. Further work is, we believe, needed to assess the full impact of TLTP in the medium to longer term.
TLTP products
4. We reviewed a substantial number of TLTP products, and found some to be first rate both in conception and implementation. Several projects, representing a wide range of subject disciplines, had produced imaginative and effective products; characteristics of these were that they were often supported by a clear model of learning and teaching and the place of the technology within it; they incorporated innovative design; and represented solutions to genuine problems. Such products often adopted a more exploratory, open, and constructive role for the student; they invariably envisaged a mode of use in which teaching was supported rather than replaced, and involved the creation of a learning situation which would have been difficult or impossible without the technology.
5. A surprisingly large number of products were in the style of software design which we have called "hyperstack", in which material is presented to students via hypertext screens and student input is evaluated by the system. We found such systems variable in quality and often lacking in imagination.
6. The effectiveness of systems was contingent upon two main factors: the mode of use envisaged and the subject discipline involved. We were more convinced of effectiveness in cases where material was envisaged as an adjunct to, rather than as a replacement for, teacherstudent interaction. Hyperstackstyle software appeared to be more effective in subject disciplines which require factuallybased, closedtask learning.
7. A number of the projects demonstrated a degree of naivete as regards the complexity of the educational task which they faced. Some projects managed to achieve success on the basis of individuals' ingenuity, existing networks of expertise and hard work. In only a small minority of cases can we say that projects had taken account of pedagogic issues in any systematic way: often existing research concerning the use of technology in higher education even in the relevant discipline was not used. Where we saw inspirational materials, we found that they had often emerged from a synthesis of computing, subject discipline and educational expertise. The presence of these three elements seemed to be a precondition for the production of materials that we could recognise as excellent.
TLTP's impact on institutions
8. Our fieldwork suggests that the "let 1,000 flowers bloom" approach adopted by TLTP has successfully involved a wide range of major teaching departments in the programme. Even where little institutional development in teaching and learning supported by technology had occurred prior to the programme, TLTP often led to such investments being made subsequently. Many HEIs have committed resources to the continuation of funding for TLTP activities and staffing.
9. This evaluation has confirmed the value of the institutional focus of TLTP, most strongly represented by the programme's twelve 'Institutional Projects'. Without a basic level of institutional resourcing, expertise and commitment, the uptake and integration of technologies in support of teaching and learning cannot be guaranteed. Future programmes and initiatives need to retain a significant institutional focus.
10. Institutions, and in particular the 'Institutional Projects', usually implemented TLTP through five main strategies: staff training, development and support; internal specialist units; awareness raising; strategic coordination; and technological infrastructure development. Particular strategies were themselves often part of a conscious effort to change the organisational culture of the HEI concerned.
11. Institutional support was a criterion for TLTP project selection. The strength of this support has proved variable: a number of projects did have very positive support demonstrated through funding or policy decisions; others, however, were simply allowed to work in the area provided that it did not affect their research output. It seems clear to us that good support from senior management, including a preparedness to make complementary institutional investment, will be an important determinant of the extent to which TLTP and other courseware materials will be used in the future.
12. The availability of specialist expertise in the uses of technology in teaching and learning was an important asset in HEIs. Projects could not, and did not always possess sufficient knowhow themselves. CTI centres and more recently TLTSNs can make an important contribution in providing such expertise and in the transfer of it between HEIs.
Dissemination and transfer
13. It is too early to describe or measure the uptake and usage of TLTP materials, although evidence is beginning to come through in some HEIs. In general, knowledge of TLTP was widely disseminated but this did not always lead to uptake. Active participation in a consortium appears to have made uptake more likely, which confirms the value of TLTP's consortiabased model for working.
14. Although comprehensive dissemination planning was rare, a great deal of effort went into dissemination through surveys, newsletters, roadshows and piloting etc laying the foundation for future dissemination activities. However, we are not aware of projects conducting cost/benefit analyses of such activities in order to target their work in the most effective way. Transportability of courseware was enhanced when the use of technology was relatively unsophisticated, intended to support and complement teachers and tutors rather than to offer a substitute. For more interactive and embedded courseware also to be transferable there would be a need for more shared curricula across HEIs. The tendency for TLTP projects to concentrate on first year courses, where introductory material is more likely to be common, supports this judgement.
15. Systematic dissemination plans were the exception rather than the rule: a developed dissemination plan should, we believe, have been a condition of funding. Many projects relied on the composition and membership of their consortia to guarantee dissemination. Experience from TLTP tends to support that of other programmes: dissemination planning needs to be built in from the beginning. In TLTP, dissemination plans were more often 'boltedon'.
16. Whilst accepting the principle that the courseware produced through TLTP should be free to all HEIs within the UK, we believe that this approach has not proved conducive to effective dissemination. This approach appears to have dampened the desire of a number of projects to put much effort into dissemination in the UK because there would be few returns to the investment. Many of our interviewees have questioned how the product can be maintained in the long term without being able to charge for it.
Productivity
17. An original TLTP objective was "to make teaching and learning more productive and efficient". Our fieldwork suggests that this objective became less prominent as the programme progressed; the emphasis instead has increasingly been placed on quality improvements. The academics to whom we have spoken have certainly been much more comfortable with the concept of working towards improving quality than improving efficiency.
18. Given the relatively early stage of development and dissemination of the software, it is too early to make any estimate of the potential impact of TLTP on efficiency across the HE sector. That said, many of the projects which we have seen nonetheless appeared to have produced, or be producing, outputs that could generate efficiency gains. Whether these efficiency gains will be fully realised depends on the application of the software, ie. pure enhancement or replacement. Some courseware was not designed to produce efficiency gains and never will.
Programme management
19. The decentralised management approach adopted for TLTP has provided for variety in approach. Projects have worked within their own frameworks and to their own objectives, rather than to any single central vision. So long as lessons including the factors that make for successful and unsuccessful projects are learned from the process and the outcomes of the programme to date, more structured and closely focused work can build upon the experimental nature of the first two phases.
20. The current TLTP coordinator has proved an invaluable source of support and advice to the projects and has attempted to put into place a number of monitoring and control procedures. However, the limited scale of centrally available staff resources has meant that central support, advice and management has been on a limited scale and thinly spread. The resulting degree of attention paid to some of the original programme objectives has resulted in their relative neglect, especially the objective of improving efficiency.
21. Central support was particularly lacking at the start of the programme with many projects working in isolation from each other leading to frequent "reinventing of the wheel" and little sharing of experience. The level of support and advice has subsequently improved considerably.
Project management
22. The selection criteria for projects did not accord significance to prior experience of consortium working. As a consequence, neither bidders not the central team realised the importance of expertise in such areas as project management. Many projects would have benefited from advice on setting up, structuring and management of the consortium way of working, as well as from more general guidance on managing an innovation process.
23. As a management model that took a number of different forms, consortia worked well in some projects and were less successful in others. This finding is not surprising since crossinstitutional collaboration was novel for the majority of project participants and many of those in lead partner coordinating roles had no prior experience of consortium working.
Further, consortia worked best with about 5 members and rarely well with more than 10; it also helped where the lead member was clearly identified.
24. We believe that many consortia would have been strengthened by the inclusion of a commercial partner, so providing a more real market perspective on material development and diffusion; we recognise that this would have required different financial arrangements.
Evaluation
25. Projects varied widely in the resources devoted to evaluation, the way in which evaluation was organised, the comprehensiveness of the evaluation activities that did take place and the evaluation expertise they were able to draw on. Whilst there were a few examples of good evaluations, the majority were limited in scope to fairly basic user feedback as part of pilot testing. Such variability may be attributed to the 'mixed messages' given to projects about the importance of evaluation by TLTP management, leading some to make required budget savings through curtailment of their evaluation plans.
26. Evaluation could have made more of a contribution to the cumulative learning from TLTP; its relatively low profile thus represents a missed opportunity. Although projects were encouraged to undertake their own local evaluations, there was no framework or mechanism whereby project evaluations could inform the overall direction of the programme or provide evidence of the achievement of objectives at a programme level.
Conclusions
27. At the outset of TLTP, the decision was made to fund a large number of projects across a diverse range of institutions with many different types of objective. With regard to raising awareness about CBL across higher education more generally, we believe that this approach has been particularly successful and that knowledge (and curiosity) about TLTP would not have been so widespread had the programme only funded a small number of highly specialised projects or institutions. Many valuable lessons have been learnt.
28. We believe that requesting institutions to develop consortia in order to encourage dissemination was the right approach albeit difficult and timeconsuming. The consortium approach has engendered greater acceptability of the curriculum content and teaching approach across different discipline areas.
29. All our interviewees have been clear that TLTP has added value in different institutions through:
- bringing forward developments that would otherwise have happened only later (often much later);
- increasing the scale of the activity concerned;
- improving its quality.
30. In some cases, the courseware developed through TLTP was a response to a recognised need. In other cases, our fieldwork suggests that the opportunity provided by TLTP was exploited to computerise material that was being taught or could have been taught adequately in other ways. Any successor programme needs, we think, to ensure that central resources are used for the most part to support TLTP type projects that respond to clearly identified needs. This approach needs, however, to be tempered by recognition of the need at the same time to encourage genuine innovation.
31. In summary we believe that TLTP's main achievements have been:
- to have produced a number of imaginative and first rate products;
- to have generated significant senior level support from within institutions in a number of instances which contributed to the success of the programme;
- to have raised the profile of the use of information technology in institutions and given credence to the work of a number of individuals who had been working in this field for some years;
- to have promoted a considerable volume of dissemination activity;
- to have provided a forum for discussion of difficult pedagogical issues at the subject level;
- to have considerably strengthened the central support initially available to the programme as the need was identified;
- to have succeeded, through using a "scatter" approach to funding projects, in helping to identify the relative merits of various different approaches to the use of information technology in teaching and learning;
- to have generated a pool of new learning, which will be helpful to future developments in the field of teaching using information technology.
32. The principal "learning points" that emerge from this evaluation seem to us to be the following:
- it is important to get the right mix of advisers in any central group coordinating the initiative;
- dissemination plans need to be more focused and targeted;
- evaluation needs to be built into the projects and the programme from the start;
- a robust and sufficiently large (in terms of personnel) system of monitoring and support for the projects is necessary for such exploratory initiatives;
- projects need to be assisted in the task of setting up mechanisms to collect information on gains in productivity once the project outputs come to be used;
- (in the view of most of our interviewees) technology in teaching needs to supplement, not replace the teacher.
The future
33. We have come across much anecdotal evidence during the course of this study that leads us to believe that IT has an important role to play in improving the future effectiveness and efficiency of teaching and learning.
34. Our work suggests that transferability and increased productivity are most likely to be achieved in the following circumstances:
- where there are large first year classes;
- where there are very different levels of preparation amongst students;
- where there are common curricula across a large number of institutions, such as for foundation classes or for courses with a significant element of accredited oursework;
- where students can be exposed to something that they would not otherwise experience in their teaching experience;
- where experiments require expensive equipment or are timeconsuming in the use of facilities.
35. The evidence at present on optimum areas for the application of technology, however, is patchy and not fully substantiated. Given that TLTP has now produced a range of teaching materials that departments are beginning to use, we believe that there is a strong argument to support a further study going beyond the scope possible in this evaluation to demonstrate types of benefit in relation to different types of use.
36 A number of complex questions need to be explored in the context of such a study, such as:
- What are the quantifiable benefits of CBL used in different scenarios in effectiveness and efficiency terms?
- How does CBL affect the role of a teacher?
- What type of investment in staff development and training is needed to support CBL?
- Is there a core curriculum across universities for some discipline areas that could be supported by CBL? If so, what are the implications for academic standards?
- What types of CBL are most effective in improving particular types of student learning?
37. Future funding of CBL developments should continue, we believe, to come from the centre; selffunding by institutions would run the risk of dissipated focus and reduced acceptability. The time has now come however, we believe, to move away from the "let 1,000 flowers bloom" approach (which provided a good starting point) to a more targeted approach. We make two proposals for further funding.
38. The first area for future funding is take up and implementation. In particular, existing materials need to be embedded into teaching and learning structures for students. This requires the addressing of issues such as cultural change within HEIs and departments, time for academics to work CBL into their teaching curricula, staff development and training and even a fundamental change in the role of teachers in some higher education institutions. We think funding should be available for this, but to the market not to the suppliers.
39. The second area for future funding would be a small amount of new courseware. We recommend that only a small number of highly targeted projects are funded centrally having been selected following a competitive process where a need can clearly be demonstrated, where it is clear that the high cost of development would preclude an institution being able to fund the venture alone, and where there is a well founded (and documented) expectation that significant benefits can be realised. We propose a two stage approach involving the funding of appropriate feasibility studies to assess need and the market.