HEFCE : Publications : 1996 : 6/96 - Quality Assessment Division Assessors' Handbook

HEFCE

Quality Assessment Division

Assessors' Handbook

October 1996 to September 1998

Reference M 6/96

Published March 1996

Contents

1. Introduction

2. Overview of the Quality Assessment Method

Introduction
Peer Review
Self-Assessment
Assessment against the Subject Provider's Aims and Objectives
Core Set of Aspects of Provision
The Assessment Visit
The Graded Profile
The Overall Summative Judgement
The Quality Assessment Report
Assessment and Accreditation

3. The Assessment Process

Preparation for Assessment

Introduction
Advance Planning
Analysis of the Self-Assessment
The Assessment Team
The Preparatory Meeting
Preparation by the Assessment Team
Practical Arrangements for Assessment Visits

The Assessment Visit

Introduction
The Programme for an Assessment Visit

Gathering Evidence: General Considerations
Observation of Teaching and Learning
Sampling Student Work
Meetings with Staff
Meetings with Students, Former Students and Employers
Scrutiny of Documentation
Review of Student Learning Resources
Making the Judgements: General Considerations
Assessment Team Meetings
Written Summaries
Grading the Aspects of Provision

The Oral Feedback Meeting
Monitoring and Evaluation

The Assessment Reports

Quality Assessment Reports
Subject Overview Reports

4. Annexes

A Quality Assessment Aide Memoire
B The Preparatory Meeting Agenda
C Documentation to be Provided by the Institution
D A Typical Schedule of an Assessment Visit Programme
E Observation Note
F Student Work Review Note
G Aide Memoire for Meeting with Student Groups
H Assessment Team Meetings
I Grading the Aspects of Provision
J The Oral Feedback Meeting Agenda

5. Appendix

Policy Statement on Equal Opportunities in Quality Assessment

Introduction

1. The purpose of this Handbook is to describe the method and procedures for the assessment of the quality of education provided in institutions funded by the Higher Education Funding Council for England (HEFCE). The Handbook informs subject specialist assessors, reporting assessors and institutions about the operation of the assessment method published by the HEFCE in Circulars 39/94 and 26/95. It describes the protocols, procedures and practical organisation of an assessment, from advance preparation through to publication of the quality assessment report by the Council. The Handbook is structured in three main sections: an Overview of the Quality Assessment Method, the Assessment Process, and a set of related Annexes. The Council's Policy Statement on Equal Opportunities in Quality Assessment is included as an Appendix.

2. In accordance with the requirements of Section 70 of the Further and Higher Education Act of 1992, the Council's purposes in carrying out the assessment of the quality of higher education (HE) are:

a. To secure value from public investment.
- i. By ensuring that all education for which the HEFCE provides funding is of approved quality, and by encouraging speedy rectification of major shortcomings in the quality of education.
- ii. By using quality assessment judgements to inform funding.
b. To encourage improvements in the quality of education through the publication of assessment reports and subject overview reports, and through the sharing of best practice.
c. To provide, through the publication of reports, effective and accessible public information on the quality of the education for which the HEFCE provides funding.

3. Visits under the current quality assessment method began in April 1995. The current method was described in HEFCE Circular 39/94 'The Quality Assessment Method from April 1995'. Institutions were informed of a timetable for completing the full cycle of quality assessments in HEFCE Circular 20/95 'The Forward Programme for Quality Assessment', and of the arrangements for 1996 to 1998 in HEFCE Circular 26/95 'Quality Assessment between October 1996 and September 1998'.

4. If you would like to make any comments on this Handbook please write to:

The Director of Quality Assessment
HEFCE
Northavon House
Coldharbour Lane
BRISTOL BS16 IQD.

2. Overview of the Quality Assessment Method

Introduction

5. In carrying out its statutory obligation, the HEFCE has developed a quality assessment method which includes both self-assessment and scrutiny through external peer review. The assessment method evaluates the quality of educational provision within a subject area as defined by a unit of assessment and is focused, at the level of the subject, on the quality of the student learning experience and student achievement. Taught programmes of study at all levels (foundation, undergraduate and postgraduate) are assessed.

6. The following are the main features of the method:

peer review
self-assessment
assessment against the subject provider's aims and objectives
core set of aspects of provision
assessment visit
graded profile
overall summative judgement at the threshold level
quality assessment report.

Peer Review

7. Assessments are carried out by a team of subject specialist assessors led by a reporting assessor. It is the subject specialist assessors' main responsibility to gather evidence and to make judgements on the quality of education provided. Subject specialist assessors are drawn mainly from higher education, although nominations and applications from industry, commerce and professional bodies are actively sought. Subject specialist assessors are trained by the Council in conjunction with the Universities and Colleges Staff Development Agency (UCoSDA) before their first assessment visit. In addition to attending the subject specialist assessor training course led by UCoSDA, each newly appointed reporting assessor is provided with training in managing visits and other relevant skills, such as IT skills, and is assigned a mentor who is an experienced reporting assessor.

Self-Assessment

8. The self-assessment is an evaluation by the subject provider of the quality of the student learning experience and student achievement, measured against the aims and objectives that the subject provider sets for the education of its students in that subject. This evaluation is organised within the structure given by the core set of aspects of provision and should be supported by evidence. A self-assessment should discuss both strengths and weaknesses in the provision. Where weaknesses are acknowledged, the subject provider is encouraged to discuss the issues and the steps taken to improve quality. The Council's full guidance on the preparation of self-assessments is included in Circular 26/95.

Assessment against the Subject Provider's Aims and Objectives

9. The quality assessment method is designed to take as full account as possible of the diversity of the HE sector. The subject provider's aims and objectives are included in the self-assessment. The statement of aims and objectives provides the framework for the assessment. In reaching judgements on the quality of the provision, assessors evaluate both the extent to which the student learning experience and student achievement in each of the aspects of provision contribute to meeting the objectives set by the subject provider, and the extent to which the objectives set, and the level of attainment of those objectives, allow the aims set by the subject provider to be met.

10. Circular 26/95 includes the Council's guidance for institutions on the use of the terms aims and objectives for the purpose of quality assessment. The aims express the subject provider's broad purposes in presenting each programme of study in the subject. The objectives set out the intended student learning outcomes and student achievements that demonstrate successful completion of a programme of study. Circular 26/95 reminded institutions particularly that:

a. A statement of aims should focus at the level of the subject provision, students and programmes of study. Discrete aims for each programme of study should be specified, and the aims of different levels of study (diploma, undergraduate, or taught graduate) should be differentiated. Where reference is made to institutional goals, the statement should establish the link between these and the subject aims.
b. A statement of objectives should communicate specific intentions for the student learning experience and learning outcomes. A statement of aims and objectives should articulate the relationship between specific intentions (objectives) and broad purposes (aims).
c. In determining whether or not a statement of aims and objectives is clear enough to enable an assessment visit to be undertaken, consideration is given to the expression of, and consistency between, the aims and objectives, and to whether there is sufficient differentiation between broad purposes and specific intentions, along the lines set out in the Council's written guidance. Where a re-statement of aims and objectives is necessary, this has to be carried out before the preparatory meeting can take place.

Core Set of Aspects of Provision

11. The Council has developed a core set of aspects of provision to provide a common structure for each stage of the quality assessment process: the self-assessment, the assessment visit, the assessment judgements and the assessment report. The six aspects of provision are:

Curriculum Design, Content and Organisation
Teaching, Learning and Assessment
Student Progression and Achievement
Student Support and Guidance
Learning Resources
Quality Assurance and Enhancement.

It is intended that the aspects of provision should be sufficiently broad and inclusive to enable providers and assessors to give full weight to the particular features of the provision in different contexts. Broad indications of the issues that might be addressed within each individual aspect are listed in Circular 26/95. These issues are grouped within each aspect according to whether they describe the key features of the provision (Key Features) or whether they are matters which inter-relate and influence more than one aspect of provision (Relationships). Subject providers may use the aspects to address issues in addition to, or other than, those listed under them in order to give an accurate description and evaluation of their provision.

The Assessment Visit

12. The purpose of the assessment visit is to gather, consider and verify the evidence of the quality of education, in the light of the subject provider's aims and objectives, and to establish a graded profile and an overall judgement on the quality of that provision. Details of the assessment visit are provided in section 3 of this Handbook.

The Graded Profile

13. The graded profile shows the extent to which the actual student learning experience and student achievement demonstrate that the aims and objectives set by the subject provider are being met. The graded profile is created by applying the assessment scale to each of the six aspects of provision.

14. There are four numerical points on the scale - 1,2,3,4 in ascending order of merit. A grade of 2 or better in an aspect means that the aspect makes at least an acceptable contribution to the attainment of the stated objectives and that the aims are at least broadly met. A grade of 1 means either that the aspect makes an inadequate contribution to the attainment of the objectives, or that the objectives do not provide students with the experiences and achievements that would support a judgement that the aims were being met.

15. The assignment of the grade is a matter for the professional judgement of the specialist assessors drawing on the evidence from both the self-assessment and the assessment visit.

The Overall Summative Judgement

16. The overall summative judgement is derived from the profile. All the profile elements are treated as of equal weight. A profile with all elements graded 2 or better will be reported as 'quality approved'. A profile with one or more aspects of provision graded 1 will be subject to re-assessment within a year and the following procedures will apply:

a. An assessment report containing the decision of the first assessment will be published. A re-assessment will take place within 12 months. During the period between the first assessment and the re-assessment the normal Council funding arrangements continue to apply.
b. If, after re-assessment within 12 months, the profile still contains one or more aspects of provision graded 1, the education will be recorded as of unsatisfactory quality and funding will be withdrawn in whole or in part.

17. Where a graded profile includes one or more gradings of 1, institutions may make representation to the Council's Quality Assessment Committee, using the procedure outlined in Circular 26/95.

The Quality Assessment Report

18. The judgements are given in an oral feedback from the assessment team to the institution at the end of the assessment visit. A quality assessment report is published after each assessment visit and is the main documented outcome of the quality assessment process.

Assessment and Accreditation

19. The potential relationship between quality assessment and accreditation by professional bodies is being examined by the Council and a variety of professional bodies, with a view to exploring ways in which these processes might be brought closer together. Possible collaborative arrangements range from sharing documentation to fully amalgamated visits for the purposes of judging quality as well as accrediting provision. Pilot projects to determine the feasibility of collaboration between assessment and accreditation will begin in 1996.

3. The Assessment Process

Preparation for Assessment

Introduction

20. The preparation for assessment may be divided broadly into five stages: advance planning and self-assessment, analysis of the self-assessment, composition of the assessment team, the preparatory meeting, and preparation by the assessment team. The practical arrangements made for assessment visits are also important. The preparations for assessment are most effective when they are undertaken with full co-operation between the institution, the officers of the Council, the reporting assessor and the subject specialist assessors.

Advance Planning

21. Advance planning is undertaken by the Council in conjunction with institutions. The HEFCE Circular 26/95 'Quality Assessment between October 1996 and September 1998' sets out full details of the arrangements for advance planning for the 16 units to be assessed in this assessment round, including a timetable for receipt of information from institutions and for scheduling assessment visits in 1996-97. The precise timing of assessment visits is a matter for discussion and agreement between the institutions and the Council's officers. An Information Form is completed by each institution for each unit of assessment in which it has provision. This includes information on the size, scope and nature of the provision to be assessed and the institution's preferred timing for the assessment. This advance information provides a basis for further discussions with institutions to plan and agree the timing of individual visits.

22. Institutions are asked to prepare the self-assessment in time to be submitted to the Council at least three months before the date of the visit. The Council's guidance to institutions on preparation of the self-assessment is set out in Circular 26/95. Receipt of self-assessments for the assessment period in 1996 to 1998 is being staggered in relation to the timing of the visits. Staggered receipt of self-assessments helps to spread the workload of preparation within institutions and to ensure that the information in each self-assessment is as current as possible.

23. Circular 26/95 invited institutions to make proposals for joint or parallel assessment of provision in the units to be assessed. In joint assessments, one self-assessment and one assessment visit cover more than one unit of assessment. There is one assessment team and the visit results in one graded profile and one published report covering all the units assessed jointly. In parallel assessments, up to three assessment visits take place in one institution at the same time. There are separate self-assessments and separate assessment teams. Each parallel assessment visit leads to a separate graded profile and published report. Separate protocols are available for joint and parallel assessment visits, covering the preparatory meeting and the arrangements during the assessment visit. These are sent by the Quality Assessment Division (QAD) to relevant institutions and to all assessors.

Analysis of the Self-Assessment

24. The self-assessment is the primary guide for assessors in setting the priorities and planning the programme for the assessment visit. It is analysed prior to the assessment visit. The main functions of this analysis are to determine whether or not the aims and objectives are clear and are of an acceptable length (up to 500 words) for publication in the quality assessment report, and to develop the agenda for the assessment visit.

25. A judgement on clarity of the aims and objectives is made by asking the following questions:

do the stated aims express broad purposes for each programme of study in the subject and state what abilities and aptitudes can reasonably be expected of the successful graduate?
do the stated objectives set out specific intentions for the student learning experience and learning outcomes for each programme of study in the subject?
are the aims and objectives expressed with sufficient clarity to form a framework for an assessment?
are the aims and objectives consistent with those of the institution, where explicit reference is made to the latter?
is there a clear relationship between the stated aims and the stated objectives?

26. If necessary, the subject provider may be asked to re-submit the statement of aims and objectives. Once clarification has been achieved, the preparatory meeting is held and the arrangements for the assessment visit proceed.

27. The evaluation section of the self-assessment is analysed by the reporting assessor, using the Quality Assessment Aide Memoire (Annex A), before the preparatory meeting with the subject provider. At the preparatory meeting the reporting assessor may wish to clarify her/his understanding of the self-assessment, in order to achieve the best possible preparation for the assessment visit.

The Assessment Team

28. A reporting assessor leads each assessment team and is responsible for the co-ordination and management of assessment visits, and for ensuring that these are conducted within the Council's guidelines. This involves preparation for the assessment visit in liaison with the institution, co-ordinating the work of the subject specialist assessors on the assessment team, ensuring that evidence is gathered and tested and that the judgements reached are robust, providing oral feedback to the institution at the end of the visit, and preparing the quality assessment report after the visit. The reporting assessor is not normally a specialist in the subject being assessed.

29. Subject specialist assessors are recruited mainly from higher education, but may also include those with industrial, commercial or professional experience of the subject being assessed. The responsibilities of subject specialist assessors include: reading the self-assessment and other documentation provided in advance of the visit; participating in the assessment visit to gather and share evidence and make judgements on the quality of education provided; and commenting on the draft quality assessment report after the visit.

30. The experience and expertise of the subject specialist assessors reflect, as far as is practicable, both the nature and scope of the education provided in the specific institution and the diversity of provision in the subject across the sector. The number of assessors in the team reflects the size and range of the education provided, taking due account of complexities such as operations on more than one site, modular schemes, franchised and other off-campus courses or local learning support centres based in England or Northern Ireland. The minimum number of subject specialist assessors for each assessment team is three.

31. The central institutional contact person and the reporting assessor are informed of the composition of the team as much in advance of the assessment visit as is possible. The composition of the assessment team may be modified in the light of discussion between the QAD and the institution receiving the assessment visit. Whenever possible, details of the proposed assessment team are communicated to the institution in advance of the preparatory meeting. The final composition of the team is confirmed a minimum of three weeks before the visit.

32. Subject specialist assessors may be asked to undertake up to six assessment visits over the two-year assessment period. Assessors are asked to complete an availability form on which they declare any conflicts of interest with institutions. Institutions are invited to inform the QAD of any perceptions they may have of possible conflicts of interest with members of the assessment team. Examples of conflicts of interest include: recent external examining, candidature for employment, and recent employment at, or graduation from, the institution to be visited. The QAD informs subject specialist assessors as early as possible of the visit allocations.

The Preparatory Meeting

33. The preparatory meeting is normally held a minimum of two months before the assessment visit. The designated reporting assessor is informed of the visit dates after these have been confirmed between the QAD and the institution. The reporting assessor is then able to arrange a preparatory meeting through the nominated central institutional contact.

34. The main purpose of the preparatory meeting is to discuss:

any necessary clarification of the self-assessment
the scope and nature of the provision to be assessed, on the basis of the completed Information Form
the range of teaching and learning sessions available for observation during the visit, and the extent to which this will enable a representative sample to be visited
the documentation to be sent to assessors prior to the assessment visit, the timetable for its receipt, and the documentation to be made available during the visit
meetings to be arranged in advance
the practical arrangements for the visit.

During the preparatory meeting the subject provider is asked to nominate a member of staff to act as a subject contact with whom the reporting assessor liaises to carry forward the arrangements for the visit. The Agenda for the Preparatory Meeting is set out in Annex B. Following the preparatory meeting, the reporting assessor writes to the institution, copying to QAD, in order to confirm his or her understanding of the arrangements agreed for the assessment visit.

35. The assessment process requires that the institution make available accurate, representative and accessible information and evidence. It is the responsibility of the institution to ensure that the assessors receive the self-assessment and other documentation not less than two weeks before the assessment visit. Annex C summarises the range of documentation and information which is usually provided prior to and during an assessment visit. Any other communication between the subject specialist assessors and the institution, before or after the assessment visit, should be channelled through the reporting assessor.

Preparation by the Assessment Team

36. Subject specialist assessors are contacted by the reporting assessor to discuss arrangements for the visit and to agree the responsibilities within the team. Subject specialist assessors normally assume responsibility for the co-ordination and sharing of evidence relating to one or more aspects of provision. They may also maintain oversight of one or more programmes of study. The reporting assessor provides a briefing in writing to all team members at least two weeks before the visit. The assessors receive information on the administrative arrangements for the visit directly from the QAD, including details of accommodation and expenses claim forms.

37. Subject specialist assessors are asked prior to the visit to produce a brief written commentary on assigned aspects, based on the self-assessment and the other advance documentation. The commentary should make full reference to the aims and objectives of the subject provider. In producing this commentary, assessors may find it helpful to refer to the questions in the Quality Assessment Aide Memoire (Annex A). Assessors' commentaries are discussed at the first assessment team meeting.

Practical Arrangements for Assessment Visits

38. Arrangements made by the QAD in advance of the assessment visit include the provision of:

hotel accommodation
travel and subsistence expenses
secretarial support
a portable computer and printer - provided in the base room, on loan to the reporting assessor by the HEFCE
coffee/tea
lunch - may be provided by the institution at meetings with students, employers, etc. Otherwise, a staff refectory is normally available, or assessors may arrange for sandwiches to be delivered

39. Subject specialist assessors who bring personal computers that are compatible with HEFCE-provided equipment may transfer written summaries electronically. However, assessors are expected to conform to procedures designed to protect against damage and computer viruses, which are outlined in the IT guidelines supplied to all assessors and are regularly updated.

The Assessment Visit

Introduction

40. The main purpose of an assessment visit is to enable assessors to gather sufficient evidence on the provision in a subject to allow them to form a collective judgement on the quality of education, as measured against the aims and objectives of the subject provider. An assessment visit will normally last three and a half days, although in certain circumstances it may be extended over a longer period. A common range of activities is undertaken during visits, but the precise order and balance may be adapted to reflect the teaching timetable, the nature of the provision and the priorities derived from the assessors' analysis of the self-assessment. It is intended that the assessment visit should be conducted in a spirit of dialogue and co-operation between the institution, the subject staff and the assessment team.

The Programme for an Assessment Visit

41. An outline programme is agreed at the preparatory meeting, so that a confirmed timetable of major meetings can be included in the advance documentation which is sent to assessors by the institution. A Typical Schedule for an Assessment Visit Programme is set out in Annex D.

42. The activities carried out by an assessment team during the visit include the following:

observation of the various forms of teaching and learning being carried out during the assessment visit (including direct observation of classroom, seminar, workshop, tutorial, laboratory and placement situations as appropriate)
sampling of student work (examination scripts, coursework, projects and dissertations)
meetings with academic and administrative/support staff
meetings with students, former students and employers where appropriate
scrutiny of institutional and course documents, reviews, and reports, including external examiners' reports
review of the available student learning resources
meetings of the assessment team
preparation of draft written summaries.

43. The reporting assessor maintains an overview of the range and balance of assessment activities, and will guide subject specialist assessors in apportioning their time. The achievement of an appropriate balance between the observation of teaching and learning, reading documentation, sampling student work, meeting staff, students and other stakeholders, producing written summaries and arriving at a collective judgement requires planning and co-ordination during the assessment visit. It is essential that the overall range and balance of activities undertaken enables the team to develop a robust evidence base for the judgements to be made in relation to each of the six aspects of provision, and subject specialist assessors are expected to agree individual timetables with a view to achieving this evidence base. Sufficient time should be allocated for scrutiny of documentation, sampling student work, and preparing written summaries of evidence and evaluation, as well as for undertaking observations of teaching and learning and attendance at meetings.

44. On arrival at the institution, the assessment team generally meets before being introduced to the head of the institution and/or other senior representatives. It can be helpful if the institution's representatives inform assessors of any matters which may be important to their understanding of the subject being assessed from the institutional perspective.

45. An initial meeting with available subject staff on the first afternoon of the visit allows a similar opportunity, at the level of the subject, for the staff representatives to make a brief presentation on the provision to be assessed and to inform assessors of any developments since the self-assessment was undertaken. The meeting with subject staff also provides an opportunity for the reporting assessor to remind staff, with team members present, of the quality assessment method, the purposes of the assessment visit, and the protocol for observations of teaching and learning. Student representatives may be included in the initial meeting with subject staff if the institution wishes.

46. The assessment visit programme includes a range of meetings between members of the institution and assessors to consider the various aspects of the provision. Throughout the visit the assessment team meets daily to discuss its programme and its findings, normally at the end of the afternoon/early evening. On the last day the assessment team meets to finalise the oral report to be presented to the institution at the feedback meeting. Contact with senior representatives of the institution is generally confined to the formal meetings at the commencement and end of the visit.

Gathering Evidence

General Considerations

47. The assessment is focused on the extent to which the actual student learning experience and student achievement demonstrate that the aims and objectives set by the subject provider are being met. In gathering evidence, assessors should refer to the Quality Assessment Aide Memoire (Annex A). All assessors are expected to identify, share, consider and evaluate evidence related to all aspects of provision.

48. Assessors should keep notes of meetings with staff and students, of teaching and learning observations and of comments on student work. The circulation of notes within the assessment team assists in the process of developing a collective evidence base for the judgements made. These are referred to by assessors in preparing written evaluations of the quality of provision and are retained by the reporting assessor at the end of the visit.

Observation of Teaching and Learning

49. The arrangements for the assessment of teaching and learning during the visit should reflect the nature of the provision and include observations of a sample of scheduled sessions. Different types of teaching and learning should be covered as appropriate to the particular subject area, such as lectures, seminars, tutorials, laboratory practicals, workshops and vocational placements. All subject specialist assessors normally participate in the observation of teaching and learning. The subject specialist assessors should try to observe a reasonably representative range of teaching and learning sessions, covering each programme of study, and each level and mode of study. Assessors should observe as many different members of staff as possible, including part-time and temporary staff. In order to achieve an effective sample, the reporting assessor may prepare a preliminary schedule of observation sessions for discussion within the team.

50. The reporting assessor supplies the subject contact with a timetable of assessors' observations of teaching and learning sessions at the start of each day. It is normally left to senior staff, such as a head of department or programme leader, to decide whether or not to inform individual staff of a planned observation. On occasion, variation to the intended timetable of observations may be necessary.

51. An assessor will try to meet the member of staff responsible for a teaching/learning session before it commences in order to introduce himself or herself, to ascertain the overall objectives of the activity and to determine how students are intended to learn from it. Understanding the overall purpose of the teaching/learning session is highly important. For example, a lecture delivered for the express purpose of transmitting information will be structured differently from one designed to stimulate extensive further reading. Assessors will not make comments during a lecture, seminar or tutorial and should not be intrusive or engage directly in the teaching. If the assessor attends only part of a session which extends for more than one hour, s/he should always inform the member of staff in advance to that effect.

52. On occasion, students engaged in learning activities in practical sessions and independent learning may be asked by assessors to talk about their learning experiences and how the activity being observed fits within their wider programme of study. As with other observations, assessors should endeavour to meet the appropriate member of staff to ascertain the overall objectives of the session and provide feedback wherever possible.

53. A standard Observation Note, included here as Annex E, is completed for each teaching/learning session observed and is shared within the assessment team. The subject specialist assessor training programme includes specific sessions on evaluating teaching, learning and assessment, in addition to the guidance provided in this Handbook.

54. Judgements about the quality of teaching and learning sessions are based on the direct observations of the subject specialist assessor. In coming to a conclusion about the quality of an individual teaching/learning session, subject specialist assessors will wish to consider a number of features, including objectives, planning, content, pace, learning resources and student participation. For example, effective delivery may be seen in sessions relying on 'chalk and talk' and in classes involving extensive use of computer-generated material, other audiovisual aids or handouts. In making their judgements about an individual session, subject specialist assessors evaluate whether the teaching and learning approaches and the materials used are appropriate to the intended learning outcomes.

55. Subject specialist assessors grade the teaching and learning sessions observed on a scale of 1-4, with 1 lowest and 4 highest:

Grade 1: The session fails to make an acceptable contribution to the attainment of the learning objectives set.
Grade 2: The session makes an acceptable contribution to the attainment of the learning objectives, but significant improvement could be made.
Grade 3: The session makes a substantial contribution to the attainment of the learning objectives, but there is scope for improvement.
Grade 4: The session makes a full contribution to the attainment of the learning objectives.

56. After attending a session, an assessor must provide a brief oral feedback to the member of staff, even if this necessitates making an appointment for a later time during the visit. Oral feedback is confidential to the member of staff, should always be given privately after the session and never in front of students. The purpose of the feedback is to offer constructive comment on the observations made rather than to prescribe preferred practice. Assessors will give the grading of the session if requested by the member of staff. The anonymity of staff must be preserved by assessors in all written reports and, especially, in discussions with other members of the institution.

57. Observations of teaching and learning sessions are undertaken predominantly by subject specialist assessors. There may be some occasions when it is appropriate for a reporting assessor to attend a teaching/learning session, but where this occurs the class will not be formally observed or graded by the reporting assessor. If a reporting assessor attends a session, it will be:

consistent with her/his responsibilities in managing the assessment visit
agreed with the institution
because it enables a reporting assessor to gain a better understanding of the particular subject context
followed as a matter of courtesy by a brief discussion (but not feedback) with the member of staff teaching.

Sampling Student Work

58. Assessment of the quality of education requires assessors to make judgements about the quality of student learning and about the means by which student progress and achievement is assessed formatively and summatively. This review of student work complements the role of internal and external examiners. Subject specialist assessors sample student work to evaluate the quality of assessment design, the quality of feedback to students and student achievement of the intended learning outcomes.

59. The sample, agreed at the preparatory meeting, typically covers: a representative range of attainments/marks, a range of levels and specialist topics in the subject, continuous assessments/coursework, practical/lab work and projects, examination scripts and dissertations. The sample should be accompanied, where they are in use, by marking and feedback sheets and assessment criteria (see Annex C).

60. Where the arrangements made by the subject provider include oral feedback to students rather than, or in addition to, a written commentary, assessors will take this into consideration. A Student Work Review Note, included as Annex F, should be completed for the sample of student work associated with each individual course or module evaluated.

Meetings with Staff

61. As a general principle, assessors will focus their attention at the level of the subject and address issues at a more central institutional level only when necessary. It may be helpful to structure meetings with staff around individual aspects of provision so that, for example, assessors might meet jointly with academic counsellors, personal tutors and representatives of the central student welfare services to gain an overview of student support and guidance; or jointly with a departmental library representative, subject librarian and representative of the central computer services and/or other relevant staff to discuss learning resources in the subject. In developing specific agendas for these meetings, assessors may find it helpful to refer to the Quality Assessment Aide Memoire (Annex A).

62. Daily meetings between the reporting assessor and the subject contact provide an opportunity to highlight particular issues which the assessors wish to explore, and for the subject contact to raise any matters concerning the conduct of the assessment. Individual assessors are encouraged to make any request for additional documentation or for further meetings through the reporting assessor. The reporting assessor will liaise with the subject contact in making these arrangements.

63. A general meeting between the assessment team and subject staff during the latter part of the visit provides assessors with an opportunity to seek clarification of points. It also enables staff to respond to assessors' queries.

Meetings with Students, Former Students and Employers

64. The views of students, former students and others with an interest in the quality of the provision, such as employers, are an important part of the evidence. Whenever possible, all of the assessment team should attend the meetings, which typically last an hour. One team member should take responsibility for making notes and ensuring that these are circulated to others in the team. Meetings with students are generally chaired by the reporting assessor. At the beginning of the meeting, the reporting assessor will introduce the assessors and provide a brief outline of the quality assessment method. It should be emphasised that the views of students are valued, as quality assessment focuses on the student learning experience and student achievement. Those attending should be assured that the contributions made will not be attributed to individuals. Assessors should also state clearly that their final evaluation of the quality of provision will be based on a wide range of evidence, of which comments made by present and former students and employers form one part.

65. Discussion should be structured and all present should be encouraged to contribute. If a meeting has been arranged over a buffet lunch, informal discussion should be kept relatively brief and the meeting should then be convened formally to ensure proper coverage of key areas. Assessors will be seeking the views of students on the six aspects of provision and should ensure that a clear agenda is outlined. An Aide Memoire for Meeting with Student Groups is provided in Annex G (which can be adapted if necessary to take account of the particular group and the subject provider's aims and objectives). Assessors will be keen to gauge students' awareness of the subject provider's aims and objectives; their experiences as learners in the subject, their responses to the teaching provided, and their views on academic and personal support, learning resources, and the arrangements made for student feedback and representation, including whether or not they were involved in the preparation of the self-assessment. Assessors should allow time to respond to questions.

Scrutiny of Documentation

66. Assessors also gather evidence through scrutiny of documentation. The evidence base drawn on by assessors includes reports and advice from external sources such as external examiners, employers, validating and accrediting bodies, as well as internal documents and reports. Emerging judgements are refined and tested against as wide a range of evidence as possible, for example, views expressed in meetings by staff or students will be checked and tested against the written documentation provided.

Review of Student Learning Resources

67. Assessors also gather evidence through direct review of the student learning resources. Assessors normally visit the facilities made available to the subject and may observe students and staff using specialist IT or other equipment in the course of normal teaching and learning activities. In reviewing library provision assessors may undertake catalogue searches or request access to on-line facilities. In evaluating the quality of learning resources, assessors' direct observations of facilities are considered alongside evidence from written documentation, meetings with relevant staff and meetings with students.

Making the Judgements

General Considerations

68. The assessors' collective judgements on the quality of education in relation to each aspect of provision are informed by all of the evidence gathered and considered by the whole assessment team. In each case, judgements are based on principles of fitness for purpose in relation to stated aims and objectives and the extent to which they are being met. Grades are assigned by the whole assessment team, on the basis of the evidence gathered in relation to each aspect of provision and using the Council's criteria for assigning grades. All team members are expected to share information gathered which is relevant to any aspect of provision, but individual assessors may co-ordinate the evidence related to particular aspects of provision. Written summaries of the information and evidence gathered should be distributed continuously within the team, along with notes of meetings, teaching and learning observations and student work reviews.

Assessment Team Meetings

69. The assessment team meets daily to discuss its findings, normally at the end of the afternoon or in the early evening. Team meetings are used to review the evidence gathered in relation to each aspect of provision, to form preliminary judgements and to determine which issues require further exploration. Assessors are expected to evaluate how the evidence gathered compares with the self-assessment and the subject aims and objectives, to discuss how gradings for teaching and learning observations have been reached, to test the strength of evidence adduced to support judgements, and to ensure that the published criteria are being applied. The assessment team meeting on the penultimate day should allow sufficient time for a discussion of the evidence gathered in relation to each aspect of provision and a preliminary discussion of gradings. Discussion of the information gathered and emerging judgements should involve the whole team. Therefore, where there is evening teaching to observe, it may be necessary to hold this meeting in the afternoon.

70. The assessors hold a final meeting early in the afternoon on the last day to review any additional evidence, to agree the outline of strengths and weaknesses in each aspect of provision, to finalise the grading for each aspect of provision, and to determine the overall conclusions to be reported to the institution. Typical agendas for assessment team meetings are included as Annex H.

Written Summaries

71. The draft summaries written by assessors during the visit focus on the evaluation of the evidence gathered in relation to each aspect of provision. Summaries should be analytical rather than descriptive and make direct reference to relevant aims and objectives and to sources of information such as meetings, documents, reviews of the sample of student work and observations of teaching and learning. The written evaluation should summarise the relevant strengths and issues relating to each aspect of provision, and overall should underpin the grade assigned.

Grading the Aspects of Provision

72. Assessors are required to make judgements on, and agree a grade for, each of the six aspects of provision. In assigning grades, assessors judge both the extent to which the student learning experience and student achievement in each of the aspects of provision contribute to attaining the objectives set by the subject provider, and the extent to which the objectives set, and the level of attainment of those objectives, allow the aims set by the subject provider to be met. The criteria for assigning grades are included in Annex I.

The Oral Feedback Meeting

73. The meeting with senior representatives of the institution at the end of the assessment visit is intended to provide oral feedback on the outcomes of the visit. Senior subject staff are normally present at this meeting. The meeting is chaired by the reporting assessor, who will convey the overall judgement, the graded profile and the main evidence that will be included in the published report. It is not a consultative meeting which might affect the judgements of the assessors, although a limited amount of clarification by the assessors may take place. The Agenda for the Oral Feedback Meeting is set out in Annex J.

Monitoring and Evaluation

74. Continual monitoring, review and evaluation is integral to the Council's activities. Each assessment is monitored through questionnaires completed by institutions, subject specialist assessors and reporting assessors. Some assessment visits are also monitored through visits by an associate director of the QAD.

The Assessment Reports

Quality Assessment Reports

75. A quality assessment report is published after each assessment visit. This includes a description of the assessment method, a 500-word statement of the aims and objectives of the provision assessed, prepared by the subject provider, the graded profile and the overall judgement on the quality of provision, an evaluative description of the quality of education, organised under the six aspects of provision, and the conclusions reached. The total length is some 4,000 words. Reports on individual assessments are published as soon as possible after the assessment visit.

76. The reporting assessor produces the first complete draft of the report in the week immediately after the visit by drawing on the summaries prepared by the subject specialist assessors during the visit. This is sent to the subject specialist assessors who then check the draft of the report to ensure that it is factually accurate and that it represents the views of the assessment team. It is particularly important that comments on the first draft of a report are returned within one week to the reporting assessor, in order that s/he may adhere to the production schedule for reports. The Council has procedures in place for monitoring closely the progress of individual reports in an effort to publish reports within a reasonable timescale after an assessment visit.

77. The published report is the main documented outcome of the quality assessment process and will be studied carefully by the institution as well as by others. Reports should concentrate on the judgements made using the specified criteria and the evidence which supported them, in relation to the subject provider's aims and objectives. They should include appropriate statements and evaluations so that the reader is not surprised by the gradings after reading the text related to each aspect. Succinct, accurate writing and a consistent style are essential. The reports should be primarily evaluative and analytical. The evidence base needs to be sound, and much depends on the accuracy with which this has been recorded by assessors during the visit. The report needs to exhibit clarity and thoroughness.

Subject Overview Reports

78. On completion of assessments in a given subject area, the Council publishes a subject overview report which provides an overall summary of the higher education provision within the subject, the strengths and weaknesses observed by the assessors and the judgements made. These reports include a discussion of the main features of education in the subject and are intended to assist in promoting good practice.

79. Subject specialist assessors are invited to attend a debrief session at the end of the programme of visits. Their comments at these sessions contribute to the subject overview reports. In addition, assessors may be invited to contribute to the analysis of the quality assessment reports, and also to the writing and editing of the subject overview reports. Each subject overview report includes a list of providers in the subject and the graded profiles achieved by all providers.

80. Quality assessment reports and subject overview reports are distributed widely. Each subject overview report, for example, is sent to some 7,500 destinations, including all secondary schools, sixth form colleges, further education colleges, careers services and main public libraries as well as all higher education institutions and the press in England and Northern Ireland. The reports are also available electronically on the World Wide Web at:

http://www.hefce.ac.uk

Annex A

Quality Assessment Aide Memoire

These notes are structured around the six aspects of provision, and offer a set of prompts which are designed to ensure focused coverage of each aspect of provision throughout the assessment and the associated report writing.

The aide memoire is used by assessors to guide:

analysis of the self-assessment prior to the visit
gathering of evidence during the visit
writing of the quality assessment report.

The key features and relationships outlined for each aspect of provision in Circulars 39/94 and 26/95 are covered broadly by this aide memoire. However, as the aims and objectives of the subject provider and the self-assessment form the framework for assessment the content of this aide memoire should not be regarded as prescriptive or exhaustive. In using the aide memoire, assessors will add and/or highlight questions and issues which, from the self-assessment, preparatory meeting and other documentation, have been identified as being of significance within the particular provision being assessed.

It is essential to maintain written records of evidence as it is gathered including notes of meetings and sources in written documentation, as well as completed observation notes and student work review notes. As a general principle, assessors should focus their attention at the level of the subject and address central institutional issues only when these have a direct impact on the subject being assessed.

Curriculum Design, Content and Organisation

What are the specific objectives relevant to this aspect of provision?

What is stated in the self-assessment?

Evaluate all levels of provision (taught postgraduate/undergraduate/HND etc).

1. Structure and content of curricula

Are the structure and content of the curricula consistent with the aims and objectives in terms of:

particular features and special characteristics of the provision
the intended student profile
levels and modes of study
breadth and depth of study
coherence
progression and achievement
flexibility and student choice
recognition of professional/accrediting bodies, where appropriate
specialisms of academic staff?

2. The intended outcomes of the curricula

Are the intended outcomes of teaching and learning defined clearly and are they generally understood by staff and students?

Are the learning opportunities provided consistent with the aims and objectives in terms of:

subject-specific skills and generic/transferable skills
vocational competences and progression to employment
progression to further study
personal development?

3. Currency and innovative features

Is there evidence that the content is informed by:

recent developments in teaching and learning
recent subject/interdisciplinary developments
research and consultancy activities of academic staff
industrial and professional experience, where relevant?

To what extent does this aspect of provision contribute to meeting the objectives set?

Do the objectives set, and the level of attainment of those objectives, allow the aims set by the subject provider to be met?

Teaching, Learning and Assessment

What are the specific objectives relevant to this aspect of provision?

What is stated in the self-assessment?

Evaluate all levels of provision (taught postgraduate/undergraduate/HND etc).

1. Strategy and methods for teaching, learning and assessment

Is there an effective teaching, learning and assessment strategy?

Is it articulated clearly in line with the subject aims and objectives and any institutional guidelines (eg code of practice)?

In the context of the aims, and given the student profile, is the structure and range of the programme of activities (teaching, learning and assessment) appropriate in terms of:

intended learning outcomes, in particular the development of:
- knowledge, understanding and other intellectual abilities
- subject-specific skills, generic/transferable skills, and independent learning abilities
- values, motivation, or attitudes to learning?
the curriculum structure and content, levels and modes of study
the student workload
the resources available, including academic and non-academic staff.

2. Observations of teaching and learning activities

What teaching and learning activities have been observed in terms of:

the representativeness of the sample by level, programme, mode, staff group
the range of sessions (e.g. lectures, seminars, practicals, tutorials)
the number of observations; percentage in each grade 1-4
the range of other materials for self-directed learning that may form part of an overall strategy for independent learning.

What were the strengths and weaknesses observed, as derived from completed observation notes (Annex E) in relation to:

the clarity of the objectives, match with the intended outcomes
the contribution of teaching, including the impact of professional activity and research where relevant
student participation
the use of accommodation and resources, TLTP materials
the overall appropriateness of the specific sessions to the achievement of the objectives?

3. Assessment

What is the assessors' evaluation of student assessment, based on the completed student work review notes (Annex F), in terms of:

the use of assessment as a learning aid as well as a means of judging performance
the quality of feedback to students
the match of methods of assessment to the intended learning outcomes
appropriateness to the student profile, level and mode of study
student understanding of assessment methods and criteria?

To what extent does the student learning experience within this aspect of provision contribute to meeting the objectives set?

Do the objectives set, and the level of attainment of those objectives, allow the aims set by the subject provider to be met?

Student Progression and Achievement

What are the specific objectives relevant to this aspect of provision?

What is stated in the self-assessment?

Evaluate all levels of provision (taught postgraduate/undergraduate/HND etc).

1. Student Profile

What is the student profile? Does it match the subject aims in terms of:

ratio of applications to places
entry qualifications, including average GCE A-Level point score, Access and other admissions criteria
range of students (e.g. age, gender, ethnicity, disability)?

2. Progression and completion rates

What are the progression rates, and are reasons for non-completion identified?

What are the trends in terms of:

annual progression
non-progression
transfers in and out of subject
completion of programme
qualifications awarded (e.g. certificate, diploma and degree results)?

3. Student achievement

What evidence is there of student attainment of the intended outcomes, including subject specific and transferable skills, derived from:

student work, completed observation notes and student work review notes (Annexes E and F)
destinations data - employment, further study, other?

What are the views concerning student achievement from:

external examiners' reports
reports of professional/validating bodies, where appropriate
employers/professional practitioners
students and graduates?

To what extent do the student learning experience and student achievement within this aspect of provision contribute to meeting the objectives set?

Do the objectives set, and the level of attainment of those objectives, allow the aims set by the subject provider to be met?

Student Support and Guidance

What are the specific objectives relevant to this aspect of provision?

What is stated in the self-assessment?

What is the overall strategy for support and guidance and is it consistent with the aims and objectives and the student profile?

Evaluate all levels of provision (taught postgraduate/undergraduate/HND etc), all areas of activity (e.g. field studies, work experience and study abroad). Establish student views on support and guidance.

1. Admission and induction arrangements

What are the arrangements for admission and induction, and are they generally understood by staff and applicants?

How effective are they?

2. Academic guidance and tutorial support

What are the arrangements for academic tutorial support? Are they clear, generally understood by staff and students, and well matched to:

curriculum structure
teaching, learning and assessment methods?

What provision is made to identify individual student needs?

3. Pastoral and welfare support

What are the arrangements for pastoral and welfare support? Are they clear, generally understood by staff and students, and well matched to:

curriculum structure
student needs?

Is there effective liaison between the subject staff and central services?

4. Careers information and guidance

What are the arrangements for careers guidance? Are they clear, generally understood by staff and students and well-matched to:

aims and objectives
curriculum
student expectations?

Is there effective liaison between the subject staff and the careers service?

To what extent does the student learning experience within this aspect of provision contribute to meeting the objectives set?

Do the objectives set, and the level of attainment of those objectives, allow the aims set by the subject provider to be met?

Learning Resources

What are the specific objectives relevant to this aspect of provision?

What is stated in the self-assessment (including in the Framework section)?

Evaluate all levels of provision (taught postgraduate/undergraduate/HND etc). Establish student views on learning resources and, where possible evidence of the extent of use by students in the subject area.

Evidence should be derived in part from completed observation and student work review notes (Annexes E and F).

1. Learning resources strategy

Is there an overall strategy for learning resources?

Is the provision of learning resources consistent with the aims and objectives in terms of:

support for the curriculum
the range of teaching, learning and assessment methods
the needs implied by the student profile?

What are the arrangements for maintenance, replacement and updating?

2. Library resources

Are the library resources available, accessible, and appropriate in terms of:

student induction and user support
book and periodical stock, learning packs
study space
learning support facilities.

Is there effective liaison between the subject staff and library services?

3. Equipment and information technology

Are equipment and information technology available, accessible, and appropriate in terms of:

general and specialist equipment
the provision of suitable learning materials, including TLTP
the provision of induction, training and support
open-access and independent learning arrangements?

Is there effective liaison between the subject staff and IT services?

4. Teaching and social accommodation

Are teaching and social accommodation available and appropriate in terms of:

the range of general and specialist accommodation (lecture, seminar, tutorial, studios, laboratories)
social, dining and recreational accommodation?

5. Technical and administrative support

Is appropriate technical and administrative support available?

Is there effective liaison between the academic staff and technical support?

To what extent does the student learning experience within this aspect of provision contribute to meeting the objectives set?

Do the objectives set, and the level of attainment of those objectives, allow the aims set by the subject provider to be met?

Quality Assurance and Enhancement

What are the specific objectives relevant to this aspect of provision?

What is stated in the self-assessment?

Evaluate all levels of provision (taught postgraduate/undergraduate/HND etc), and the impact on the student experience and achievement.

1. Internal quality assurance for the subject

What quality assurance processes are in place to gauge the success in achieving the stated subject aims and objectives? How do the subject-based practices at departmental/faculty level relate to the institution's quality assurance policies and processes?

Is there coherence and fitness for purpose in the processes?

Are the arrangements clear, rigorous and understood by staff and students, and are roles and responsibilities clearly identified?

How effective are arrangements for evaluation of and action as appropriate on:

HEQC audit report
the views of students
external examiners' reports
the views of employers and professional bodies, where appropriate
equal opportunities in relation to quality assurance policies?

2. Quality Enhancement

Is there evidence of development in the subject; how is the need for change identified and considered? What evidence is there of quality enhancement, informed by staff development, in terms of:

appraisal and peer review of teaching skills
systematic identification of staff development needs
induction arrangements for new staff
programme of activities and financial support for continuing professional development
participation by staff in the subject area in staff development activities
impact on teaching, learning and assessment and student support?

How accurate and incisive is the self-assessment? What contribution does it make to quality enhancement?

How effective are arrangements for periodic review of courses, modules and new programmes of study?

To what extent does this aspect of provision contribute to meeting the objectives set?

Do the objectives set, and the level of attainment of those objectives, allow the aims set by the subject provider to be met?

Annex B

The Preparatory Meeting Agenda

1. Introductions.

2. Any necessary clarification of the self-assessment.

3. Scope of the assessment visit - refer to and up-date the completed Information Form

undergraduate programmes
taught graduate programmes
student and staff full-time equivalent (FTE) numbers.

4. Documentation - see Annex C

sent in advance
available during visit.

5. Schedule of meetings for the assessment visit

initial welcome from senior representatives of the institution
staff in the subject area
- initial briefing, including any update of self-assessment
- clarification of assessors' queries
students
graduates/diplomates
employers (if appropriate)
meetings with staff to discuss the aspects of provision
- learning resources - library services, information technology (IT), specialist
- student support - academic, pastoral, financial, counselling, careers
- separate meetings may also be arranged as appropriate to discuss the curriculum, teaching and student progression
- quality assurance
oral feedback.

6. Protocols - see paragraphs 49-57, 61-65 and 73

observation of teaching and learning
liaison with subject contact
meetings.

7. Practical Arrangements

IT and secretarial support - HEFCE provides these
base room
- access and security
- electrical sockets and furniture
- access to telephone and photocopier
- refreshments
charges to assessors and to HEFCE for assessors' expenses
car parking.

Annex C

Documentation to be Provided by the Institution

Items in bold, italicised type are those to be generally sent to team members two weeks in advance of the visit. A single duplicate copy of this material should also be provided in the base room

Course/Programme-related Items

a copy of the self-assessment
subject information provided for students (course handbooks, leaflets etc)
the most recent available subject monitoring and/or external examiners' reports
timetables for all relevant programmes for the period of the visit.
internal subject monitoring reports for the last three years, including those from professional bodies, students or other sources as appropriate
external examiners' reports for the last three years (if not contained within the monitoring reports)

General/Institutional

prospectus
location map
site/building plan.

Student Work

samples to reflect the range of courses/programmes, levels and attainment/marks
samples to include written work, examination scripts/papers, project/lab reports, dissertations, practical work or other types of work as appropriate
where used, marking and feedback sheets and assessment criteria should be made available.

Statistics/Support Information

staffing list
staff CVs, details of research and staff development activity, including policy and strategy statements where available
student intake and progression data, covering the last three cohorts
a description of student support/welfare services, plus any recent analysis of student use, subject to normal constraints of confidentiality in respect of counselling and similar activities.

Management-related

academic development plans (if available)
academic management structure
quality assurance arrangements (including the institution's response to the HEQC audit).

Resources

summary of the learning resources and accommodation strategy and internal administrative arrangements (eg subject liaison with library services)
summary of the main physical resources available to support the subject being assessed, including accommodation and significant items of specialist equipment
summary of the learning resources available, including the library services and general IT.

Annex D

A Typical Schedule of an Assessment Visit Programme

Most visits start on Monday afternoon and end on Thursday afternoon.

Day One

The assessment team usually assembles in the afternoon.

first private meeting of assessment team in base room
welcome from Head of Institution and senior staff, briefing on institutional context
initial meeting with subject staff (student representatives may also be present).

The order of meetings may vary according to the availability of senior representatives of the Institution.

Day Two

Assessors gather evidence by observing a range of teaching and learning activities including lectures, seminars, small group teaching, practical workshops; sampling student work; attending meetings; reviewing learning resource facilities; and reading documentation provided. Evidence is shared continually and evaluated in relation to the aims and objectives. Individual assessors begin to formulate written summaries.

Meetings with subject and other staff take place throughout the second and third day of the visit, generally last no longer than one hour, and address the various aspects of provision.

early morning

Meeting between reporting assessor and subject contact

lunch time

Meeting with undergraduate students (separately or jointly as appropriate to the provision)

end afternoon

Meetings with former students, and employers (where appropriate).

early evening

Meeting of assessment team.

Day Three

Assessors gather evidence by observing a range of teaching and learning activities including lectures, seminars, small group teaching, practical workshops; sampling student work; attending meetings; reviewing learning resource facilities; and reading documentation provided. Team members continue to share and evaluate evidence in relation to the aims and objectives. Individual assessors further develop their written summaries.

A meeting is held with subject staff in the latter part of the programme, typically on the afternoon of the third day, to enable the assessors to clarify any issues that have arisen during the assessment visit.

early morning

Meeting between reporting assessor and subject contact

lunch time

Meeting with taught postgraduate students

end afternoon (as appropriate)

Meeting with subject staff

early evening

Meeting of assessment team.

Day Four

Observation of teaching and learning continues as appropriate. Assessors work on final draft of written summaries of evidence and judgements.

early morning

Meeting between reporting assessor and subject contact

lunchtime/early afternoon

Final meeting of assessment team, written summaries of evidence and judgements formally established

afternoon

Oral feedback to Head of Institution and senior subject staff.

Assessors depart.

Annex E

Observation Note

(for use in observed teaching and learning sessions)

Institution

Subject Module/Course title

Assessor

Length of session _______________hrs Length of observation _______________hrs

Number of students present _______________Composition of the group _______________

Level/Year of study _______________Mode of study FT PT Other _______________

Topic/Title of the session

Type of Activity Lecture / Seminar / Workshop / Practical / Tutorial / Other

Which of the learning objectives are relevant to this teaching/learning session?

Teaching What are the particular objectives planned for this session?

Please comment on the effectiveness of each of the following elements and the extent to which the planned objectives are achieved.

Planning

Content

Methods

Pace

Use of examples

Overall comments:

Student participation: to what extent was student participation intended and how far did it occur as planned; what evidence was there of student engagement with the session; what evidence was there of the learning objectives being achieved?

Accommodation and Resources: how effective was use of the room and layout, specialist equipment and material, visual aids, IT, etc.?

Overall Quality of the Session: judgement of the appropriateness of the specific session objectives and their achievement to the overall attainment of objectives

Strengths

Weaknesses

Other Comments

Overall Grade 1 2 3 4

Grade 1: The session fails to make an acceptable contribution to the attainment of the learning objectives set.

Grade 2: The session makes an acceptable contribution to the attainment of the learning objectives, but significant improvement could be made.

Grade 3: The session makes a substantial contribution to the attainment of the learning objectives, but there is scope for improvement.

Grade 4: The session makes a full contribution to the attainment of the learning objectives.

Please indicate where this observation note provides evidence of particular relevance to one or more of the six aspects by ticking the appropriate box(es) below

CDCO

TLA

SPA

SSG

QAE

Annex F

Student Work Review Note

Please complete one note for the sample of work assessed for each individual course or module

Institution

Assessor

Subject

Module/Course title

Level/Year of study____________ Number of items of work in the sample____________

Relation of the sample to the spread of student marks for the course

Nature of student work:

Essay

Examination Script

Lab or Workshop Project

Other (specify please)

Which of the learning objectives are being assessed?

Relevant External Examiner Report(s) seen Y/N Date of Report/Examiner's name

Quality of Assessment Design: please comment on the clarity of the assessment design and the match between the assessment criteria and learning objectives; appropriateness to the stage of study, student profile and mode of study

Grade 1 2 3 4

Quality of Staff Input: are the marking criteria clear and consistently applied; is the quality of the feedback appropriate; is there evidence of moderation of marking and is there evidence of appropriate external examiner scrutiny?

Grade 1 2 3 4

Overall Quality of the Student Achievement: does the student work demonstrate attainment of the stated objectives?

Grade 1 2 3 4

Overall Comments

Strengths

Weaknesses

Other Comments

Please indicate where this observation note provides evidence of particular relevance to one or more of the six aspects by ticking the appropriate box(es) below

CDCO

TLA

SPA

SSG

QAE

Annex G

Aide Memoire for Meeting with Student Groups

The meeting is generally chaired by the Reporting Assessor, who will introduce the assessors and provide a brief summary of the quality assessment method.

Curriculum Content and Organisation

reason for choosing course/institution
match between curriculum and expectations (eg flexibility, choice, content)
appropriateness of the curriculum content to the development of knowledge and skills
relevance to prospective career/further study
student timetable and workload
opportunities for practical and vocational experience, where appropriate.

Teaching, Learning and Assessment

range of methods experienced
student views on quality of teaching
support for independent study
student understanding of assessment methods and criteria
feedback on assessed work.

Student Progression and Achievement

advice on progression
attainment of intended learning outcomes.

Support and Guidance

admission and induction procedures
quality of written guidance
arrangements for academic and personal support, including personal tutorial system
support during study abroad and/or work experience
availability and use of welfare support, including counselling and financial advice
knowledge of and accessibility of referral systems for support
careers advice and guidance.

Learning Resources

library services (opening hours, practical access, user support, availability of stock)
course materials, including learning packs
IT provision (opening hours, practical access, user support, availability of terminals)
specialist equipment, including relevant software
teaching accommodation, including laboratory or studio provision
study space
common rooms, refectory, social areas.

Quality Assurance and Enhancement

ways in which students' views are sought
representation on course/departmental committees
the degree to which students' views are influential
students' contribution to the self-assessment.

Annex H

Assessment Team Meetings

Day One - Afternoon

Typical Agenda

1. Introductions, including clarification of agreed roles and the respective responsibilities of reporting assessor and subject specialist assessors

2. Reminder of assessment method:

evaluate attainment of subject provider's aims and objectives
evidence needed to support judgements
criteria for assigning grades.

3. Confirmation of aspects of the process and subject specialist assessors' division of responsibility. Reminders to:

share evidence with appropriate colleagues: discussion of mechanisms
use Assessors' Handbook for guidance, including aide memoire (Annex A)

4. Identification of issues to be pursued:

key points of self-assessment
discussion of brief commentaries prepared by assessors
discussion of process of developing written summaries.

5. Strategy to be adopted to ensure appropriate coverage of teaching and student work, range of levels/modes/programmes/courses/staff/types of class. Reminders and discussion of:

protocol for observation and feedback to teaching staff
use of observation and student work review notes

6. Meetings with staff, students, graduates and others:

use aide memoire (Annex A) and agenda (Annex G)
keeping and using notes of meetings.

7. Assessment programme for the following day:

agree coverage of class observations
agree coverage of meetings
time and location of team meeting(s).

8. Logistics:

transport, security of base room, administrative support, drinks and meals, photocopying and use of telephone
procedures for requesting additional documentation and making appointments with staff.

9. AOB.

Assessment Team Meeting Day Two - Early Evening

Typical Agenda

1. Discussion of the day's activities. Review evidence gathered:

observations, staff/courses/programmes/
modes/levels seen
discussion of observations
discussion of sampled student work and other documentation read
meetings held and information gathered: summary of key points shared with team
issues to be tracked further in relation to each aspect of provision.

2. Assessment programme for the following day:

agree coverage of observations
agree coverage of meetings
time and location of team meeting(s).

3. Arrangements for production of assessors' written summaries of evidence and for the tabling of first drafts of evidence and judgements at the next team meeting.

4. AOB.

Assessment Team Meeting Day Three - Early Evening

Typical Agenda

1. Brief review of the day's activities:

observations, staff/courses/programmes/
modes/levels seen
any necessary discussion of observations
student work and other documentation read
meetings held and information gathered.

2. Review evidence gathered in relation to each aspect of provision. Sharing of written summaries of evidence.

Initial consideration will be given to the appropriate grading for each aspect.

3. Agree priorities and assessment programme for the following day:

issues to be tracked further
agree coverage of observations
agree coverage of any meetings
preparation redrafts/drafts
time and location of final team meeting.

4. AOB.

Assessment Meeting Day Four - Early Afternoon

Typical Agenda

1. Summary of evidence gathered. Recap on the coverage of the previous evening and any new evidence. Debate on the quality of education provided in relation to subject aims and objectives:

agree conclusions
finalise grading of each aspect of provision

2. Arrangements for the oral feedback

issues to be presented
conduct of meeting.

3. Timescale for the production of the report. Importance of returning comments on draft report.

4. Arrangements for:

clearing the base room
returning documents and keys to institution
collation of documentation to be retained by the Reporting Assessor
removal or shredding of confidential papers which do not need to be retained
payment of bills to the institution if appropriate
booking of taxis if appropriate.

5. AOB.

Annex I

Grading the Aspects of Provision

1. The graded profile shows the extent to which the actual student learning experience and student achievement demonstrate that the aims and objectives set by the subject provider are being met.

2. The creation of the graded profile is achieved by applying the assessment scale to the aspects of provision. There are four numerical points on the scale - 1,2,3,4 in ascending order of merit.

3. A grade of 2 or better in an aspect means that the student learning experience and student achievement in that aspect make at least an acceptable contribution to the attainment of the stated objectives, and that the aims are being at least broadly met.

4. A grade of 1 means either that the actual student learning experience and student achievement in that aspect make an inadequate contribution to the attainment of the objectives, or that the objectives do not provide students with the experiences and achievements that would support a judgement that the aims were being met.

5. The criteria for assigning grades are set out below.

6. The assignment of the grade for each aspect is a matter for the professional judgement of the assessors drawing on the evidence from both the self-assessment and the assessment visit.

The Overall Judgement

7. The overall summative judgement is derived from the profile. All the profile elements are treated as of equal weight. Any provision where one or more of the profile elements receives a grade of 1 is subject to the following procedures:

a. If, on first assessment, one or more of the profile elements receives a grade of 1, the quality of the education will be recorded as subject to re-assessment within a year. An assessment report containing this decision will be published. A re-assessment will take place within 12 months. During the period between the first assessment and the re-assessment the normal Council funding arrangements continue to apply.
b. If, after re-assessment within 12 months, the profile still contains one or more elements graded 1, the education will be recorded as unsatisfactory quality and funding will be withdrawn in whole or in part. A second assessment report will be published.

8. A profile with all elements graded 2 or better will be reported as quality approved.

9. The judgements are given in an oral feedback from the assessment team to the institution at the close of the assessment visit.

Annex J

The Oral Feedback Meeting Agenda

The meeting is expected to last around 30 minutes, and is chaired by the reporting assessor.

1. Announcement of:

overall judgement
gradings for each aspect of provision

2. Summary of strengths and issues in each aspect of provision.

3. Conclusion and issues to be addressed.

4. Any points for clarification.

5. Timetable for the production of the quality assessment report. (The institution will receive the draft report to comment on the factual accuracy, after the reporting assessor has incorporated specialist assessors' comments on the draft, and will see the final report before publication.)

6. Thanks to the institution.

5. Appendix

Policy Statement on Equal Opportunities in Quality Assessment

Introduction

The Higher Education Funding Council for England (HEFCE) has developed practices in its quality assessment method that build on its equal opportunities policy. These practices are set out below for the information of institutions and individuals associated with the quality assessment process.

The policy is presented separately as it applies to the work of Divisional Staff, Reporting Assessors (Contract Assessors) and Subject Specialist Assessors. The general policy which applies to all assessors, and to Council staff working within the Quality Assessment Division, is presented first before the separate statements for each staff group.

General Policy Statement

1. In keeping with the Equal Opportunities policy of the HEFCE, the Council through its Quality Assessment Division (QAD) is committed to equal opportunities in the operation of its assessment method and in the contractual relationship with assessors.

2. The QAD values equally the contribution of all those associated with the assessment of educational provision. It aims, in any matter relating to its work, to ensure that no-one is treated less favourably than another on the grounds, for example, of ethnic origin, nationality, disability, gender, religion, marital status, age, family responsibility or sexual orientation. In these aims, the QAD's intention is to reflect not only the letter but the spirit of equal opportunities legislation.

3. The QAD is committed to the pursuit and maintenance of professional standards in the assessment of the quality of educational provision, including those standards necessary to avoid any impropriety.

4. The QAD is opposed to harassment in any form and is committed to providing a working environment that is free from harassment and in which the dignity of the individual is respected.

5. The Quality Assessment Method evaluates the student learning experience and student achievement against the aims and objectives set by the subject provider, including statements about equal opportunities.

6. The QAD will recognise and work where appropriate with the equal opportunities policies, procedures and objectives established within specific institutions. Statements for different categories of staff follow.

Divisional Staff Statement

1. The Division will provide training and continuing development for all its staff, in line with the Council's policy.

2. All staff will be encouraged and enabled to raise in confidence any incident or concern relating to harassment or equality of opportunity in employment.

3. Through its Senior Executive Group, QAD will consider annually equal opportunities monitoring data relating to staff within the Division.

Contract Assessor Statement

1. In the recruitment and selection of assessors, QAD will consider all applications on their merit against the published person specification.

2. QAD will provide training for contract assessors on recognising and enabling sensitive and professional behaviour commensurate with their management role.

3. QAD recognises a 'duty of care' in relation to contracted and seconded assessors who undertake work on its behalf. By 'duty of care' it recognises the entitlement of assessors to be treated professionally and in a manner consistent with the effective exercise of the specific tasks for which they are contracted. The terms and conditions of contracts and the expected working practices will be kept under review to ensure that they reflect the Council's commitment to equal opportunities and to the discharge of appropriate support to assessors whilst they are undertaking duties on behalf of the Council.

4. QAD will expect of all HEIs that assessors will be treated with respect and in accordance with the Council's equal opportunities policy and will take action appropriate to this expectation.

5. QAD will provide assessors with written advice summarising the QAD's policy and its associated procedures, and will implement the complaints and grievance procedures specified in the attached annex.

6. QAD will provide written advice on the appraisal of equal opportunities issues, against the aims and objectives set by subject providers, in relation to each of the six aspects of provision which underpin the assessment judgement.

7. QAD will provide briefings on subject-specific issues which include equal opportunities where and as appropriate to the subject.

8. QAD is committed to a policy of continuous improvement in order to deliver more effectively its commitment to equality of opportunity, and, accordingly, will review regularly the support and guidance that it gives to assessors to enable them to deliver their contracted duties.

Specialist Assessor Statement

1. In the recruitment and selection of assessors, QAD will consider all applications on their merit against the published person specification

2. QAD will provide training for assessors on sensitive and professional behaviour commensurate with its equal opportunities policy as an embedded part of its Specialist Assessors' training programme.

4. QAD will expect of all HEIs that assessors will be treated with respect and in accordance with the Council's equal opportunities policy and will take action appropriate to this expectation.

5. QAD will provide assessors with written advice summarising the QAD's policy and its associated procedures, and implement the complaints and grievance procedures specified in the attached annex.

7. QAD will provide briefings on subject-specific issues which include equal opportunities

where and as appropriate to the subject.

8. QAD is committed to a policy of continuous improvement in order to deliver more effectively its commitment to equality of opportunity for assessors, and, accordingly, will review regularly the support and guidance that it gives to assessors to enable them to deliver their contracted duties.

Grievance and Complaints Procedures for Assessors

1. The QAD will take seriously every incident and complaint brought to its attention. All reported incidents will be treated in confidence and will be investigated promptly and fairly.

2. The QAD will have a nominated officer responsible as a first point of contact for all complainants, additional to the normal reporting structure. This designated person is referred to below as the QAD EO officer.

3. Reporting Assessors (RAs) and Subject Leaders (Council officers) who as part of their professional responsibilities routinely monitor and support the assessment process will be required to be alert to potential and actual breaches of appropriate professional conduct.

4. The QAD will maintain a confidential log of all incidents and complaints including those which are not subsequently confirmed in writing. The confidential log will record the nature of the alleged incident and the identity of the parties involved. The contents of the log will be subject to periodic review by the EO officer and the Director of Quality Assessment with a view to eliminating any entries that are no longer current. The results of these reviews will be communicated to any named parties.

5. The procedures for dealing with a reported incident or complaint will comprise the following steps:

establish who is involved
establish what is alleged to have taken place
assess the integrity of the assessment process in the light of the alleged complaint
record in the confidential log the nature and circumstances of the complaint, the result of the investigation, and action taken.

6. The QAD will initiate its formal investigative procedures, as defined in paragraph 12 below, only in relation to those incidents and complaints which are reported in writing.

7. The QAD will advise all the appropriate parties, normally those directly affected and whoever has reported the incident, of the outcome of the investigation of a formal complaint.

8. The QAD will request all assessors to record on the assessment evaluation form any incident which may have compromised processes or caused harassment, whether or not it has been subject to complaint.

9. Where a reported incident involves a higher education institution (HEI), the QAD will wish to co-operate fully with the HEI and have regard where appropriate for the HEI's equal opportunities policy and procedures.

10. Where a reported incident involves a HEI, QAD will inform the appropriate officer of the HEI of the outcome of any investigation.

11. In the first referral of an incident or complaint, the reporting structure will be as follows:

Subject Specialist Assessor (SA) on a visit: first contact would normally be the Reporting Assessor but, where this would not be appropriate, may be the Subject Leader, the Duty Officer (Associate Director) or the QAD EO Officer.
Reporting Assessor (RA) on a visit: first contact would normally be the Subject Leader or Duty Officer but, where this would not be appropriate, may be through the QAD EO Officer to the Director of Quality Assessment.

In the case of a complaint being initiated by a member of staff, a student or a client of a HEI, the HEI's liaison with the QAD will normally be through the Reporting Assessor, the Subject Leader or the Duty Officer, as appropriate, to the Director of Quality Assessment.

12. The procedures for dealing with a formally reported incident or complaint in writing will comprise:

Stage 1: Define the problem

establish who is involved
establish what is alleged to have taken place
assess the integrity of the assessment
process in the light of the alleged complaint
record the nature and circumstances of the complaint and action taken
refer to the officer responsible for the complaints procedure.

Stage 2: Investigate and record the complaint

seek written statements from the persons involved
advise parties to the complaint on the procedures for proceeding with the investigation, confirming in writing the steps being taken
invoke the formal investigatory procedures as defined in the relevant contract and this procedure.

Stage 3: Managing the complaints process

monitor compliance with the agreed procedures
discuss the evidence, advise on action and endorse the final outcomes
implement the outcome of the investigative procedure in accordance with the terms and conditions of the contracts involved
report final outcome and close file
consider any necessary changes in procedures in the light of the individual complaint and other accumulated complaints.

13. Responsibility for each stage of the process resides with the Director of Quality Assessment who may delegate all stages as appropriate within the terms of the contracts for assessors.

14. The QAD expects of all HEIs that assessors and other staff employed within the Division will be treated with respect and in accordance with the Division's equal opportunities policy. In this context, the QAD will undertake a similarly staged approach to any complaints raised by assessors, in the course of their duties on behalf of the Funding Council, against HEI staff, students or clients.

15.. The QAD EO officer is:

Trish Archer
Human Resources Team
Northavon House
Coldharbour Lane
Bristol BS16 1QD

telephone 01179 317364.