You need cookies enabled

Cookies

You need cookies enabled

Introduction

1. When the funding bodies in England and Northern Ireland launched the new operating model for quality assessment in March 2016, we said that we would publish a more detailed account of the Annual Provider Review process and that this would include details of the data to be used and the approach we would take to determining the basis for further investigation. This document addresses these issues.

2. Annual Provider Review (APR) will take place from 2016-17 as the core mechanism to assess quality in the higher education providers we fund[1]. Its key features are:

  • it will draw on existing data and information: there will be no new data requests nor a requirement for any annual APR submission
  • indicators and metrics will be used in a contextualised and rounded way
  • existing risk processes will be used to make more consistent judgements about financial sustainability and good management and governance[2]
  • judgements on quality and standards matters will be reached through peer review.

3. We understand that the introduction of a more systematic and formalised annual process represents a significant change in our quality assessment arrangements and that this has coincided with other changes in the higher education landscape, for example the implementation of the Teaching Excellence Framework (TEF). We therefore plan to run a number of webinars and events to ensure that providers have a well-developed understanding of the way in which APR will operate in this broader context[3].

Context

4. In March 2016, the funding bodies in England and Northern Ireland published a revised operating model for quality assessment for implementation from 2016-17. The new operating model is designed to be proportionate, risk-based and grounded in the context for an individual provider and its students: it will target regulatory scrutiny and activity on those areas and providers that represent greater risk to the student interest or to the reputation of the higher education sector as a whole.

5. The revised operating model for quality assessment consists of the following components:

  1. Baseline regulatory requirements, which include quality-related requirements.
  2. A single gateway for entry to the higher education system.
  3. A developmental period of closer monitoring, engagement and scrutiny for recent entrants, and for providers requiring this for other reasons.
  4. Risk-based and context-sensitive review arrangements for all providers, building on established and tested approaches to data benchmarking and analysis, intelligence gathering, risk assessment, and assurance.
  5. Strengthened arrangements for securing degree standards and their reasonable comparability across the UK.
  6. Rapid tailored intervention where necessary.
  7. Protection of the international reputation of the UK higher education brand, including the assurance of transnational education.

6. More detail about each of these elements and the operating model as a whole can be found at www.hefce.ac.uk/pubs/year/2016/201603/.

7. This document sets out the approach that will be taken to the ‘risk-based and context-sensitive review arrangements’ referred to in paragraph 5d above, to be delivered through the Annual Provider Review process from 2016-17.

8. The Government has recently published a technical note to support the passage of the Higher Education and Research Bill through Parliament[4]. This confirms that, subject to the will of Parliament and the outcome of a future consultation, the future Office for Students will adopt an Annual Provider Review process to discharge its duty to assess quality and standards as part of a single risk-based regulatory framework for all of the providers it regulates. As part of this process, the designated quality body will carry out in-depth quality review visits where there are serious concerns about an individual provider. We expect the approach taken by the Office for Students to build on, and learn lessons from, the approach set out by HEFCE here.

Our approach to Annual Provider Review

9. Annual Provider Review builds on two streams of established work undertaken by HEFCE:

  • our longstanding assurance work on financial sustainability, good management and governance that is governed by the terms of the Memorandum of Assurance and Accountability and our funding agreement with individual higher education (HE) providers[5]
  • since the removal of student number controls in England we have, at the Government’s request, annually been examining the impact of student recruitment patterns at individual providers to ensure these have not resulted in adverse effects on quality. This has been a desk-based exercise to analyse benchmarked student outcomes data. In addition, in December 2015, we required an additional assurance from governing bodies about the arrangements in place to handle informal and formal student complaints, and the causes of these, effectively.

10. Neither of these streams of work are mechanistic: they involve using the understanding of a provider’s context and strategy that we gain through our regular visits and engagement. One or both streams of work may have resulted in heightened engagement with a provider to explore its strategy and plans of action. What they have in common is that the work is conducted in a risk-based and proportionate way, where the provider’s context shapes both the assessment and any HEFCE response.

11. For example, a provider’s data might reveal very significant growth in student numbers. For some providers this can signal unplanned and unmanaged growth with negative consequences for the quality of the student academic experience. For others it will be intentional, well-managed and supported by appropriate resources. Our understanding of each provider, and the formal assurances we have received, tell us which of these scenarios is accurate; we are then able to focus our attention on the providers that present the greatest risk to students.

12. The APR process formalises these existing activities and brings them together in a more structured way. Burden for providers will be kept to a minimum: there will be no new data requests nor a requirement for any annual APR submission. A provider will only need to take action where we have identified areas of concern.

13. The key elements of the APR process are described in more detail in the sections that follow.

Assurance statements from a provider’s governing body

14. HEFCE-funded higher education institutions (HEIs) will be familiar with our routine collection and testing of annual assurances from governing bodies through the Annual Accountability Return. These assurances currently relate to financial sustainability, good management and governance, data quality, and value for money. In December 2016 governing bodies will be asked to provide some additional assurance statements on quality and standards matters.

15. Further education colleges (FECs) directly and indirectly funded by HEFCE (including sixth form colleges that deliver higher education) will be asked to submit the new assurances through a tailored template. We will not collect information that duplicates the broader set of assurances that colleges already provide to the Skills Funding Agency and Education Funding Agency.

16. The same process will apply in Northern Ireland, with providers required to submit their assurance statements to DfENI by noon on 1 December 2016.

17. The new assurance statements relate to the governing body’s oversight of academic governance arrangements as set out in the Higher Education Code of Governance[6], and focus on:

  • the continuous improvement of the student academic experience and of student outcomes
  • the reliability of degree standards.

18. It is not our intention that the governing body be drawn into quality management activities itself, but rather that it receives reports and challenges assurances from elsewhere in the provider. This does not alter the well-established relationships between governing bodies and senates or academic boards (or equivalent). The evidence on which an individual governing body’s assurances are based, and the way in which it chooses to receive and challenge information, will be shaped by the particular arrangements in that provider. Each governing body is, therefore, responsible for determining the approach that it will take. For example, there is no standard template for reports or action plans and each governing body will determine its own requirements.

19. The precise wording of the assurance statements, and the process we will adopt during 2016-17 to collect these, can be found at:

20. The assurances given by a provider’s governing body – in the traditional areas of financial sustainability, good management and governance; more recently in relation to student complaints; and in the new quality and standards areas – will be used during the APR process to confirm that the provider is effectively discharging its responsibilities in all of these areas. The reliability of the annual assurances, and the evidence on which they are based, will be tested during the familiar five-yearly HEFCE Assurance Review visit. Our schedule for this activity for HEIs, together with an indicative schedule for FECs, can be found at www.hefce.ac.uk/reg/QualityAssessment/whatdo/.

21. HEFCE and DfENI are working with the Leadership Foundation for Higher Education to identify any additional support that governing bodies might need to strengthen their work in relation to the new assurances.

Scrutiny of key pieces of data

22. The APR process will make use of the student data and other data that providers already submit to the Higher Education Statistics Agency (HESA), the Skills Funding Agency, the Education Funding Agency, the Department for Education and/or the Department for the Economy in Northern Ireland, or to HEFCE: there are no new data requirements[7]. The process will draw together data and other information about each provider and present this in an ‘APR dashboard’, which we tested successfully with groups of providers earlier in the year.

23. Some of the metrics that will be used to monitor quality as part of the APR process will be the same as those used to assess performance in the Teaching Excellence Framework[8]. This will ensure that APR can use metrics that report students in relation to the provider teaching them rather than in relation to the provider with which they are registered. In particular, HEFCE and DfENI will continue to monitor student non-continuation rates, National Student Survey outcomes, and student employment outcomes using the metrics that inform TEF assessments in Year Two.

24. In addition to TEF metrics, the APR process will monitor student recruitment patterns. In 2016-17 we will consider supplementary indicators in relation to degree outcomes (including differential outcomes for students with different characteristics) and taught postgraduate student cohorts.

25. We will use time series data to ensure that we understand a provider’s performance over time and, where possible, we will use benchmarked metrics to ensure that our judgements take account of operating context, the nature of provision and the characteristics of a provider’s students. We will consider all of the indicators and metrics (both benchmarked metrics and absolute values) for a provider and the picture that these present when considered in a contextualised and rounded way. For example, financial indicators will be supplemented by the findings of our analysis of a provider’s annual accountability return.

26. It is not the case that performance against a single indicator, or series of indicators or metrics, will automatically result in a particular assessment, but attention will focus on areas in which:

  • performance is significantly below benchmark levels
  • performance is on a downwards trajectory
  • the absolute value of an indicator or metric gives cause for concern
  • there are other significant changes or events, for example, in borrowing levels.

The intention here is to form a rounded view of the performance of a provider and to make a judgement about any areas of concern.

27. The elements of the APR dashboard and the approach we will take to benchmarking are set out in more detail in Annex B.

Intelligence gathering

28. Through our engagement with providers in England we have developed a sophisticated understanding of the higher education sector and a strategic view of the context in which individual providers are operating. This intelligent understanding is already used in the operation of our assurance work on financial sustainability, good management and governance, and in assessing the impact of the removal of student number controls. It ensures that judgements are contextualised and not overly mechanistic. DfENI adopts a similar approach in relation to providers in Northern Ireland.

29. We will draw on our engagement activities throughout the 2016-17 academic year to ensure that the APR process is similarly underpinned by an up-to-date understanding of the HE sector and the strategies and plans of individual providers.

30. All providers in England will receive a visit from their HEFCE regional team during each academic year. In Northern Ireland, the existing arrangements for accountability meetings and visits between DfENI and all HE providers will continue. This visit will provide an opportunity for a discussion of the trends and patterns in the provider’s data and any actions that are under way or planned in areas that the provider has identified for improvement. It will provide an opportunity for the provider to explain the current context in which these plans are taking place. A consistent agenda will be used in these meetings and a written record of the visit will be shared and agreed with the provider.

Quality review visits where there are serious concerns

31.Where serious concerns about quality and standards matters are found during the APR process, the process set out in the second stage of HEFCE and DfENI’s Unsatisfactory Quality Scheme will be followed. The provider will receive a quality review visit from the Quality Assurance Agency for Higher Education (QAA) to investigate in more detail the areas of concern.

The APR process in 2016-17

32. Annex A sets out the process we will use for APR during 2016-17.

33. We will review the APR process, including the elements of the APR dashboard, after the first year of operation.



[1] Throughout this document ‘we’ refers to the Higher Education Funding Council England (HEFCE) and the Department for the Economy, Northern Ireland (DfENI).

[2] Existing risk processes for England are set out in HEFCE’s Memorandum of Assurance and Accountability which can be found at www.hefce.ac.uk/pubs/year/2016/201612/.

[3] Our programme of activities can be found at www.hefce.ac.uk/reg/QualityAssessment/whatdo/

[7] There may be a need for additional information to be collected in future years from FECs in Northern Ireland.

[8] References to the Teaching Excellence Framework in this document relate to the operation of the TEF in Year Two, which corresponds to academic year 2016-17, when assessment takes place and TEF outcomes are published. TEF awards made in Year Two will be primarily relevant to those students applying in 2017-18 for courses starting in 2018-19.

Date: 24 October 2016

Ref: HEFCE 2016/29

To: Heads of HEFCE-funded higher education institutions, Heads of HEFCE-funded further education colleges, Heads of universities in Northern Ireland
Heads of further education colleges in Northern Ireland Chairs of governing bodies of higher education institutions and further education colleges in England and Northern Ireland Heads of other higher education providers Students’ representatives Professional, statutory and regulatory bodies Key stakeholders with an interest or involvement in the regulation of higher education Individuals and organisations with an interest in quality assessment in higher education

Of interest to those
responsible for:

Quality assessment and quality assurance; Academic governance; Planning and management; Student experience and engagement; Learning and teaching

Enquiries should be directed to:

Scott Court, tel 0117 931 7417, email qualityassessment@hefce.ac.uk.

Institutions and organisations in Northern Ireland are welcome to discuss this document with DfENI. Please email claire.thompson@economy-ni.gov.uk.