2016-17 ILR data checking tool
1. This circular letter notifies colleges of the upcoming release of HEFCE’s data checking tool for Individualised Learner Record (ILR) data. This data checking tool is provided to colleges so that they can verify, and where appropriate correct, their 2016-17 ILR R14 data before signing off this data with the Education and Skills Funding Agency (ESFA). The first outputs for the data checking tool will be released in late August. We will notify contacts at colleges when these outputs, and the tool, are available.
2. There has been a change in terminology for the uses of individualised student data. This includes the renaming of the ‘web facility’ to the ‘data checking tool’ and changes to the categories of outputs. Further information can be found in ‘Change to terminology for uses of individualised student data’ (HEFCE Circular letter 22/2017).
3. The ESFA and HEFCE strongly encourage use of the data checking tool, which we regard as an essential element of all colleges’ data quality processes. It will help colleges:
- to return accurate data to the ESFA
- to verify the accuracy of the derived fields that may inform and monitor funding allocations
- to check, before submission, that the ILR R14 data is fit for publication on the Unistats website
- to verify the accuracy of the target list of students for the National Student Survey (NSS) and Destinations of Leavers from Higher Education (DLHE), and to start preparing the contact details for these students
- to verify the accuracy of some student characteristics which may affect institutional performance metrics
- to respond to data verification queries (DVQs) raised by HEFCE staff
- to reduce the likelihood of selection for the reconciliation exercise for 2016-17 (selected colleges are typically subject to considerable additional work and potential funding adjustments)
- to identify discrepancies between within-year HEFCE data returns, such as the Higher Education in Further Education: Students survey (HEIFES), and the outturn position for 2016-17
- to identify errors in HEFCE data returns.
4. Data submitted to us and signed off as correct by the head of institution, or finally submitted to the ESFA, is the final data that we will use; this is also consistent with published data. We will only accept amendments after this point in exceptional circumstances, where any identified data errors are widespread and significant and make a material difference to HEFCE’s use of the data. If a college identifies errors and wishes to submit amendments, it will be required to complete an amendments action plan. Once an action plan is received, amendments will be required to pass a HEFCE assessment process. In assessing amendments we will consider the multiple purposes for which we use the data, in the categories below:
- regulation and assurance
- quality and institutional performance, including the Teaching Excellence Framework and the Annual Provider Review
- public information, including Unistats, the DLHE and the NSS
- statistical and policy analysis.
For further information regarding data amendments, please see the ‘FEC Technical guidance 2017-18’, which was published as part of the 2017-18 spring grant announcement.
5. We strongly encourage colleges to analyse the data checking tool outputs as part of their ILR data quality processes. Past reviews have confirmed that this is an essential element of most colleges ILR data quality processes. HEFCE staff may query potential errors in ILR data that could affect our future use of the data. In particular, colleges that do not scrutinise the funding‑related outputs are more likely to be selected for the reconciliation exercise described in paragraphs 25 to 29, which may have funding consequences.
6. Where discrepancies occur between within-year HEFCE data returns (such as HEIFES) and the outturn position for 2016-17, we expect colleges to take full account of the outputs from the data checking tool when preparing future data returns. We therefore strongly encourage colleges to analyse the data checking tool outputs as part of their data quality, planning and audit processes.
7. We would be grateful for feedback on the algorithms used to create the data checking tool outputs, in particular where these algorithms have changed from previous years.
The algorithms used for each output are available on our website. Feedback should be emailed to firstname.lastname@example.org.
Data checking tool outputs
8. The 2016-17 ILR data checking tool will generate the following outputs:
- HEIFES16 comparison
- student premium allocations data summary
- 2018 Unistats summaries
- 2018 NSS target list
- 2016-17 DLHE target list
- 2016-17 Student characteristics data summary
These outputs will not all be available initially. We will release outputs in phases and will notify contacts at colleges when new outputs become available.
9. The HEIFES16 comparison output compares 2016-17 ILR data with the college’s HEIFES16 data returned at the start of the academic year. There is also a comparison between the college’s latest 2017-18 grant and grant calculated using the 2016-17 ILR data.
10. We will be re-creating Column 3 of Table 5 of HEIFES16 (years countable on apprenticeships) in the 2016-17 ILR data checking tool. This is because the recent apprenticeship reforms have resulted in an increased number of apprentices undertaking higher education as a part of their learning, and as HEFCE has a statutory duty of the quality assurance of all apprenticeships containing prescribed higher education, we need to monitor the scale, nature and performance of this provision. This comparison aims to help assure data quality with respect to ILR fields that are related to HEFCE’s identification and monitoring of students recorded on an apprenticeship at Level 4 or higher in the 2016-17 ILR data.
11. We will be re-creating Column 1 of Table 6 (years of course taught under sub-contractual arrangements by other providers) in the 2016-17 ILR data checking tool. This is to help ensure years of course are recorded correctly in the ILR where they are taught under a sub-contractual arrangement by another provider, for use in quality and institutional performance measurements, including the Teaching Excellence Framework.
12. The algorithms used to generate the HEIFES16 comparison output are intended to be those for the reconciliation exercise next spring. We may, however, make changes where we believe these will improve the algorithms.
Student premium data summary
13. The student premium data summary presents the 2016-17 ILR data we intend to use to calculate the 2018-19 student premium allocations. The 2017-18 Disabled students’ premium funding method is applied to the 2016-17 ILR data submitted to the data checking tool, to illustrate the effect that changes to ILR data may have on funding. Student characteristics derived from the 2014-15 and 2016-17 ILR data are compared to allow institutions to check the quality of the data we are likely to use to calculate the full-time student premium. Because an element of our funding method involves linking to other data sources, we cannot present a full calculation of the full-time student premium with 2016-17 ILR data. Instead we will flag where colleges have neglected to complete certain key fields for large numbers of students.
Unistats and NSS and DLHE target lists
14. We intend to publish Unistats summaries from the 2016-17 ILR R14 data in 2018. The data checking tool provides an opportunity for colleges to check, before submission, that their ILR R14 data is fit for publication on the Unistats website.
15. The 2016-17 ILR data will be used to produce separate target lists of students to be included in the 2018 NSS and the 2016-17 DLHE survey. Colleges are encouraged to use these lists to start preparing the contact details for these students.
- In the case of the NSS, these will be passed to the agency running the survey. These contact details will need to be returned to the agency in mid-November 2017. Further details about the arrangements for the 2018 NSS will be published in due course.
- In the case of the DLHE, these should either be passed to HEFCE’s framework contractor or be used to prepare for carrying out the DLHE survey by other means, depending on the college’s choice of method.
16. On Friday 20 October 2017we will extract final target lists for the 2018 NSS and the 2016‑17 DLHE from data submitted to the data checking tool. Colleges should therefore ensure that the most recent data submitted to the data checking tool on this date generates complete target lists for both surveys. Once we have received the 2016-17 ILR R14 data from the ESFA (in December 2017), we will use it to validate colleges’ target lists. HEFCE staff will routinely access NSS and DLHE target lists generated by the data checking tool.
Student characteristics data summary
17. The student characteristics data summary identifies fields taken from the ILR that are used in the construction of institutional performance and their associated benchmarks. These are fields that classify a student’s background or other characteristics for use in the benchmarking process for quality and institutional performance measurements, including the Teaching Excellence Framework and Annual Provider Review metrics.
18. The data summary is provided to help identify potential errors and reduce the numbers of unknown or unpopulated student characteristics in ILR data that will affect our uses of the data.
Data verification queries
19. HEFCE staff may raise DVQs during the data collection window where we assess the impact that potential errors in data could have on the areas listed in paragraph 8.
20. Further details of this process are described in the ‘Data verification’ section of our website. We expect all colleges to respond to DVQs before the ILR sign-off deadline.
21. In the HEIFES16 comparison and student premium allocations data summary we generally assume the funding rates used in the spring 2017 grant announcement.
22. The version of the Learning Aims Search database that we refer to is taken from 8 August 2017.
23. We will link ILR data with other data sources (other colleges’ ILR data and schools’ National Pupil Database data) to calculate students’ UCAS tariff points for the purpose of assigning them to risk groups for the student premium allocation. For data protection reasons, links to these data sources are not included in the data checking tool’s student premium data summary output, meaning that a student’s risk group assignment may change between the data checking tool and the calculation of actual student premium allocations.
24. The data checking tool calculations for the HEIFES16 comparison will incorporate any ILR amendments (for 2015-16 and earlier years) that were approved by HEFCE’s data amendment panel on 9 June 2017 and were signed off by colleges by Friday 28 July 2017. Any overrides to derived fields also signed off by 19 July 2017 will be included in all data checking tool calculations. We do not intend to update the data checking tool outputs to reflect any amendments or overrides that may be signed off subsequently.
Relationship with ILR reconciliation exercise
25. We use the annual ILR reconciliation exercise to monitor colleges’ HEIFES returns using ILR data. This reconciliation exercise occurs after we have received a final copy of all colleges’ data from the ESFA, typically in the December following the data checking tool launch.
26. Our funding allocations are informed by the data provided by colleges. If we find, either through reconciliations with ILR or other organisations’ data, or during the course of any data audit, that erroneous data has resulted in colleges receiving incorrect funding allocations, then we may adjust their funding accordingly (subject to any appeals process that may apply and the availability of our funds).
27. Any funding adjustments arising from the reconciliation of a comparison between ILR R14 student data and HEIFES16 are likely to affect the funding previously announced for 2017-18.
28. Colleges selected to respond to the reconciliation exercise must typically undertake a substantial amount of work, which may take several months to complete. The data checking tool is provided to complement the reconciliation exercise but it does not replace it.
29. The ‘Data reconciliations’ area on the HEFCE website provides further detail regarding the reconciliation exercise.
30. We will only access data processed via the data checking tool to support colleges in returning high quality data to the ESFA. We do not intend to use data processed by the data checking tool to make decisions about individual providers but may use responses to DVQs in our assessments of data quality. Further information on how we handle personal data can be provided on request. See our website for further details.
31. HEFCE staff may examine outputs from the data checking tool in any of the following circumstances:
- where a college explicitly gives permission
- to produce early modelling of funding allocations
- to verify HEIFES17 data
- to study general data quality and algorithmic issues as part of the DVQ exercise described in paragraphs 19 and 20
- to study general data quality and algorithmic issues with the Unistats summaries
- to retrieve NSS and DLHE target lists.
32. A number of help guides are available in the ‘How we use individualised student data’ area of our website (www.hefce.ac.uk/data/indstudata/help/). Some specific areas may be of interest:
- Guidance for using the data checking tool can be found in the ‘How to use the HEFCE data checking tool’ web page.
- Extranet locations, deadlines and documentation can be found under ‘2016-17 uses of individualised student data overview’.
- Technical information (such as algorithms and problems of fit) is listed under ‘2016-17 uses of individualised student data overview’.
- Information on how to obtain data from the HEFCE extranet can be found in the ‘How to access outputs’ guide.
- Answers to frequently asked questions regarding the data checking tool are on the HEFCE website (www.hefce.ac.uk/data/indstudata/help/ilr/1617/faq/).
33. For any further information or guidance, contact Sarah Cronk (tel 0117 931 7401, email email@example.com).
Director (Analytical Services)