ELPAC and the ICAO Doc 9835 checklist

Is the test is designed to assess speaking and listening proficiency in accordance with each component of the ICAO Language Proficiency Rating Scale and the holistic descriptors in Annex 1?

Yes. Listening is assessed in Paper 1 and speaking in Paper 2. Paper 2 is rated on the six ICAO rating scale criteria (Pronunciation, Structure, Vocabulary, Fluency, Comprehension and Interactions), with the holistic descriptors of Annex 1, Appendix 1 as the frame of reference. See about the test and the construct section of the validity argument.

Is a definition of the test purpose that describes both the aims of the test and the target population accessible to all decision-makers?

Yes. The purpose and the target population (air traffic controllers and pilots, in licensing and routine competency checks) are stated on about the test, in the FAQ and in the ELPAC Handbook under documents.

Is a description of and rationale for test construct and how it corresponds to the ICAO language proficiency requirements accessible to all decision-makers in plain, layperson language?

Yes. The construct is defined and mapped to the ICAO requirements in the validity argument, which also sets out the reasoning inference by inference in non-technical language.

Does the test comply with principles of good practice and a code of ethics as described in Chapter 6 of ICAO Doc 9835?

Yes. ELPAC publishes a Code of Ethics covering examiners, raters and interlocutors, test centres, test takers, and item writers and test developers, grounded in the ILTA Code of Ethics, Doc 9835 and EAQUALS quality principles. It is supported in practice by supervised administration (an invigilator supervises Paper 1 and an assessor is present throughout to ensure results are the test taker's own work), rating by two accredited examiners with a third-examiner dispute mechanism, confidentiality obligations in the licence agreement (Article 9), and EUROCONTROL monitoring of examiner integrity, validity and reliability. See the ELPAC Handbook and the Licence Agreement under documents and examiner training.

Does the test focus on discrete-point items, on grammar explicitly or on discrete vocabulary items?

No. Paper 1 items ask what an operational message means, not what a grammatical form is; there are no explicit grammar items and no isolated vocabulary items. Paper 2 rates observed performance in interaction rather than knowledge of language forms. See about the test.

Is a specific listening section with individual items included?

Yes. Paper 1 is a separate listening test with individual items, taking about 35 minutes. ICAO's accompanying note asks that this not be at the expense of assessing interaction: interaction is assessed separately and in full in Paper 2, and both papers must be passed.

Does the test include voice-only interaction?

Yes. Paper 2 Task 1 includes voice-only items, so interaction is assessed without visual support, as in radiotelephony. Paper 1 is delivered entirely through audio recordings.

Is the test is specific to aviation operations?

Yes. Both papers use radiotelephony and operational material, and there are separate instruments for controllers and for pilots. The operational basis of the input is described in the Doc 10197 self-assessment.

Does the test assess plain language proficiency in an aviation context?

Yes. Standard phraseology alone is not sufficient to pass: the tasks develop into non-routine situations that require plain English in an operational context, which is where the higher levels are elicited.

Does the test avoid items that are designed to elicit highly technical or very context-specific language?

Yes. Tasks are written to the operational role rather than to a specific unit, aircraft type or local procedure, so that no candidate is advantaged or disadvantaged by narrow technical knowledge.

Is the final score for each test-taker the lowest of the scores in each of the six ICAO language proficiency skills?

Yes. The overall ICAO level awarded for Paper 2 is the lowest of the six sub-ratings, in accordance with the ICAO rating scale criteria. See results.

Is a statement of evidence for test validity and reliability accessible to all decision-makers in plain, layperson language?

Yes. This Validity Argument section is that statement. It includes the construct, the inference-by-inference argument, and the operational monitoring of Paper 1 and Paper 2 with their reliability and rater statistics.

Is a description of the development process that includes the following information accessible to all decision-makers: a) a summary of the development calendar? b) a report on each development phase?

Yes, at summary level. ELPAC development began in 2004 and the first ELPAC test was released in 2007. Records from the original development process are still available, although they may no longer be directly relevant for the current versions of the test. The ELPAC test does not change frequently: further development is carried out on a project basis together with the Zurich University of Applied Sciences (ZHAW), and changes are always discussed with the ELPAC user organisations at the annual ELPAC User Group meetings.

Is an appraisal of the expected test washback effect on training accessible to all decision-makers?

Yes. See washback.

Is the rating process documented?

Yes, in the ELPAC Assessment Scheme and the ELPAC Handbook, both published under documents.

To fulfil licensing requirements, do at least two raters participate in the rating of tests, with a third expert rater consulted in case of divergent scores?

Yes. Paper 2 is rated by two accredited examiners, an English Language Expert and an Operational Expert (an operational or former air traffic controller or pilot). Where the two cannot reconcile their ratings, a third examiner is called in under the documented dispute-resolution mechanism. The observed escalation rate is reported in the Paper 2 monitoring.

a) Are initial and recurrent rater training documented? b) Are rater training records maintained? c) Are raters audited periodically and reports documented?

a) Yes. Initial accreditation (ELPAC ACR) and recurrent standardisation (ELPAC REF) are documented courses delivered by the EUROCONTROL Aviation Learning Centre; see examiner training. b) Yes. Examiner training activities are documented in the EUROCONTROL Learning Zone. c) Examiner rating behaviour is monitored statistically through the Paper 2 rater analysis, where examiner severity and fit are reported; see the Paper 2 monitoring. Periodic audit of individual examiners at a test centre sits with the centre and its local competent authority.

If rating is conducted using new technology, including speech recognition technology, is the correspondence of such rating to human rating, on all aspects of the Rating Scale, clearly demonstrated in layperson language?

Not applicable. Paper 2 is rated live by human examiners; no speech recognition or automated scoring of speech is used. Paper 1 responses are scored against a prescribed answer key.

Is a complete sample of the test published, including the following: a) test-taker documents (paper instructions, screen display, etc.)? b) interlocutor instructions or prompts? c) rater documentation (answer key, rating scale, instructions)? d) one complete sample of audio recordings (for listening sections or semi-direct prompts)? e) a demonstration of test-taker/interlocutor interaction?

Partly. A complete sample test is published for each role, with the real interface, candidate instructions and audio, which covers a) and d). The ICAO rating scale criteria and the rating arrangements are published in the Assessment Scheme under documents. Paper 2 interlocutor scripts, prompts and answer keys are live test material and are not published on this site; they are provided to authorities and user organisations through the ELPAC team under the licensing arrangements. e) No recorded demonstration of a Paper 2 interaction is published, for the same reason: it is live test material. The structure of the interaction is described on about the test, and the test format is shown in the published sample tests.

Is the test rating process documented, including instructions on the extent and nature of evidence that raters should collect?

Yes. The rating process, the examiner documentation and the evidence examiners record during Paper 2 are set out in the ELPAC Assessment Scheme and the examiner support material under documents.

Are the test instructions to the test-taker, the test administration team and test raters clearly documented?

Yes. Procedures and roles for test takers, test administrators and examiners are consolidated in the ELPAC Handbook under documents; candidate instructions are also visible in the published sample tests.

Are the requirements for equipment, human resources and facilities necessary for the test included in the instructions?

Yes. Technical requirements, roles and test centre requirements are part of the ELPAC Handbook under documents.

Is the testing location moderately comfortable, private and quiet?

Yes, by standard. EUROCONTROL is the test instrument developer. ELPAC tests are administered by ELPAC test centres. Through examiner and test administrator training, close contact with ELPAC user organisations and the conditions in the ELPAC Licence Agreement, EUROCONTROL ELPAC sets the standard. Test centres operate under oversight of the local competent authority.

Is a full description of test administration policies and procedures available to all decision-makers? Does it include the following: a) policies and procedures for retaking the test? b) score reporting procedures? c) record-keeping arrangements? d) plans for quality control, test maintenance and ongoing test development? e) purchasing conditions?

Yes. a) Re-sit arrangements are set out in the ELPAC Handbook and summarised under results. b) Score reporting is documented in the ELPAC Assessment Scheme: results are released to the ELPAC test centre administrator, who reports them to the candidate and, where applicable, to the licensing authority. c) Record-keeping arrangements, including retention periods, are set out in the ELPAC Test Security Policy. d) Quality control, test maintenance and ongoing development rest on item bank calibration and the operational monitoring published under operational validity, with development carried out together with the Zurich University of Applied Sciences and updates discussed at the annual ELPAC User Group meeting. e) The conditions under which an organisation becomes an ELPAC user organisation are set out in the ELPAC Licence Agreement template under documents.

Has a documented appeals process been established and made available to test-takers and decision-makers at the beginning of the testing process?

Yes, as a suggested process. ELPAC provides an appeals process in the ELPAC Handbook. ELPAC test centres may use this process, but are often bound to the appeals process prescribed and approved by the local competent authority.

Is a full description of security measures required to ensure the integrity of the testing process documented and available to all decision-makers?

Yes. The ELPAC Security Policy is published on this website under ELPAC Policies and describes the security measures required to protect the integrity of the testing process.

In the case of semi-direct prompts, are there adequate versions of the test to meet the needs of the population to be tested with respect to its size and diversity?

Yes. Paper 1 forms are assembled from a calibrated item bank, and Paper 2 uses several scripted scenarios per role written to a common task specification. Item difficulties are monitored on a common scale; see the Rasch analysis.

Are test questions and prompts held in confidence and not published or in any way provided to test-takers prior to the test event?

Yes. Only the dedicated sample tests are public. Live Paper 1 items and Paper 2 scripts are not published and are not made available to candidates before the test event.

Is a documented policy for all aspects of test security accessible to all decision-makers?

Yes. The ELPAC Security Policy covers all aspects of test security and is published on this website under ELPAC Policies.

Are all proficiency tests of speaking ability involving interaction between the test-taker and interlocutor recorded on audio or video media?

Yes. ELPAC Paper 2 tests are audio-recorded, and the recordings are kept as part of the candidate test record; see the ELPAC Test Security Policy.

Are evaluation sheets and supporting documentation filed for a predetermined and documented period of time of sufficient duration to ensure that rating decisions can no longer be appealed?

Yes. Candidate records, including examiner assessment scores, are kept for 10 years. Where the final result is ICAO Level 6, records are kept for 50 years. The retention periods are set out in the ELPAC Test Security Policy.

Is the record-keeping process adequate for the scope of the testing and documented?

Yes. The record-keeping process meets regulatory requirements and is documented in the ELPAC Test Security Policy. ELPAC test centres may also choose to keep records of their candidates on file themselves.

Is the score-reporting process documented, and are scores retained for the duration of the licence?

Yes. The scoring process is set out in the ELPAC Assessment Scheme, which is included in the ELPAC Handbook. Scores are retained for 10 years, or 50 years where the final result is ICAO Level 6, as set out in the ELPAC Test Security Policy.

Are results of testing held in strict confidence and released only to test-takers, their sponsors or employers, and the civil aviation authority, unless test-takers provide written permission to release their results to another person or organization?

Yes. Test results are available only to the ELPAC test centre administrator, who is responsible for reporting scores to the candidate and, where applicable, to the licensing authority. Results are not released to anyone else without the written permission of the candidate; see the ELPAC Test Security Policy.

Has an aviation language TSP provided clear information about its organization and its relationships with other organizations?

Yes. ELPAC is the EUROCONTROL English Language Proficiency test, developed and maintained with the Zurich University of Applied Sciences (ZHAW) and operated with ENOVATE. The roles of the parties and of user organisations are described on implementing ELPAC and in the ELPAC Licence Agreement under documents.

If a TSP is also a training provider, is there a clear and documented separation between the two activities?

Yes, by structure. EUROCONTROL ELPAC develops and maintains the test instrument but does not provide language training. ELPAC test centres are responsible for ensuring a clear separation between testing and training activities; this is usually also part of the oversight by the local competent authority in light of the local test centre approval. The responsibilities of each party are summarised in the ELPAC Test Security Policy.

Does the TSP employ sufficient numbers of qualified interlocutors and raters to administer the required tests?

Yes, by structure. EUROCONTROL ELPAC, as test instrument developer, does not itself employ interlocutors and raters. The ELPAC team does provide ELPAC examiner (interlocutor and rater) training for ELPAC user organisations, and centres are responsible for deploying sufficient qualified examiners to administer their tests.

Has the TSP provided an explanation of how the test is maintained, including an explanation of how ongoing test development is conducted?

Yes. Test maintenance is described in the ELPAC Handbook and Assessment Scheme, both versioned and dated under documents, and the ongoing statistical monitoring that feeds item and form maintenance is published under operational validity monitoring.

Are all testing team members familiar with the following ICAO publications? a) the relevant SARPS and Recommended Practices of Annex 1? b) holistic descriptors (Appendix 1 to Annex 1) and the ICAO Rating Scale (Attachment A to Annex 1)? c) Manual on the Implementation of ICAO Language Proficiency Requirements (Doc 9835)? d) ICAO Rated Speech Samples CD?

Yes. All ELPAC examiners have undergone ELPAC examiner accreditation training and regularly follow ELPAC examiner refresher training. Familiarity with the ICAO publications listed is part of the ELPAC examiner accreditation course, and any changes or updates are covered by refresher training.

Does the test design and development team include individuals with aviation operational, language test development, and linguistic expertise?

Yes. ELPAC is developed and maintained by EUROCONTROL together with ZHAW, combining operational expertise (controllers and pilots), applied linguistics and language testing expertise.

Do test administrators and interlocutors have a working knowledge of the test administration guidelines published by the test organization?

Yes. Examiners and test administrators are nominated and must complete the ELPAC ACR accreditation course, which covers the ELPAC procedures and documentation; see examiner training.

Do interlocutors demonstrate language proficiency of at least ICAO Extended Level 5 in the language to be tested and proficiency at Expert Level 6 if the test is designed to assess ICAO Level 6 proficiency?

Yes. Language proficiency at the required ICAO levels is part of the ELPAC examiner accreditation course pre-selection (entry requirements).

Have interlocutors successfully completed initial interlocutor training?

Yes. Nobody may act as an ELPAC examiner without accreditation through the ELPAC ACR course, which includes the interlocutor role in Paper 2.

Have interlocutors successfully completed recurrent interlocutor training at least once each year?

In principle yes. EUROCONTROL ELPAC offers ELPAC examiner refresher training every month, free of charge for accredited ELPAC examiners. The ELPAC guideline is that examiners complete refresher training at least once every three years, or after a period of inactivity of six months or more. Many local competent authorities that oversee ELPAC test centres require annual refresher training. Training activities are documented in the EUROCONTROL Learning Zone.

Do interlocutors have appropriate aviation operational or language testing expertise, or both?

Yes, by design: each Paper 2 is conducted by an Operational Expert (an operational or former air traffic controller or pilot) together with an English Language Expert (a qualified English language professional).

Do raters demonstrate language proficiency of at least ICAO Extended Level 5 in the language to be tested, and Expert Level 6 if the test is designed to assess ICAO Level 6 proficiency?

Yes. This item duplicates 6.3.8.5 in the ICAO checklist. The same pre-selection entry requirements apply to ELPAC raters: language proficiency at the required ICAO levels is part of the ELPAC examiner accreditation course.

Are raters familiar with aviation English and with any vocabulary and structures that will likely be elicited by the test prompts and interactions?

Yes. The ELPAC ACR course works through the ELPAC task types and materials with calibration exercises, and the Operational Expert brings first-hand radiotelephony experience to every assessment.

Have raters successfully completed initial rater training?

Yes. Accreditation through the ELPAC ACR course is a precondition for rating ELPAC Paper 2.

Have raters successfully completed recurrent rater training at least once each year?

This item duplicates 6.3.8.7 above. ELPAC examiner refresher training is offered every month, free of charge for accredited examiners. The ELPAC guideline is that examiners complete refresher training at least once every three years, or after a period of six months or more of inactivity. Many local competent authorities responsible for oversight of ELPAC test centres require annual refresher training. ELPAC examiner training activities are documented in the EUROCONTROL Learning Zone.