Examiner Standardization: How to Calibrate Assessors in Aviation Training

Examiner standardisation is the routine that makes different examiners, instructors and assessors reach the same result when they watch the same performance. In aviation training it means a written standard, practice on reference cases, regular agreement checks and records. Some rules require parts of it, but none of the texts reviewed here prescribe the method.

This guide is for heads of training, quality managers and chief instructors who run more than one assessor. It shows what the rules say and leave open, a calibration cycle you can run, a worked agreement calculation, and a drift-monitoring plan. It is not legal advice; rules differ by authority, so check your Competent Authority. Regulatory facts are as of November 2026.

Key takeaways

• Standardization works only if assessors grade reference cases on their own before anyone discusses them; otherwise you measure deference, not agreement.
• Percent agreement overstates consistency when most candidates pass: in the worked example below, 90% agreement equals a kappa of only about 0.51.
• Rules and authority guidance fix that standardization must happen for some roles; the content, frequency, statistic and tolerance are usually yours to define and document.
• Monitor assessors between sessions with pass rates, mean grades, hidden reference cases and second-assessor samples, and treat gaps as a prompt to investigate, not as proof of fault.

What is examiner standardisation, and why does aviation training need it?

Examiner standardisation is a recurring process that aligns how every examiner, instructor and assessor applies one standard to one performance. It matters because two qualified assessors can watch the same candidate and disagree, so a pass or fail then depends on who was in the room rather than on what the candidate did.

Assessors differ in three ways. Some are stricter or more lenient overall (severity). Some read the same standard differently (interpretation). Some weigh the same evidence differently when a candidate is borderline. Standardization targets all three.

EASA’s rationale document for Opinion No 08/2019 (A) gives the regulator’s reasoning for EBT programs. It states “Concordance should be required to prevent drift in instructor quality over time” (quoted), and its proposed guidance wording says “‘Instructor concordance’ is also called ‘inter-rater reliability'” (quoted). Both come from an explanatory appendix to a proposal, so confirm the wording in the final rule texts that apply to you.

What do the rules require for examiner standardisation, and what is left to you?

The texts reviewed require standardized personnel in specific roles, such as instructors in an evidence-based training (EBT) programme, and point to examiner standardisation courses. They do not set an agreement statistic, a tolerance or a session design. Those choices belong to the organization, within what its authority accepts.

The table sets out what each source says and what it leaves open. The flight-crew sources below do not cover Part-147 or Part-66 assessors, cabin crew, dangerous goods or ground handling; those roles have their own rules, so check your Competent Authority.

The ORO.FC.146 wording above comes from the UK CAA Regulatory Library copy of the text that applied before 30 October 2025. The Malta row comes from the Transport Malta Instructor Standardisation Guidelines, which is guidance, not a regulation. Neither source sets a statistical threshold.

What does a standardization cycle look like in practice?

A standardization cycle has five repeating steps: fix the standard in writing, train assessors on reference cases, check agreement, monitor live grading, and record and act on what you find. Repeat it at least as often as your authority requires, and again whenever the standard or grading sheet changes.

Step 1: Fix the standard. Write each behavior or criterion so one person can observe it, with the pass condition inside the wording. The CBTA assessment guide shows how to write observable behaviors.

Step 2: Train on reference cases. Use a set of anchor performances with agreed grades (below).

Step 3: Check agreement. Every assessor grades the same cases independently, then you compare.

Step 4: Monitor live grading. Track patterns between sessions (see the drift section).

Step 5: Record and act. Keep the evidence, update guidance and re-calibrate anyone outside your tolerance.

How do you build a set of anchor performances?

An anchor set is a small library of performances whose correct grade senior assessors have agreed in advance. Include clear passes, clear fails and borderline cases for each competency, with a written reason for every reference grade, so assessors can learn from the reasoning and not only the answer.

Have two or three senior assessors grade each case independently, discuss splits, and record the agreed grade and the evidence behind it. If those seniors cannot agree on a case, the standard is ambiguous; rewrite the wording before you use the case to calibrate anyone. Use simulated or anonymized performances with consent, and never real exam items.

Ofqual’s 2014 description of marking in general qualifications is not aviation, but its structure transfers. It describes practice scripts and approval scripts with “true scores” agreed by senior staff, examiners cleared to mark only after acceptable approval marking, and hidden “seeds” with known scores at about 5 percent of an examiner’s allocation in on-screen marking. It also describes team leaders sampling live marking, with tolerances that vary by board.

How should a calibration session run?

Run the session so that grades are recorded before discussion: assessors grade each case alone, submit the grades, then see the spread, then discuss the splits against the written standard. This order stops the most senior voice from setting every outcome.

Pro tip: Grade Before You Talk. Collect every assessor’s grade in writing before showing anyone else’s. If a senior examiner speaks first, later grades drift toward theirs and your agreement figure measures deference, not shared judgment.

Discuss only the cases where grades split, and anchor each discussion on the behavior wording, not on the person. Where the wording caused the split, fix the wording. Where one assessor misapplied it, note the point for follow-up. Then run one fresh case to see whether the discussion changed anything.

Do not rely on the meeting alone. In a 2010 Cambridge Assessment experiment with 24 psychology examiners (Raikes, Fidler and Gill, Research Matters, issue 10), standardization improved marking accuracy with or without a meeting, and the meeting’s added benefit was described as “variable, small and questionable.” It was school exam marking with a small sample, so treat it as a caution, not a rule: comparing your marks to reference grades, with written feedback, does much of the work.

How do you measure whether assessors agree?

Compare each assessor’s grades with a reference or with each other on the same performances, and report percent agreement and Cohen’s kappa for each competency. Kappa measures agreement beyond what chance alone would produce, so it is safer than percent agreement when most candidates get the same grade.

McHugh’s 2012 paper in Biochemia Medica explains that percent agreement does not account for raters guessing, so it may overestimate agreement. Kappa runs from -1 to +1, where 0 is the agreement expected from chance. McHugh argues that values below 0.60 mean inadequate agreement for research, while Cohen’s scale calls 0.61 to 0.80 substantial. Those are health-research rules of thumb, not aviation rules, so set your own minimum for each decision’s risk and write it down.

What does a worked agreement calculation look like?

The example below is invented for illustration. Two assessors independently grade 60 recorded practical attempts as pass or fail. The table shows how many attempts fell into each combination of their grades.

Observed agreement is (50 + 4) / 60 = 0.90. Each assessor passes 53 of 60 attempts, or 0.883, so chance agreement is 0.883 × 0.883 + 0.117 × 0.117 = 0.794. Kappa is (observed minus chance) divided by (1 minus chance): (0.900 – 0.794) / (1 – 0.794), about 0.51.

That is the point: 90% agreement looks strong, but because both assessors pass most attempts, chance alone would produce 79%. Kappa can also move with the pass rate, so read the table, not only the single number. Six attempts split, and in three of them Assessor A passed a candidate Assessor B failed. In a safety-critical setting, send every split to a third assessor.

For a five-point competency scale, report exact matches and matches within one grade separately, and ask a psychometrician about a weighted kappa. To find which direction an assessor errs against a reference, the CRM assessment guide works through that comparison.

How do you spot assessor drift between standardization sessions?

Track each assessor’s live results against their peers and against reference cases, and investigate large gaps. Useful signals are pass rate, mean competency grade, accuracy on hidden reference cases, agreement on a second-assessor sample, and how often grades come with written evidence.

Transport Malta’s notice asks organizations to monitor for “inconsistent pass/fail patterns or high drop-out rates tied to specific instructors” (quoted from section 7.3), with corrective actions such as retraining or re-standardisation. The table turns that idea into a monitoring plan. Tolerances are yours to set; no text reviewed gives a number.

Pro tip: Test The Reference Set Too. If three experienced assessors split on one reference case, rewrite the standard for that case before you retrain anyone. A confusing case teaches the whole team the wrong lesson.

Differences in pass rate can reflect which candidates an assessor saw, so treat gaps as questions, not findings. Assessor performance data is also personal data about staff; check your data-protection and employment rules before you share it.

What records show that standardization took place?

Records should show who attended, which version of the standard and grading sheet applied, which reference cases were used, each assessor’s grades, the agreement result and the actions taken. Transport Malta’s guidance says minutes are recorded, distributed and archived, and that observation records form part of each instructor’s training record (paraphrased).

The sources reviewed set no retention period for these records, so ask your authority and follow your quality system. A practical record set has the following parts:

• Session date, attendees and facilitator, with the standard and sheet version
• Reference case identifiers and the agreed reference grades with reasons
• Each assessor’s independent grades, before discussion
• Agreement figures per competency, and the threshold you set
• Disputed points, decisions made and wording changes
• Follow-up actions, owners and the date each was closed

Can software help you calibrate assessors?

Software can hold reference cases, collect grades before discussion, calculate agreement and keep the history. It cannot decide your standard or make a program meet the rules; that judgment stays with your organization and your authority. A spreadsheet can do the same job at small scale if versions are locked.

SimpliAssess’s published features page lists multi-user collaboration, item banking, test management, tag management and analytics and reporting. It does not describe manual scoring, rubrics, second-marker workflows or rater agreement reports, so ask about those directly. Useful questions for any vendor:

• Can I store reference performances with agreed grades and reasons?
• Do assessors submit grades blind, before seeing others’ grades?
• Does it report agreement per competency and per assessor, and show how it is calculated?
• Is each grading sheet version locked and linked to the grades recorded on it?
• Can I export the full history for an audit or authority review?

What should you do next?

Pick one competency or practical task and build a set of eight to ten reference cases, covering clear passes, clear fails and borderlines. These counts are a practical starting point, not a rule. Have two senior assessors set the reference grades and write the reasons.

Then run a baseline session: every assessor grades the cases alone, you calculate percent agreement and kappa, and you discuss the splits. Fix any wording that caused them, set your own agreement threshold in writing, schedule the next session, and confirm with your Competent Authority what it expects to see.

If you want to see where standardization dates and assessor records could sit next to crew training records, you can ask for a SimpliTrain demo. Its demo page lists scheduling, assessments and certification tracking, but its published pages do not describe assessor calibration, so raise that question in the demo. For checklist design, see the practical assessment checklists guide.

Frequently asked questions

What is examiner standardisation?

Examiner standardisation is the process of aligning examiners, instructors and assessors so they apply one written standard the same way. It usually combines a shared standard, practice on reference cases, an agreement check and records. In some aviation roles a rule or authority requires it; the method is typically left to the organization to design and document.

How often should examiners be standardised?

The ORO.FC.146 text reviewed requires initial and recurrent standardisation for instructors in an EBT programme but gives no interval. Transport Malta’s guidance sets at least annual sessions for organizations it oversees, and recommends semi-annual where feasible. Your Competent Authority and quality system decide your interval, so check them.

What is inter-rater reliability in aviation training?

Inter-rater reliability is how far different assessors give the same grade to the same performance. EASA’s rationale document for Opinion No 08/2019 says instructor concordance is also called inter-rater reliability. You can measure it with percent agreement and Cohen’s kappa, and by checking each assessor against agreed reference grades.

What is a good kappa score for assessors?

No aviation text reviewed sets a kappa threshold. Cohen’s published scale calls 0.61 to 0.80 substantial, while McHugh argues values below 0.60 are inadequate for research. Those are rules of thumb from other fields. Set your own minimum based on how serious a wrong pass would be, and document the reasoning.

Is percent agreement enough to prove assessors are consistent?

No. Percent agreement ignores agreement that happens by chance, so it can look high when most candidates pass. In the worked example above, 90% agreement produced a kappa of about 0.51. Report both figures, show the table of splits, and review every disagreement where one assessor passed and another failed.

How many reference performances do you need to calibrate assessors?

No source reviewed sets a number. As a starting point, cover clear pass, clear fail and borderline performances for each competency, and refresh the set when the standard changes or assessors start to remember cases. Treat a kappa from only a dozen cases as a rough signal, and use more cases for decisions that carry higher risk.