Measuring Aviation Training Effectiveness With Assessment Data
Training effectiveness is the degree to which a course produces the competence and on-the-job performance your organization needs, not just completed courses and passed tests. Assessment data covers part of that picture. It shows what people learned and where they struggle, but on its own it cannot show what changed in the hangar, on the ramp or on the flight deck.
This post is for heads of training, quality managers and chief instructors at Approved Training Organisations, Part-147 maintenance training organisations, FAA-certificated schools, airline training departments and aviation course providers. It shows what the rules read say about effectiveness, then gives a measurement plan that links assessment results to records you already hold. It is not legal or airworthiness advice.
The sources below were read through a fetch tool in November 2026, and wording marked (quoted) came from the pages cited. Rules differ by country and change, so check your competent authority before you change how you measure or record training.
Key takeaways
- The regulation texts read ask for trained, competent people and for evaluating whether safety controls work, but none sets one method for judging training effectiveness in general.
- Assessment data mainly answers Kirkpatrick level 2 (learning) and, with calibrated practical assessments, level 3 (behavior); level 4 needs operational data such as audit findings.
- A first-attempt pass rate shows whether a cohort cleared a bar, not which topics were weak, so read it beside item-level and topic-level scores.
- Safety events are too rare to show a training effect, so track leading indicators tied to the trained task and claim a contribution, not proof of cause.
What do regulators and aviation authorities say about training effectiveness?
The texts read require trained and competent personnel and, for SMS holders, assessment of whether safety risk controls work. Some set an outcome number, such as a minimum exam pass rate. Authority guidance from CASA and ICAO uses the four Kirkpatrick levels. No text read prescribes one effectiveness method for all training.
The table shows what each source says (as of November 2026), whether it is a rule or guidance, and what it leaves to you.
| Source | What it says | What it leaves open |
|---|---|---|
| FAA 14 CFR 5.73(a)(3), safety performance assessment (rule) | An SMS holder must “Evaluate the effectiveness of the safety risk controls” it established, and identify ineffective ones (quoted in part) | Whether training counts as one of your controls, and what data you use |
| FAA 14 CFR 5.71(a), safety performance monitoring (rule) | Data must be acquired through items including “Monitoring of operational processes.” and “Investigations of incidents and accidents.” (both quoted); it sets no indicators, targets or thresholds | Which metrics, and how training results feed in |
| FAA 14 CFR 147.25, minimum passage rate (rule) | A certificated aviation maintenance technician school must keep a pass rate “for the most recent 3-year period”; for students who test within 60 days after graduation, “at least 70 percent of students must pass” (quoted in part) | Why students fail, and learning that sits below the pass line |
| FAA 14 CFR 141.5, pilot school certificate (rule) | An applicant must have “established a pass rate of 80 percent or higher on the first attempt” on listed tests (quoted); the text read sets it as a condition for issuing the certificate and does not say how the rate is calculated | Calculation method and what to do when topics are weak |
| EASA Part-ORO, ORO.GEN.200(a)(4) (rule) | The management system includes “maintaining personnel trained and competent to perform their tasks” (quoted) | How competence or training effectiveness is shown |
| UK CAA Standards Document 80, ATQP guidance (July 2013, guidance) | The programme is “supported by a data monitoring / analysis system”; the training record system should “show weaknesses in crew performance”; training is reviewed “at least every 6 months” (all quoted in part) | Applies to UK operators in an alternative training programme; check current UK and EASA material |
| CASA, SMS for Aviation: A Practical Guide, book 5 (revised 2014, guidance) | “You need to measure how effective your training program is,” using four levels from feedback to material results (quoted in part) | Australian guidance, not a rule; no test design or interval |
| ICAO, Measuring Effectiveness of AVSEC Training (June 2023, guidance) | Uses the Kirkpatrick levels plus ROI, with tests at level 2 and a follow-up at level 3 best placed between three and six months after training (paraphrased from the French edition) | Written for aviation security training; not for other course types |
Three layers apply. The regulation sets duties such as training, competence and pass rates. Your authority decides what evidence it accepts, so ask it what it expects to see. You choose the metrics, intervals and thresholds, and you should write them into your training procedures.
Which assessment data shows training effectiveness at each level?
Use the four Kirkpatrick levels as a filing system. Assessment results mainly cover level 2, learning, and level 3, behavior, when you use practical assessments. Levels 1 and 4 need feedback forms and operational data. Kirkpatrick Partners define each level by what it measures, so put each data source where it belongs.
Kirkpatrick Partners write that “Level 3: Behavior measures the degree to which the target audience performs the critical behaviors in its environment” and that “Level 4: Results measures the degree to which targeted organizational outcomes occur” (both quoted). The table is a measurement plan for one course. The timings are suggestions unless noted.
| Level | Question it answers | Assessment evidence | Other aviation evidence | Timing and caution |
|---|---|---|---|---|
| 1. Reaction | Was the course relevant to the learner’s role? | None; use a short survey | Instructor notes | End of course. Satisfaction does not show competence. |
| 2. Learning | Did learners gain the intended knowledge? | Pre/post scores by objective, first-attempt score by topic, item facility, a delayed retest | Your own course records | Before, at the end, and about 3 months later (suggested). Use parallel forms; small cohorts swing widely. |
| 3. Behavior | Do people perform the critical tasks correctly at work? | Practical checklist results and scenario items, scored by calibrated assessors | Supervisor observation, task audits | Three to six months after training (ICAO AVSEC range). Watch assessor drift and observer bias. |
| 4. Results | Did the targeted outcomes move? | First-attempt pass rate where a rule sets one | Audit findings and repeat findings on the trained process, hazard reports, corrective-action closure | Each review cycle you set. Many causes act at once; see the section on rare events. |
Our guides to pre and post course assessments and to turning EBT grading data into training needs cover the level 2 and level 3 mechanics, so this post does not repeat them.
Does a first-attempt pass rate show training effectiveness?
Not by itself. A pass rate is a lagging, whole-course number. It tells you whether a cohort cleared a bar, not which topics were weak or whether the questions were sound. Treat it as a level 4 result, and use item-level and topic-level data to explain it.
FAA 14 CFR 147.25 shows why. It counts students, not topics, and its text read treats a failure on any one test in the 60-day window as a student failure unless that test is later passed in the window. Our post on FAA Part 147 and ACS-based testing walks through that rule in depth. The same logic applies to the 80 percent first-attempt condition in 14 CFR 141.5.
Here is an invented example, not a benchmark. Forty graduates test inside the window and 30 pass: 75 percent, above the 70 percent line. The topic view shows the cohort answered 52 percent of electrical fundamentals items correctly, while five other topics sat between 74 and 86 percent. The pass rate says “fine”; the topic data says revise the electrical module before the next cohort. With 40 students, one student moves the rate by 2.5 points, so do not chase small swings.
Pro tip: Chart Topic Scores Beside The Pass Line. For every cohort, plot topic-level scores next to the pass rate, and mark any topic more than about 15 points below your course average for review. The 15 points is a starting assumption, not a rule, so adjust it after two or three cohorts.
How do you measure training effectiveness when safety events are rare?
Use leading indicators tied to the trained task, because accidents and serious incidents are too rare to show a training effect. Observed task performance, audit findings on the trained process and the quality of hazard reports can move within a review cycle. Treat each indicator as a prompt to investigate, not as proof.
The FAA text in 14 CFR 5.73 asks an SMS holder to evaluate whether its safety risk controls work. If your SMS lists training as one of those controls, that is where its effectiveness gets judged; that is this post’s reading, not a statement in the rule. The UK CAA’s ATQP guidance lists the data an operator should correlate, including FDM, safety reports, electronic training records and SMS data.
Four indicators that fit most training providers:
- Task error rate on a checklist item for the trained task, scored by calibrated assessors.
- Audit or quality findings on the trained process, and whether the same finding repeats.
- Hazard or occurrence reports about that process, read for quality and timeliness, not only count.
- Time to close corrective actions that name a training cause.
Be careful with report counts. A rise after training may mean people trust the reporting route more, and a fall may mean less reporting, not fewer hazards. Our post on SMS knowledge checks for aviation staff explains why check scores and reporting behavior are separate measures.
How do you separate the effect of training from other changes?
Compare against something and log everything else that changed. Use an earlier cohort, a later cohort trained after a delay, or a similar group. Without a comparison, a better result may come from a new procedure, a new instructor or an easier exam, and the course gets credit it did not earn.
Kirkpatrick Partners say their contributive ROI approach “offers a more credible approach than trying to isolate training as the sole cause” (quoted), so claim a contribution, not sole credit. Our guide to training ROI and assessment scores covers attribution and the money calculation, which this post leaves aside.
Pro tip: Keep A Change Register. Keep one log of every change to procedures, instructors, exam versions, equipment and fleet during the measurement period, with dates. When a result moves, the register is the first place to look for competing explanations.
What should a training effectiveness review decide each cycle?
It should decide, for each course, whether to keep it, revise it or retrain people. Agree the thresholds before you see the data, check the data’s quality first, and record the decision. A review that only reports numbers does not show that anyone acted on them.
Step 1: Check the data. Run item analysis and check assessor agreement before drawing conclusions. A flawed item or a drifting assessor makes a training problem look real or hides one.
Step 2: Compare with baseline and thresholds. Use the last two or three cohorts and the thresholds you wrote down in advance.
Step 3: Choose an action. Revise content, fix or retire an item, recalibrate assessors, retrain a group, or take no action. Name an owner and a date.
Step 4: Record the decision. Keep the data used, the decision, the owner and the follow-up check. Ask your authority what record format it expects.
The interval is your call. The UK CAA guidance for ATQP operators asks for a review “at least every 6 months” that is auditable by the Authority, but that regime covers those operators only. Many providers pick six or twelve months and tie it to their quality system.
What should you do next?
Pick one course and fill in the measurement plan table for it. Pull the last two cohorts’ first-attempt scores by topic, note what you already hold for levels 3 and 4, and mark the gaps. One course is enough to find out which data you cannot yet get.
Then write your review interval, thresholds and record fields into your training procedure, and ask your competent authority what evidence it expects. If a client later asks for a money figure, the training ROI guide linked earlier is the place to take the data next.
Frequently asked questions
What is training effectiveness?
Training effectiveness is how well training produces the knowledge, skills and on-the-job performance an organization intended. It is broader than completion or pass rates. In aviation it usually combines assessment results, observed task performance and operational data such as audit findings, so you need more than a test score to judge it.
What are the four levels of training evaluation?
They are reaction, learning, behavior and results, from the Kirkpatrick model. Reaction covers whether the audience found the training relevant, learning covers knowledge gained, behavior covers critical behaviors performed at work, and results covers targeted organizational outcomes. Assessment data mostly informs learning and, with practical assessments, behavior.
Can assessment scores prove that training worked?
No. Scores show that learners know more or can do more at the time of the test. They do not show that the change came from the course, or that it carried over to work. Add a comparison group, delayed testing and operational data, and say “contributed to” instead of “caused.”
Do the FAA or EASA require training effectiveness evaluation?
The texts read set related duties, not one method. FAA 14 CFR 5.73 asks SMS holders to evaluate the effectiveness of safety risk controls, and 147.25 sets a 70 percent pass rate for maintenance technician schools. EASA ORO.GEN.200(a)(4) requires trained and competent personnel. Check your competent authority for what applies to you.
When should you measure whether training changed behavior on the job?
ICAO’s AVSEC training guidance places the level 3 follow-up between three and six months after training. That range comes from security training, so treat it as a reference point. Choose a window long enough for people to use the skill but short enough that other changes do not swamp the result.
What is the easiest training effectiveness measure to start with?
Start with pre-test and post-test scores by objective on one course, plus first-attempt scores by topic. You probably have that data already. Add one level 3 measure, such as a calibrated practical checklist, after the first review cycle, and one operational indicator after that.
