Aviation Exam Analytics: Spotting Weak Topics Across a Cohort
Exam analytics is the practice of turning a cohort’s exam results into decisions: which topics need re-teaching, which questions need review, and which results are too thin to act on. For an aviation training team, the task is to find weak topics, such as an ATA chapter or a syllabus area, without mistaking a flawed question or a small sample for a teaching problem.
This post works through an example built on invented data: 24 learners, 60 multiple-choice items and six ATA chapters. It shows what each view of the data can and cannot tell you and gives a decision table for separating teaching gaps from item problems.
The example is illustrative only, and no item in it stands for an official exam question. Nothing here is legal or airworthiness advice, so check your competent authority and your approved procedures before changing items, forms or pass marks.
Key takeaways
- Read item flags before topic means: in the worked example, a miskeyed item sat inside a chapter whose mean (62%) matched the cohort mean.
- A topic mean rests on its items and learners: with 24 learners, each topic’s 95% interval ran roughly 10 to 15 points either side of the mean, so small gaps between chapters are noise.
- A teaching gap, a weak item and a different cohort look alike in a summary table but need different actions, so separate them with a written decision table.
- Set action thresholds before results arrive, and let your approved procedures, not the analytics, decide changes to items or pass marks.
What is exam analytics, and what can it show an aviation training team?
Exam analytics is the analysis of results at three levels: the cohort, the topic and the item. It shows where learners struggled and which questions behaved oddly. It does not show why learners struggled, and it does not show practical competence. Those answers come from instructors, item reviewers and observation records.
Cohort-level analysis covers score distribution and pass rate. Topic-level analysis covers the cohort’s score on each ATA chapter or syllabus area. Item-level analysis covers the share who answered correctly, how well the item separates strong from weak learners, and which wrong options were chosen. The earlier post on item analysis for trainers covers the item formulas; this post builds on them.
What does a published weak-topic report look like in aviation?
The FAA publishes one at national level. Its quarterly lists of frequently missed Airman Certification Standards (ACS) codes show, for several knowledge tests, the codes applicants missed most often and the percent missed. As of November 2026, the latest quarterly list on the FAA’s test statistics page covers the second quarter of calendar year 2025.
The FAA test statistics page says that “Knowledge test ACS code statistics are prepared quarterly by the FAA’s Safety Analysis Branch” (quoted). The list reviewed for this post identifies each topic only by its ACS code, with no written description.
Two lessons carry over. A national list says nothing about your own cohort, and a topic report is only as useful as the code behind it: every item needs a topic tag that maps to a syllabus reference, as described in the post on structuring an aviation question bank by ATA chapter.
What data does a cohort analysis need before you open the results?
You need every learner’s score on every item, plus tags on each item and on each sitting. Without item tags you cannot roll results up into topics, and without sitting details you cannot compare classes or forms fairly.
The minimum fields are a permanent item ID, a primary topic, a syllabus reference and the form version for each item. For each sitting you need the learner ID, class, instructor or base, date, attempt number, and whether the learner sat the exam or only enrolled, because pass rates change with the denominator, a point covered in the post on assessment reports for corporate clients.
How do you use exam analytics to find weak topics in a cohort?
Calculate each topic’s mean score per learner, add an interval around it, then open the items inside any topic that looks low or odd. Only then decide whether the topic is weak. The worked example uses invented data: 24 learners, 60 items and six ATA chapters, with an overall mean of about 62%.
What does the topic-level view show?
Table 1 summarizes the invented cohort by ATA chapter. The interval is the 95% confidence interval for the topic mean, calculated from each learner’s score on that chapter. It reflects variation between learners only and cannot tell you whether the items fairly represent the topic.
| ATA chapter (invented data) | Items | Topic mean | 95% interval | Items under 30% correct | Items under 0.15 discrimination | First reading |
|---|---|---|---|---|---|---|
| 21 Air conditioning and pressurization | 8 | 69% | 58 to 81 | 0 | 0 | No concern; wide interval |
| 24 Electrical power | 12 | 69% | 57 to 81 | 0 | 0 | No concern |
| 27 Flight controls | 12 | 62% | 51 to 73 | 0 | 1 (negative) | Mean looks ordinary; check the flagged item |
| 29 Hydraulic power | 12 | 41% | 31 to 52 | 4 | 0 | Gap candidate; well below the cohort mean |
| 32 Landing gear | 12 | 74% | 64 to 84 | 0 | 1 (0.13) | Strong; three items above 85% correct |
| 34 Navigation | 4 | 47% | 32 to 62 | 1 | 0 | Too few items for a verdict |
ATA 29 is the clear candidate. Its interval sits entirely below the 62% cohort mean, and 12 of 24 learners scored under 50% on it. ATA 34 also looks low, but it has four items, so each item moves a learner’s topic score by 25 points and the interval is about 15 points either side. That topic goes on a watch list, not into a re-teaching plan.
What does the item-level view add?
It shows where inside a topic the problem sits, and it exposes faults a topic mean hides. In ATA 29, four of the 12 items were answered correctly by only 21% to 29% of learners (5 to 7 of 24), and all four discriminated positively (0.26 to 0.44, measured as the correlation between the item and the rest of the test). Another item in the same chapter was answered correctly by 79% with a discrimination of 0.82. The weakness is concentrated, so check whether those four items share a syllabus reference.
In ATA 27, one item was answered correctly by 75% of learners but had a discrimination of negative 0.29. Of the six highest-scoring learners, 3 got it right; of the six lowest-scoring, all 6 did. Penn State’s guide to item analysis says negative discrimination can signal a scoring-key error, guessing by weak learners or confusion among strong learners. Nothing in the chapter mean revealed it.
Pro tip: Read Item Flags Before Topic Means. A topic mean averages good and bad items together, so a miskeyed question can hide inside a chapter that looks fine. Sort every item by discrimination first and send the negative ones to review before you read any topic table.
How many items and learners does a topic need?
No guide reviewed for this post sets a minimum number of learners for item or topic statistics, so set your own and write it down. Short topics are the larger risk: one item swings a four-item topic.
A 2012 Cambridge Assessment research report by Tom Benton found that on a notional 180-mark test, subscores based on fewer than 30 marks added little over the total score. That finding concerns scores for individual learners under favorable assumptions, and cohort means are steadier because errors average out. Even so, the 12-item chapters here rest on few questions, so use topic results to plan cohort-level revision, not to give an individual learner a topic verdict.
Is a weak result a teaching gap, a weak item or a different cohort?
It depends on the pattern. Low results spread across most items in a topic point to a teaching or coverage gap. Low results driven by one or two items point to the items. A gap in one class but not another points to delivery or starting level. Table 2 maps each pattern to a first check and a typical action.
| Pattern in the data | Most likely explanation | Check first | Typical action |
|---|---|---|---|
| Topic mean low; most items hard; discrimination positive | Teaching or coverage gap, or low starting level | Syllabus coverage against the course plan; pre-course results | Targeted revision, then retest on new items |
| Topic mean low because one or two items are very hard | Item problem, or one sub-topic missed | Key, wording, syllabus references | Item review; revise the sub-topic only if confirmed |
| Item with negative or near-zero discrimination | Wrong key, ambiguous stem or guessing | Answer key and option counts | Review before the item is used again |
| One class or base lower than others on the same form | Delivery, instructor or starting-level difference | Class sizes, earlier results, delivery notes | Compare with pre-course data before concluding |
| Topic has very few items | Not enough evidence | Item count per topic | Watch list; add items to the next form |
What thresholds should trigger a review?
Published guides disagree, so choose thresholds for your purpose and record them. As general references, TAO’s documentation treats about 50% correct as generally desirable, above 80% as possibly too easy and below 20% as possibly too hard or confusing, while Penn State’s guide recommends discrimination of at least 0.15.
Neither is an aviation standard or comes from a regulator. TAO’s item statistics documentation also treats discrimination typically above 0.2 as a positive sign. Safety-critical must-know items may be answered correctly by nearly everyone by design, so a high share correct alone is no reason to drop one.
For topics there is no published rule to borrow. These starting rules are this post’s own editorial suggestions, to adjust for your cohort sizes:
- Flag a topic when its mean is at least 15 points below the cohort mean and its interval does not reach the cohort mean.
- Put any topic with fewer than 10 items on a watch list and pool it with the next sitting.
- Send any item with negative discrimination, or discrimination below 0.15, to review before it is used again.
- Confirm a flagged topic with a second source, such as the previous sitting, pre-course results or instructor observation, before changing how it is taught.
Pro tip: Fix The Thresholds Before The Results Arrive. Record your cutoffs in the item review log before a sitting. A cohort that fails badly creates pressure to move the goalposts, and a written rule makes it obvious when that is happening.
Who decides what changes after the analysis?
Your organization decides, under its own approved procedures. Analytics inform the decision. They do not change an exam standard, a pass mark or a question by themselves. Where an authority approves or oversees your exam procedures, check your competent authority before changing items, forms or pass marks, and record the reason.
Keep three questions apart: what your authority requires, what your approved procedures say, and what you may choose within them, such as re-teaching a topic or rewriting a distractor. A standing review step for flagged items, like the item review board process template, keeps those decisions recorded. Also protect small groups: a cut by instructor or base from a class of five can single out individuals, so set a minimum group size for what you show.
What should you do next?
Take your last cohort’s export and build Table 1 for it: tag each item by topic, calculate each topic’s mean with an interval, then list the items under 30% correct or under 0.15 discrimination. Bring flagged items to your reviewers with option counts, not just headline numbers.
SimpliAssess’s features page lists “Analytics & Reporting” and “Tag Management” without detail, and its FAQ says you can create custom reports and charts and export all test scores into a spreadsheet for further analysis. Its published pages do not describe item statistics or topic-level reports, so ask for a demo using your own tagged data. The calculations in this post also work in a spreadsheet on exported scores.
SimpliTrain is a separate training platform from the same company. Its question bank page describes tagging every item with topic, competency, difficulty level and question type, and surfacing difficulty and discrimination index scores per item. That is the vendor’s own description, not something tested for this post.
Frequently asked questions
What is exam analytics?
Exam analytics is the analysis of exam results to guide decisions. It works at three levels: the cohort (scores and pass rates), the topic (results by syllabus area) and the item (how each question performed). Its job is to show where to look, not to explain why results fell.
How many learners do you need before topic results mean anything?
No source reviewed for this post sets a minimum. In the invented example, 24 learners gave 95% intervals about 10 to 15 points either side of each topic mean, so gaps under roughly 10 points were noise. Fewer learners widen the interval.
What is a good item difficulty and discrimination for aviation exams?
No aviation standard was found in the sources reviewed. General guides differ: TAO suggests about 50% correct is desirable and discrimination typically above 0.2, while Penn State recommends at least 0.15. Safety-critical must-know items can legitimately be answered correctly by almost everyone, so set thresholds by purpose.
Can exam analytics prove a topic was taught badly?
No. A low topic mean fits a teaching gap, but it also fits a flawed item, a low starting level or a short topic with few items. Confirm it with a second source, such as the previous sitting, pre-course results or instructor observation, before changing teaching.
Should you change a question as soon as analytics flag it?
Not on the numbers alone. Send the flagged item to review with its option counts, check the key and the wording, and follow your approved procedures for changing items or forms. If an authority oversees your exam procedures, check your competent authority first and record the reason for any change.
