Item review boards for aviation exams: a process template

An item review board is a standing group that decides, one exam question at a time, whether an item goes live, is revised or is retired. Item review is how an aviation training organization shows that it controls the questions it sets. Below is a template: roles, stages, a review form, decision codes and a decision log.

It is for heads of training, exam officers and quality managers at Part-147 organisations, approved training organisations and AMT schools that write their own questions. Writing items is a separate job, covered in our guide to writing aviation MCQs that test understanding.

Regulatory facts are as of November 2026. This is not legal or airworthiness advice, and no regulator approves a review board template or a software tool. Check your competent authority before you change how your exams are controlled.

Key takeaways

• No rule we read requires an item review board by name. A board is how you produce evidence for rules on exam standards, question security and authority sampling of your question data bank.
• Keep writers and reviewers apart. Two independent technical reviewers answer each item blind before the meeting, and authors never vote on their own items.
• Six decision codes and a decision log with a reason for every decision make any live item traceable.
• Triggers should send live items back: a manual or regulation change, an upheld challenge, an item-analysis flag or a security event.

Does aviation regulation require an item review board?

No rule we read requires a board by name. Aviation rules set the exam format, require questions to be kept secure and let the competent authority check your question data bank. An item review board is a design choice that produces the evidence those rules point toward.

The table separates what each source says from what a board adds. Part-66 and Part-147 wording was read in the UK copy of Part-66 Appendix II and the UK copy of Annex IV on legislation.gov.uk. Confirm the EU wording your authority applies.

Two limits apply. The EASA FAQ dates from 2015 and Part-147 has changed since. The UK CAA page describes the CAA’s own exams, not a duty on you.

Who should sit on an aviation item review board?

A workable board has five roles: a chair, two or more independent technical reviewers, an exam officer who acts as secretary, a quality representative and an assessment lead who brings data. The author attends to explain the item but does not vote on it. These roles are a template, not a regulatory list.

The Standards for Educational and Psychological Testing (AERA, APA and NCME, 2014) do not name a board. Standard 1.9 says procedures for selecting expert judges “should be fully described” and should “indicate whether participants reached their decisions independently” (quoted). The roles below put that into practice.

What stages should item review follow?

Item review runs in six stages: intake, blind pre-read, board meeting, decision and change, release check and post-use review. Each stage changes the item’s status, so only approved items reach the live pool. Our guide to an aviation question bank structured by ATA chapter covers the status field.

Step 1: Intake. The author submits the item with its key, rationale, source document and revision, syllabus reference and purpose. Status moves from draft to ready for review.

Step 2: Blind pre-read. Each technical reviewer answers the item without the key before the meeting and records comments. Status moves to under review.

Step 3: Board meeting. The board discusses items where any blind answer differs from the key, fewest matches first. Items where every reviewer matched go on a consent list approved in one vote. That is a design choice, not a rule.

Step 4: Decision and change. The secretary records a decision code and a reason. A change to the key, stem or options creates a new version that is reviewed again. A typo fix does not.

Step 5: Release check. The secretary confirms the purpose is exam, sets status to live and removes the old version from the exam pool.

Step 6: Post-use review. Challenges, source changes and item statistics can return a live item to the board.

How does the blind pre-read work in practice?

The secretary compares each blind answer with the key, and any mismatch puts the item on the agenda. The table shows six invented items and three reviewers.

AV-0206 goes first, because one match out of three often means a wrong key. AV-0204 reads: “Which of these is a unit of pressure? A. Pascal, B. Newton per square meter, C. Joule.” Reviewer 2 chose B, and B is also correct, so the board sends it back for rewriting. The item is invented basic physics, not from any exam. If reviewers often miss the key, they may need calibrating, as our post on instructor and examiner standardization explains.

Pro tip: Freeze Items Before The Pre-Read. Set every agenda item to under review when the pack goes out. Otherwise an item can be drawn into a live exam while the board is still deciding whether it is sound.

What should the item review form check?

The form should record nine checks as pass, fail or not applicable, plus item ID, version, reviewer, date and blind answer. Each check is a question answerable from the item and its stored source. Add any check your authority or manuals require.

Checks 4, 5, 7 and 8 follow rules written for Part-66 basic multiple-choice exams. For other exams, treat them as good practice. For essays, add a check that a model answer and Key Points exist, as Appendix II, 1.5 and 1.6 describe.

What decisions can the board make, and what sends an item back?

Use six decision codes: approve, approve with edit, revise and re-review, reject, suspend and retire. Each sets the item’s status and says whether the board must see it again. Triggers, not only a calendar, send live items back.

The codes are our design, not a published standard. Five events should reopen a live item:

• Source change. A manual revision or regulation amendment touches the item’s stored source. Suspend it, then review it.
• Challenge. A candidate or instructor disputes the key.
• Statistics flag. Item analysis shows a negative discrimination or an unused distractor. Statistics flag items. People decide.
• Security event. The item may have been exposed, so suspend or retire it.
• Scheduled cycle. The rules we read set no period. An Alpine Testing Solutions presentation reported one company reviewing items every 18 to 24 months, which is one organization’s practice.

Pro tip: Review The Neighbors Of An Upheld Challenge. When a challenge succeeds, also review the other items by the same author or from the same chapter. One bad key is often a sample, not an accident.

What does a decision log look like?

A decision log holds one row per decision: meeting, item and version, code, reason with evidence, and next action with owner. It answers who decided, on what evidence and what changed. The rows are invented.

What should you do next?

Pilot the process on one module. Pick 20 items, appoint two reviewers who did not write them, run the blind pre-read, hold one meeting and fill in the log. The number of items the board changes shows whether the effort pays.

Before you write a charter, ask your competent authority what it expects to see when it samples your question data bank, then adjust the form and log to match.

If you are comparing platforms, ask each vendor to show item status, reviewer assignment, comments and version history on a test item. SimpliAssess’s features page lists Item Banking, Tag Management and Multi-User Collaboration as headings without describing them, and does not mention review or version history, so confirm those in a demo. SimpliTrain’s question bank page describes tagging, version control and flagging weak questions for review, but not approval steps, statuses or roles. A feature list does not show that any tool meets an authority’s rules.

Frequently asked questions

Do aviation regulations require an item review board?

No rule we read names one. Part-66 Appendix II sets how basic exam questions are written, point 147.A.135(a) requires examination staff to keep questions secure, and EASA FAQ 19074 says the authority must sample check your question data bank. A board is one way to show control. Ask your competent authority what it expects to see.

How often should an aviation question bank go through item review?

The rules we read set no review period. The UK CAA describes routine review of all presented questions for its own exams, and one practitioner presentation reported a company reviewing items every 18 to 24 months. Set your own cycle, then add triggers: a manual revision, an upheld challenge, an item-analysis flag or a security event.

Can an item’s author sit on the board that reviews it?

The author should present the item but not vote on it or act as a blind reviewer. Standard 1.9 of the Standards for Educational and Psychological Testing asks you to report whether expert judgments were made independently, and an author is not independent. This is a template rule, not an aviation regulation.

What should you do when a candidate challenges an exam question?

Log the challenge, suspend the item if the complaint is credible and have two technical reviewers answer it blind against the stored source. The UK CAA says every comment made during an exam is reviewed and results may be amended where genuine issues are identified. Check your own appeal rules and your competent authority’s expectations.

Does using this template make an exam compliant?

No. The template is our own design, built from the sources named above, and no regulator approves a review process template or an exam tool. Compliance depends on your approved procedures and your competent authority’s rules. Use it as a draft, then have your quality manager and authority check it.