AI question generation for training companies: a review workflow that keeps quality high

An AI question generator is software that uses a large language model to draft assessment items, such as multiple-choice questions, from text or learning objectives you supply. For a training company, the safe way to use one is as a drafting step inside a review workflow: ground it in your course material, have a subject matter expert (SME) check every item, then let pilot data decide what stays.

Three peer-reviewed studies published in 2025 found AI-drafted items that fell short of human-written ones on at least one measure, so review is not optional. This guide gives you seven gates, a reviewer rubric and a sample prompt that makes the model quote your source. The studies come from medical and dental education, not training companies, and the workflow is our recommendation, not a published standard.

Key takeaways

• In three 2025 studies, ChatGPT-drafted items were weaker than human-written ones on difficulty, discrimination or distractors, so every AI-drafted item needs an SME before learners see it.
• A seven-gate workflow covers source grounding, drafting, SME review, a bias and accessibility check, a pilot, item analysis, and a retire-or-revise decision, with a record kept at each gate.
• Asking the model for a verbatim source quote with every item turns “is this true?” into a quick search of your own material.
• Tag each item as AI-drafted in your item bank and log what reviewers did with it, because no published study covers training-company courses.

What is an AI question generator, and how good are the questions it drafts?

An AI question generator drafts test items from source text or learning objectives, usually in seconds. Quality is uneven. In the studies below, AI-drafted items differed from human-written ones in difficulty, discrimination or distractor quality, so each item needs an SME check before any learner sees it.

Tools range from general chatbots to generators built into quiz makers and assessment platforms. What they share is that the model predicts plausible text. It does not know your course, your learners or which wrong answer a real learner would pick.

What does research on AI-generated multiple-choice questions show?

The results are mixed. In three peer-reviewed 2025 studies of health-professions courses, AI-drafted items were weaker than human-written items on at least one measure, and in one study discrimination did not differ significantly. Samples were small and the models are now dated, so read the findings as a reason to review, not a verdict.

The table shows what each study compared, what it found and what limits the findings.

Three cautions apply. First, none of these studies tested training-company courses, and all used small item sets. Second, the models were ChatGPT 3.5, a March 2023 ChatGPT version and ChatGPT-4o, so newer models may behave differently. Third, in the Chauhan study independent reviewers could not reliably tell which items were AI-generated, so you cannot screen out weak items by how they read.

The faculty in that study said ChatGPT items lacked local context and calibration to student level, and that human review was needed. That gap is what the workflow below is built to close. The authors of the Kiyani study go further and advise limiting LLM-generated items to practice use unless they pass standard quality assurance.

What does a review workflow for an AI question generator look like?

A review workflow for an AI question generator is a sequence of gates every drafted item must pass before it is scored: source grounding, drafting, SME review, bias and accessibility check, pilot, item analysis, and a retire-or-revise decision. No item skips a gate, and each gate leaves a record someone can audit later.

The sequence is our recommendation, built from the study limits above and from a 2025 report by researchers at the American Institutes for Research. They used ChatGPT-4 to draft 45 items for each of three constructs and piloted them with about 700 people. They still found redundant items and statistical flags despite careful prompting, and concluded that a hybrid approach, where generative AI augments human work, is the right model.

The table lists the seven gates, who owns each, what an item must show to pass and what to record.

How do you ground the generator in your course material?

Paste one learning objective and the specific source passage into each prompt, and tell the model to use only that text. Then require a verbatim source quote with every item, so a reviewer can check the answer key against your material in seconds instead of trusting the model’s memory.

Grounding narrows what the model can invent but does not remove it. The quote is a checkable claim, not a guarantee: the reviewer searches your source for that exact sentence, and an item whose quote is missing or altered fails. The sample prompt below is a starting point. Replace the bracketed fields, run one objective at a time and store the prompt text with the item record.

You are an item writer for a professional training course.

TASK
Write [5] multiple-choice items that test ONE learning objective, using ONLY
the source text below. Do not use outside knowledge. If the source does not
support a question for this objective, write "NOT SUPPORTED" instead of
inventing one.

LEARNING OBJECTIVE
[Paste one objective that starts with a verb, for example: Apply the lockout
procedure to a given machine fault.]

LEARNERS
[Role, experience level, reading level, workplace or region]

SOURCE (version [v2.1], dated [date])
[Paste the course passage here]

ITEM RULES
- One correct answer and [3] plausible distractors, similar in length and
  grammatical form.
- Each distractor must reflect a specific mistake a learner could make.
  Name the mistake.
- No "all of the above" or "none of the above", no negatively worded stems,
  no clues in the stem.
- Aim at the [apply] level, not recall. Put a short workplace situation in
  the stem.
- Vary the position of the correct answer.

OUTPUT FOR EACH ITEM
1. Stem
2. Options A to [D]
3. Key (letter)
4. Source quote: the exact sentence from the SOURCE that supports the key
5. Distractor notes: the mistake each wrong option represents
6. Cognitive level: recall, apply or analyze

What should an SME check on every AI-drafted item?

The SME should answer the item without seeing the key, confirm the key against the source quote, then score the item on a short rubric. Any failed row means edit or reject. Review AI items as strictly as human-written ones, because well-written weak items read just as smoothly.

The table shows eight checks, the question the reviewer asks for each and the signal that the item fails.

For scenario items, add the checks in our guide to scenario based multiple choice questions.

Pro tip: Hide the Key From the Reviewer. Send SMEs the stem and options without the answer and ask them to pick one before they see the key. A mismatch is the fastest way to catch a mis-keyed item or a second defensible answer, and it stops reviewers from nodding along to a confident rationale.

How do you check AI-drafted items for bias and accessibility?

Give each item to a second reviewer who did not do the content check. They read for fairness and access: scenario details that favor one group, idioms, extra reading load, and anything that would trip up a second-language learner or someone using assistive technology. Fix the item or reject it.

Ask the reviewer to check four things:

• Names, roles and settings do not stereotype or assume one background.
• Idioms and slang are removed unless the objective requires them.
• The reading level fits the learners, and the stem has no extra words that the skill does not need.
• Any image, table or chart in the item has a text alternative, and the question can be answered without color alone.

When should pilot data and item analysis decide whether an item stays?

After enough learners have answered the item in a low-stakes pilot, item analysis decides. Flag items with poor discrimination, a distractor nobody chooses, or difficulty that does not suit the purpose, then send them back to an SME. Statistics trigger review. They should not retire an item automatically.

A formative pilot is a natural proving ground for AI-drafted items, given the Kiyani authors’ advice to keep unreviewed LLM items to practice use. Keep the cohort size in mind. Our item analysis guide for trainers shows that with 20 learners a single changed answer moves the discrimination index by 0.17, so small pilots are for spotting problems, not proving quality.

Store the outcome in your bank. The fields, versioning rules and retirement checklist in our guide to building an item bank for a training company apply unchanged to AI-drafted items.

Pro tip: Tag Origin and Reviewer Action. Add two fields to every item: whether it was AI-drafted, and what the SME did (accepted unchanged, edited, rejected). After a few cycles you can compare AI-drafted and human-drafted items in your own bank, which gives you the training-company evidence the published studies lack.

Where does an AI question generator inside an assessment platform fit?

A generator built into an assessment platform replaces only the drafting step, which is gate 2. Source grounding, SME review, bias checks, piloting and item analysis still need people and records, whether the draft came from a chatbot or a built-in tool. Choose tools that let you keep those records with the item.

SimpliAssess publishes AI-driven test authoring. As of October 2026, its AI page says questions can be generated from existing tests, articles, eBooks and websites, and its launch announcement describes choosing a subject and test objectives before the AI generates questions. Neither page describes a human review step, so gates 3 to 7 are yours to run, whichever tool you use. The AI page also mentions automated psychometric checks. Treat any built-in check as one more signal and confirm what it measures before you rely on it.

Whatever tool you evaluate, ask four things:

• Can I attach the source passage and objective to each generated item?
• Does it show which model produced the draft, and can I keep the prompt?
• Can I mark items as draft, approved or retired, and keep versions?
• Where does my course content go, and who can see it?

What should you do next?

Pick one course and one learning objective this week. Run the sample prompt on a single source passage and push five drafted items through all seven gates. Count how many were accepted unchanged, edited and rejected, and write down why each rejected item failed.

Repeat for two or three more objectives and keep the log. It shows whether the generator saves time once review is counted, and which rubric rows fail most often so you can tighten the prompt.

If you need somewhere to keep approved items, SimpliTrain’s question bank page describes tagging, versioning and retiring questions, and tracking difficulty and discrimination for each one. As of October 2026 the page covers storing and reviewing items, not drafting them.

Frequently asked questions

Can AI write good exam questions?

Sometimes, but not reliably without review. In three 2025 studies of health-professions courses, AI-drafted items were weaker than human-written ones on at least one measure, such as difficulty, discrimination or distractor quality, though discrimination did not differ significantly in one study. Treat AI output as a first draft that an expert checks and learners then pilot.

Do I need a subject matter expert to review AI-generated questions?

Yes. An SME is the only gate that can confirm the key is correct and the scenario is realistic for your course. Faculty in the Chauhan study said ChatGPT items lacked local context and student-level calibration and needed human review, and the AIR researchers call human expertise essential in prompting, item review and psychometric evaluation.

How do I stop an AI question generator from making up facts?

You cannot stop it entirely, but you can limit and detect it. Paste only the relevant source passage, tell the model to use only that text, and require a verbatim source quote with each item. The reviewer then searches your source for that quote. A missing or altered quote means the item fails.

Is it safe to paste course material into an AI tool?

It depends on the tool’s data terms, so check them before you paste anything. The AIR researchers advise using secure platforms that limit access to generated content and setting clear team policies on AI-generated items. Ask any vendor where your content is stored, who can see it and whether it is used for training.

Should I tell learners that questions were AI-generated?

The AIR researchers recommend notifying candidates when AI was involved in creating an assessment and explaining how items were generated, validated and checked for fairness. Whether you are required to depends on your jurisdiction and use case, so ask your legal or compliance contact.

Can AI-generated questions be used in certification or summative exams?

Only after the same gates as any other item. The authors of the Kiyani study advise limiting LLM-generated items to practice use unless they undergo standard quality assurance. For high-stakes exams, add pilot data, item analysis and a documented SME decision for every item, and have a psychometrician review the process.