How to create an online exam, step by step
How to create an online exam comes down to a handful of decisions made in a fixed order: purpose, blueprint, items, scoring, timing, settings, pilot and launch. The order matters more than the software, because most problems found after launch trace back to an early decision that was skipped. This guide is for trainers, instructors and L&D teams who need a scored exam that holds up.
Most guides on page one walk through one vendor’s screens. This one is tool-agnostic. It adds a one-page spec sheet, a worked blueprint with illustrative numbers, a pilot plan and a launch checklist. It draws on published guidance from universities, a testing company and the W3C, linked where used.
Key takeaways
- Write a one-page spec covering purpose, stakes, audience, length, pass rule and accommodations before you open any exam tool.
- Build a blueprint that turns each learning objective into a number of items, so coverage is planned and not accidental.
- Set the timer from pilot finish times, and plan extra time for accommodations before launch.
- Pilot with people who resemble real test takers, in the real platform, and revise before the live sitting.
What should you decide before you open any exam tool?
Decide nine things on one page: the exam’s purpose, its stakes, who takes it and on what devices, the objectives it covers, its length, the pass rule, the retake rule, how accommodations work and who owns problems on the day. Writing these first stops platform menus from making the decisions for you. Table 1 shows each row with an illustrative entry.
| Decision | Question to answer | Illustrative entry |
|---|---|---|
| Purpose | What decision will the score support? | Confirm new hires can handle customer data before they get system access |
| Stakes | What rides on a pass or a fail? | Medium: the result gates system access |
| Audience and devices | Who sits it, on what device and connection? | 60 new hires, laptops and some phones, office Wi-Fi |
| Objectives | What must a passer be able to do? | Five objectives, listed in Table 2 |
| Length | How many items and how many minutes? | 40 items, timer set after the pilot |
| Pass rule | What counts as a pass, and who decided? | Fixed pass mark, written before the pilot |
| Retake rule | How many attempts, after what delay? | One retake after a review session |
| Accommodations | How is extra time given? | Per-learner time extension, set before the exam opens |
| Owner | Who fixes problems on exam day? | A named lead and a backup |
Stakes decide how much of the rest you need. A low-stakes knowledge check can skip a formal pilot and standard setting. An exam that gates access, pay or a credential needs every step here, and it is where the access, timer and randomization rules in our exam security settings checklist matter most.
How to create an online exam blueprint from your objectives
A test blueprint is a table that lists each learning objective, how heavily it is weighted and how many items it gets. The University of Tennessee’s teaching center says a blueprint “outlines the learning outcomes that you intend to assess” and helps you plan how material and points are spread across the exam.
How do you turn objectives into item counts?
Multiply each objective’s weight by the total number of items, round to whole numbers, and check that the counts still add up. Table 2 shows an illustrative 40-item onboarding exam. Base the weights on how much each objective matters on the job, not on how much course time it took.
| Objective (illustrative) | Weight | Cognitive level | Items | Points |
|---|---|---|---|---|
| Classify customer data by sensitivity | 20% | Understand | 8 | 8 |
| Apply the retention schedule | 25% | Apply | 10 | 10 |
| Spot and report a data incident | 25% | Apply | 10 | 10 |
| Share data through approved channels | 15% | Understand | 6 | 6 |
| Describe your role in a data request | 15% | Remember | 6 | 6 |
| Total | 100% | – | 40 | 40 |
The cognitive level column stops the exam drifting toward recall: here 20 of the 40 items are Apply items. Recheck the totals whenever you add or drop an item.
How many questions fit, and how long should the exam run?
Let the time available set the length, then confirm it with a pilot. The Tennessee guide says the time students have “will determine how many items the exam should have”, and it warns that work that takes you 30 minutes will most likely take novices longer. Time a colleague who is new to the topic, not the person who wrote the items.
Take illustrative pilot numbers: of ten takers, the fastest finishes the 40 items in 22 minutes, the median in 31 and the slowest in 47. A 45-minute timer would cut off the slowest taker, so lengthen the timer or cut items. Our editorial rule: unless speed is part of what you are measuring, the timer should not decide who passes.
How do you write items that match the blueprint?
Write each item for one blueprint row, with a stem that poses a clear problem on its own and options that a learner who has not mastered the objective would find plausible. In 2002, Haladyna, Downing and Rodriguez validated 31 multiple-choice item-writing guidelines, aimed mainly at classroom assessment.
Cynthia Brame’s multiple-choice guide for the Vanderbilt Center for Teaching boils the advice down to a short checklist:
- Put the problem in the stem and leave out irrelevant material.
- Build distractors from common learner errors.
- Keep options concise, mutually exclusive and similar in length and grammar.
- Avoid “all of the above” and “none of the above”, which let learners use partial knowledge.
- Use negative stems only when the outcome requires them, and emphasize the negative word.
Match the item type to the objective’s verb as well. For objectives that ask learners to apply or analyze, compare question types from multiple choice to case-based before defaulting to multiple choice.
Pro tip: Sort Items Back Into The Blueprint. Give a colleague your items without the tags and ask them to file each under a blueprint row. Any item they file differently is testing something other than you intended, so rewrite or cut it first.
How do you set scoring, the pass mark and feedback?
Fix the scoring rule and the pass mark before anyone sits the exam. Decide whether each item is all-or-nothing or earns partial credit, who marks open responses against which rubric, and how the pass mark was chosen. Changing a pass mark after seeing results makes the outcome hard to defend.
For a high-stakes exam, set the pass mark with a formal standard-setting method, and see the post on cut scores for professional training courses. For a low-stakes check, a stated policy mark is fine if you record why you chose it. Decide too when learners see correct answers, because early release lets them circulate.
How should timing, accommodations and retakes work?
Settle your accommodation rule before launch: who can get extra time, how much, and how it is applied. Your platform should let you give one learner a longer timer without changing the exam for everyone. Extra time must lengthen both the timer and the closing time, or the learner is cut off anyway.
The University of Glasgow’s online exam checklist makes the second point: when a group override is used, add the extra time to the exam timer, not only to the end time. The W3C’s explanation of WCAG 2.2 success criterion 2.2.1 on adjustable timing says a third party may control time limits where learner control would invalidate the outcome, and gives “granting double time on a test” as an example.
Whether WCAG binds your organization is a legal question, so check with counsel. For practical steps, see the post on accessible online exams under WCAG. Put the retake rule in the spec too: a retake on identical items rewards memory.
How to create an online exam in your platform without losing the plan
Build in the order of the spec: create the exam, add sections or tags for each blueprint row, add items with points, then set timing, access, attempts and results. Setting names differ by platform, so Table 3 maps each spec decision to what to look for and what to check before moving on.
| Spec decision | What to look for in the platform | Check before moving on |
|---|---|---|
| Blueprint rows | Sections, tags or random draws by tag | Each row has the item count from Table 2 |
| Points | A score per item | Total points match your pass-mark arithmetic |
| Time | A timer plus a start and close window | The window is longer than the longest timer |
| Pass rule | A pass mark and a result message | The pass mark equals the written rule |
| Accommodations | A per-learner time override | Extra time shows on a test account |
| Attempts | An attempt limit and a delay | The retake rule matches the spec |
| Results | Release rules for scores and answers | Answers stay hidden until the window closes for everyone |
As of November 2026, SimpliAssess’s features page lists item banking, test management, bulk import, tag management and schedule management, and its AI page says the AI can create questions from source material you provide. Its published pages do not describe timers, randomization or per-learner time extensions, so ask any vendor, including us, to show the rows of Table 3 on a test account. Treat AI-drafted items like any other draft: file each under a blueprint row and have someone answer it cold.
How do you pilot an online exam before learners see it?
Run the exam with a small group that resembles your real test takers, in the real platform, under live conditions. Collect finish times, comments on unclear items and basic item statistics, then revise. Questionmark’s piloting guide advises choosing participants “similar to who you will deliver the live assessment to”.
For technology-enhanced items, the ITC and ATP guidelines for technology-based assessment likewise call for pilots on the real delivery platforms under the same conditions as the live sitting.
What can a small pilot tell you, and what can it not?
A small pilot reliably shows unclear wording, settings that misbehave and realistic finish times. It cannot give stable item statistics. Prometric, a testing company, recommends “a minimum of 100 candidate exposures per pretest item” to judge statistical viability.
Most training teams will not reach that number, so treat difficulty figures from a small pilot as flags to reread an item, not verdicts. In the pilot, collect these:
- Finish time for every taker.
- Comments on specific items.
- Items that nearly everyone got right or wrong.
- Settings that behaved differently than you expected.
- Instructions that confused anyone.
Pro tip: Report Two Finish Times. Record the median finish time and the slowest finish time from the pilot. The median shows whether the exam is too long, and the slowest finish shows whether the timer is fair to learners who read carefully or work more slowly.
What belongs in the instructions and launch plan?
Learners should know before they start how long the exam is, how many items it has, what is allowed, whether they can go back, and who to contact if something breaks. A University of Technology Sydney template for timed LMS exams lists those fields, and its page was last updated on 8 October 2026.
The University of Victoria’s online exam checklist adds a contingency plan, plus an optional practice quiz. Answer these five questions in advance:
- How will you send updates during the exam?
- How will learners reach you?
- Who handles technical problems?
- How will you handle suspected integrity issues?
- What will you do after a major outage?
What should you review after the first live sitting?
Review learner comments, the items most learners missed and the items nearly everyone got right, and check the answer key first, because a miskeyed item looks like a hard item. Record each fix as a new version so later results tie back to the wording learners saw.
What should you do next?
Fill in Table 1 for the exam you need, then build the Table 2 blueprint from your own objectives. Then write the items and pilot them with a handful of real learners.
If you want to see these steps in an authoring and delivery platform, you can ask for a SimpliAssess demo and bring your spec sheet. Ask to see the timer, randomization, per-learner extra time and results settings on a test account.
Frequently asked questions
How many questions should an online exam have?
There is no universal number. Length follows the time available and the number of objectives. Start from the minutes learners have, as the University of Tennessee guide advises, then use your blueprint to split items across objectives by weight. A pilot confirms whether novices can finish in the time you set.
Can I create an online exam for free?
Often yes, for a low-stakes knowledge check. The spec sheet and blueprint work in any tool, free or paid. Before you commit, check the tool’s published pages for question types, timers, attempt limits and per-learner time extensions. For an exam that gates access or a credential, those controls matter more than price.
Can AI write my exam questions?
AI can draft items quickly, but a draft is not a finished item. Check each one against its blueprint row, confirm the answer key, test that distractors are plausible, and ask a colleague to answer it without seeing the key. Treat AI drafts the way you would treat a new author’s first submission.
How do I pilot an online exam with a small team?
Choose people who resemble your real learners, run the exam in the live platform and collect finish times and comments. Questionmark advises using as many participants as you can. With a small group, fix wording, timing and settings; do not judge item statistics.
How should I set the pass mark?
Decide it before the exam is delivered and write down how you chose it. For a high-stakes exam, use a formal standard-setting method such as Angoff with qualified reviewers. For a low-stakes check, a stated policy mark is acceptable. Do not change it after seeing results.
