Competency Checklist for Hands-On Training: How to Build an Observation Checklist
A competency checklist is a list of observable actions that an assessor ticks off while watching a learner perform a practical task, ending in a competent or not-yet-competent decision. This guide shows trainers and training providers how to build one for hands-on, instructor-led sessions, with a template, a worked example and a way to test whether two assessors score the same performance the same way.
It is written for people who sign off physical or procedural skills in a classroom, workshop or yard. Many of the competency checklist templates you will find online are spreadsheets for tracking employee skills across a job role. This post covers the narrower job of judging a live performance, where two assessors can watch the same learner and disagree.
Key takeaways
- A competency checklist works when every criterion describes something an assessor can see or measure, so a second assessor would reach the same verdict.
- Tag each criterion as critical, required or supporting, and decide before the first learner that one missed critical item means not yet competent, whatever the rest of the score.
- Record evidence, not impressions: what the assessor saw, how many times, under what conditions, plus the assessor’s name and the date.
- Checklists are not automatically more reliable than expert judgment, so add one overall judgment line and measure agreement between assessors with percent agreement and Cohen’s kappa.
- Calibrate before the course: have two assessors score the same performances independently, compare line by line and rewrite any criterion they read differently.
What is a competency checklist, and how is it different from a rubric?
A competency checklist is a set of yes-or-no criteria describing observable actions in a practical task, scored by an assessor while or just after the learner performs it. A rubric instead describes levels of quality for each criterion. Use a checklist when an action is either done correctly or not done.
The Australian National University’s teaching guide draws the same line: a rubric needs clear descriptions of performance levels, while a checklist needs yes or no decisions on specific criteria. A practical sign-off often comes down to a yes-or-no question: the learner can safely do the task, or cannot yet.
A checklist also does not replace the written part of a course. Well-built scenario-based multiple-choice questions test whether a learner chooses the right action, while the checklist tests whether they carry it out.
How do you write checklist criteria that two assessors will score the same way?
Write each criterion as one action an assessor can see or measure, with the standard inside the line. Australia’s VET regulator ASQA says evidence criteria “should not be open to interpretation” and must give enough clarity for judgments to be consistent across assessors and points in time. If two assessors could read a line two ways, rewrite it.
That wording comes from ASQA’s Guide to assessment tools (version 1.1, September 2021), written for Australia’s Standards for RTOs 2015. US providers are not bound by it, but it explains observation-checklist design in plain terms.
Three habits make criteria observable:
- Use one action per row. Split “inspects and tags the ladder” into two rows.
- Put the measurable standard in the line. Replace “sets a safe angle” with “base is about one-quarter of the working length from the wall, measured with a tape”.
- Choose verbs an assessor can see, such as measures, states, tags or positions. Avoid “understands”, “is careful” and “shows confidence”.
What should a competency checklist template include?
A usable template has eight parts: the task and conditions, observable criteria, a tier for each criterion, a rating code, an evidence note, a decision rule, an overall judgment line and assessor sign-off. ASQA’s guide adds that a checklist should say how to treat items not observed and let assessors record the number of instances and the observation period.
The table below lists each part, what to put in it and why it matters.
| Template part | What to put there | Why it matters |
|---|---|---|
| Task and conditions | The task, equipment, time limit, supervision level and what the assessor may prompt | Every learner is judged under the same conditions |
| Criteria | One observable action per row; this post’s rule of thumb is 6 to 12 per task | Two assessors can score the same row the same way |
| Tier | Critical, required or supporting | Tells the assessor which misses block sign-off |
| Rating code | Met, Not met or N/O (not observed) | Stops assessors from guessing when they did not see an action |
| Evidence note | What was seen, how many times, over what period; required for every Not met | Lets a reviewer who was not there follow the judgment |
| Decision rule | Written before first use, printed on the form | Every assessor applies the same pass logic |
| Overall judgment | One line: could this learner do the task unsupervised? Yes or Not yet, with a reason | Captures what separate criteria miss |
| Sign-off | Assessor name, date and signature; learner acknowledgment; feedback given; reassessment date | Creates the audit trail and a path to retry |
The block below is a copyable footer for the form. The thresholds are one workable design, not a standard, so change them with your subject-matter expert.
DECISION RULE (printed on every form) Competent = every Critical row Met, unprompted, AND no more than one Required row Not met. Not yet competent = any Critical row Not met or N/O, or two or more Required rows Not met. Supporting rows: coaching comments only, never part of the decision. - - - OVERALL JUDGMENT: Could this learner do the task unsupervised? Yes / Not yet Reason: ______________________________________________ - - - ASSESSOR: name ____________ date ________ signature ____________ LEARNER: feedback given Yes / No acknowledged ____________ REASSESSMENT DATE (if Not yet): ____________
What does a worked competency checklist look like for one hands-on task?
The example below is an observation checklist for one task: setting up and climbing a portable extension ladder in a working-at-heights course. Each row is an observable action with a tier. The critical rows follow OSHA’s construction ladder rules, 29 CFR 1926.1053, as published on eCFR and current as of October 15, 2026.
The table shows nine rows, the tier of each and where the standard comes from.
| # | Observable criterion | Tier | Basis |
|---|---|---|---|
| 1 | Inspects rails, rungs and feet of the ladder and states what was checked | Critical | Inspection for visible defects, 1926.1053(b)(15) |
| 2 | Rejects the ladder with the planted defect and tags it “Do Not Use” | Critical | Defective ladders marked or tagged and withdrawn, (b)(16) |
| 3 | Places the feet on a stable, level surface | Critical | (b)(6) |
| 4 | Sets the base about one-quarter of the working length from the wall; assessor checks with a tape | Critical | (b)(5)(i) |
| 5 | Side rails extend at least 3 feet above the landing, or the top is secured and a grasping device is provided; assessor measures | Critical | (b)(1) |
| 6 | Faces the ladder and keeps at least one hand on it going up and down | Critical | (b)(20) and (b)(21) |
| 7 | Names two hazards near the setup before positioning, such as doors or overhead lines | Required | Provider’s own standard |
| 8 | Carries tools so one hand stays free to grasp the ladder | Required | Provider’s own standard |
| 9 | Lowers and stores the ladder without leaving a trip hazard | Supporting | Provider’s own standard |
These rows are illustrations. Check the current text of 29 CFR 1926.1053 and your own governing rule before use, and have a subject-matter expert approve every safety-critical row.
An illustrative evidence note for row 4 would read: “Not met. Base measured 5 ft 6 in from the wall for a 16 ft working length, about 4 ft expected. Learner moved it after a prompt.”
Pro tip: Plant the Defect. Give learners two ladders, one with a planted defect such as a bent rung, kept on the floor and never climbed. Without it, the assessor can only record that the learner said the ladder was fine, which is a claim rather than evidence.
How do you decide whether a learner is competent or not yet competent?
Set the decision rule before the first learner is assessed. In the design used here, a learner is competent only when every critical row is met and at most one required row is missed, with feedback recorded. One missed critical row means not yet competent, whatever else was met. Print the rule on the form.
This rule is a design choice, not a regulation. Supporting items feed coaching comments instead of the decision. Real skill sheets use the same idea. The National Registry’s EMT psychomotor skill sheet for bleeding control and shock management (a copy hosted by the State of Minnesota, form e213/10-16) scores seven steps at one point each and lists critical criteria separately, such as failure to control hemorrhage in a timely manner. It requires a written rationale whenever one is checked.
A percentage pass mark does not suit this job. It fits written tests, as explained in the guide to setting cut scores for professional training courses. On a hands-on task, a learner who meets eight of nine rows but skips the base measurement has not shown the skill.
Record every attempt. When a learner misses a critical row, give feedback tied to that row, then reassess on a new occasion and keep both records. ASQA’s guide says tools should require the assessor to give feedback both where competency has been shown and where further evidence is required.
Is a checklist enough, or should you add an overall judgment line?
A checklist alone may not be enough. A 2015 meta-analysis of simulation-based assessment found similar inter-rater reliability for checklists (0.81) and global ratings (0.78). Global ratings scored higher on inter-item (0.92 against 0.66) and inter-station reliability (0.80 against 0.69). This post suggests adding one overall judgment line.
The review, by Ilgen, Ma, Hatala and Cook in Medical Education (2015), pooled 45 studies of health professionals. It also found that global ratings can be used across tasks, while each task needs its own checklist. The checklist inter-item figure rests on four studies and is imprecise.
The studies mostly involved physicians in training, not workplace trainers, so read them as direction, not proof. The authors do not propose a combined form, so the overall judgment line is this post’s own design suggestion.
How do you keep assessors consistent with calibration and double-marking?
Calibration means assessors score the same performances independently, compare results and agree what each criterion looks like. Double-marking means a second assessor also scores some learners. The 2014 Standards for Educational and Psychological Testing say that “if raters are used to assign scores to responses, the variability in scores over qualified raters is a source of error.”
That sentence opens the reliability chapter of the Standards for Educational and Psychological Testing (AERA, APA and NCME, 2014). A calibration routine for a small provider can run in five steps.
Step 1: Agree the anchors. Before the course, assessors write what a Met and a Not met look like for each row, using photos, short clips or a demonstration.
Step 2: Score independently. Two assessors watch the same three to five performances, live or recorded, and score without talking. ASQA’s guide says learners should be told if an observation is recorded by audio or video.
Step 3: Compare line by line. Discuss each disagreement and fix the wording, not the people. Rows where assessors split are usually the ambiguous ones.
Step 4: Double-mark a sample. Have a second assessor score the whole first cohort and a sample afterward. The sources used here give no fixed sample size, so set it by risk, with more for critical rows.
Step 5: Repeat when anything changes. Recalibrate when criteria, equipment or assessors change.
Pro tip: Calibrate on the Borderline. Agreement on a clear pass or a clear fail tells you little, because anyone can score those. Include at least one borderline recording, where one missed critical row decides the outcome, and make assessors justify the call out loud.
How do you measure agreement between two assessors?
Count how often two assessors give the same rating, which is percent agreement, then adjust for agreement expected by chance with Cohen’s kappa. Percent agreement alone can overstate consistency. In McHugh’s 2012 review, any kappa below 0.60 indicates inadequate agreement among raters, a stricter line than Cohen’s original scale.
McHugh’s paper, Interrater reliability: the kappa statistic (Biochemia Medica, 2012), says percent agreement “may overestimate the true agreement among raters”. The table below shows invented results for 40 Met or Not met judgments scored by two assessors.
| Counts of judgments | Assessor B: Met | Assessor B: Not met | Row total |
|---|---|---|---|
| Assessor A: Met | 28 | 4 | 32 |
| Assessor A: Not met | 2 | 6 | 8 |
| Column total | 30 | 10 | 40 |
Observed agreement is (28 + 6) / 40 = 0.85. Assessor A said Met 80% of the time and assessor B 75%, so chance agreement is 0.80 × 0.75 + 0.20 × 0.25 = 0.65. Kappa is (0.85 − 0.65) / (1 − 0.65) = 0.57.
An 85% match sounds comfortable, yet a kappa of 0.57 falls under McHugh’s 0.60 line and reads as only moderate on Cohen’s scale. Review the six rows where the assessors split. With 40 judgments the estimate is rough, and McHugh warns that a kappa can have a confidence interval wide enough to include anything from good to poor agreement.
What evidence should you keep with each signed-off competency checklist?
Keep enough that someone who was not there can understand the judgment: the completed checklist, evidence notes, assessor identity, date and conditions. ASQA’s guide frames good evidence as valid, sufficient, authentic and current. Where a regulator requires certification of practical evaluation, as OSHA does for powered industrial truck operators in 29 CFR 1910.178(l)(6), the record must include the operator’s name, the training and evaluation dates, and who performed them.
On sufficiency, ASQA’s guide says evidence should be gathered “over time and in different ways” and gives an example in which a task must be performed on at least two occasions. If you award CEUs, the post on IACET assessment evidence notes that observed tasks can count when they measure the outcomes and you can document the result.
What should you do next?
Pick the practical task you sign off most often. Write six to ten criteria, tag the tiers, add the decision rule and overall judgment line, then pilot the checklist on three learners with two assessors. Calculate percent agreement and kappa on the pilot, and rewrite every row where the assessors split.
If your course also assesses soft skills such as coaching or de-escalation, where behavior is harder to see, the post on case-based assessment for soft skills training covers a better-suited method.
On tooling, SimpliAssess’s published feature list names item banking, test management and analytics and reporting, and its published pages do not describe observation checklists as of October 2026. Keep the checklist as a controlled form or sheet and use an assessment platform for the knowledge test.
Frequently asked questions
What is the difference between a competency checklist and a skills matrix?
A competency checklist records evidence that one learner performed one task to a stated standard, judged by an assessor. A skills matrix is a grid showing which people on a team hold which skills. The matrix summarizes results and can be fed by checklists, but it does not show how any individual judgment was made.
How many criteria should a competency checklist have?
There is no fixed number. As a rule of thumb, 6 to 12 criteria per task works. Fewer usually means actions are bundled into vague rows. More makes the form hard to score live. If a task needs more, split it into separate checklists for setup, performance and finish, each with its own sign-off and decision rule.
What is a critical fail item on a checklist?
A critical fail item is a criterion that must be met for sign-off, so one miss means not yet competent whatever the rest of the score. Reserve it for safety, legal or core-purpose actions. The National Registry’s EMT bleeding control skill sheet lists critical criteria separately from its point-scored steps and requires a written reason when one is checked.
How many assessors do you need to observe each learner?
One qualified assessor can sign off a learner. A second assessor is for calibration and sampling, not every learner. ASQA’s guide says that even with only one assessor you must still develop evidence criteria to ensure consistency in that assessor’s judgments, so write the criteria and decision rule before assessing anyone.
What is a good kappa for assessor agreement?
McHugh’s 2012 review says any kappa below 0.60 indicates inadequate agreement among raters. Cohen’s original scale is more lenient, calling 0.61 to 0.80 substantial and 0.81 to 1.00 almost perfect. Treat 0.60 as the floor for a pilot, then investigate the rows where assessors split rather than chasing a single number.
Should a competency checklist replace a written test?
No. A checklist shows that a learner can perform a task, while a written test shows that they know why and when. ASQA’s guide lists direct observation, product-based methods, questioning and third-party evidence, and says using a range of methods helps produce valid decisions. Pair the two: knowledge questions before or after, and an observed task for the skill itself.
