Item analysis for trainers: difficulty, discrimination and distractors explained
Item analysis is the check you run after a test to see which questions did their job. For each question it looks at three things: difficulty (the share of learners who got it right), discrimination (whether stronger learners got it right more often than weaker ones) and distractors (which wrong options learners picked, and who picked them).
Several of the guides reviewed for this post give cutoffs without saying how much data you need. This one works through a small class of 20 learners and 5 questions, shows every calculation, and ends each item with a keep, revise or drop decision. It also says plainly what 20 learners can and cannot tell you.
The learners and answers below are invented for illustration, not real results. Every number was computed in Python from the data set printed in the post.
Key takeaways
- Difficulty is the proportion correct, so a higher value means an easier item. Discrimination compares upper and lower scorers, and distractor counts show which wrong answers worked.
- The cutoffs you will see are rules of thumb from different sources, and they disagree. Judge each item against its purpose, not against one number.
- In the worked example one item’s answer key looks wrong, one item has a distractor nobody chose, and one item is too easy to separate learners.
- With 20 learners, one changed answer moves the discrimination index by 0.17. Use the statistics to decide which items an expert reviews first, not to retire items on their own.
What does item analysis tell you after a training test?
Item analysis tells you whether each question was too easy, too hard, unfair or misleading, using the answers learners already gave. It cannot tell you whether an item measures the right objective. That stays a content judgment. Run it on any test you plan to reuse.
If you keep questions in an item bank, the results belong in each record next to its status and version, as described in our guide to building an item bank for a training company.
How do you calculate item difficulty, and what counts as a good value?
Item difficulty, written p, is the number of learners who answered an item correctly divided by the number who answered it. If 17 of 20 learners get an item right, p is 0.85. The name is backward: a higher p means an easier item.
Cutoffs vary by source. A University of Florida teaching module says difficulty should range between about 0.25 and 0.75 for most items. The University of Arizona College of Medicine, Phoenix accepts 0.80 to 1.00 for mastery items and 0.30 to 0.70 for items meant to separate learners. Testing company Castle Worldwide recommends 0.30 to 0.92 for its item reports.
These are different rules of thumb for different purposes. A must-know item can sit at 0.95 and be doing its job. An item meant to rank learners cannot. A very easy item in a pre-test also adds little, as covered in the guide to pre and post course assessments for instructor-led training.
What data set does the worked example use?
The data set is 20 invented learners answering 5 four-option multiple-choice items. Learners are ranked by total score, and the top 6 and bottom 6 form the upper and lower groups. Six learners is 30 percent of the class.
That sits between the 27 percent split in the Arizona guide and the one-third split the Florida module lists. The upper group is the six learners who scored 4 or 5 (L03, L04, L07, L11, L14, L20). The lower group is the six who scored 1 or 2 (L02, L05, L06, L08, L10, L16). The other eight scored 3.
No tie decides who sits in a group here, because the sixth and seventh learners differ in total score, and so do the fourteenth and fifteenth. Real classes are less tidy, so set your tie rule before you look at any item.
Illustrative data: answers by learner (key: B, D, A, B, C) Learner Q1 Q2 Q3 Q4 Q5 Total L01 B D C D C 3 L02 B B A D A 2 L03 B D B B C 4 L04 B D C B C 4 L05 A C A A C 2 L06 B A B A C 2 L07 B D C B C 4 L08 A B A D C 2 L09 B A A D C 3 L10 B A B D A 1 L11 B D A B C 5 L12 B D C A C 3 L13 B D C A C 3 L14 B D A B C 5 L15 B A A A C 3 L16 A C D A C 1 L17 B A C B C 3 L18 B D B A C 3 L19 B A A A C 3 L20 B D C B C 4
How do you calculate the discrimination index?
The discrimination index, D, is the proportion of the upper group who answered an item correctly minus the proportion of the lower group who did. It runs from -1 to 1. A positive D means stronger learners got the item right more often, which is what you want.
The Florida module names 0.20 as the lowest recommended value and advises you to reject, revise or rewrite items with a negative one. The Arizona guide calls 0.30 or higher highly discriminating and flags an item when fewer than 60 percent of the top group answer it correctly, or when the upper group does worse than the lower group.
The point-biserial does the same job using all learners instead of only the extremes. It is the Pearson correlation between the 0 or 1 item score and the total score on the other items, as the University of Washington’s scoring office describes its report. That office labels values above .30 good, .10 to .30 fair and below .10 poor, and says values seldom exceed .50. Castle Worldwide recommends a positive value of at least 0.15 for its own reports.
| Item | Key | Difficulty (p) | Upper correct (of 6) | Lower correct (of 6) | Discrimination (D) | Point-biserial |
|---|---|---|---|---|---|---|
| Item 1 | B | 0.85 | 6 | 3 | 0.50 | 0.21 |
| Item 2 | D | 0.50 | 6 | 0 | 1.00 | 0.37 |
| Item 3 | A | 0.40 | 2 | 3 | -0.17 | -0.34 |
| Item 4 | B | 0.35 | 6 | 0 | 1.00 | 0.46 |
| Item 5 | C | 0.90 | 6 | 4 | 0.33 | 0.20 |
Point-biserial here is each item against the total on the other four items. Correlating an item with a total that includes itself inflates the value on a short test.
How do you read distractors?
Count how many learners chose each option, and note which group they came from. A working distractor attracts some learners, mostly from the lower group. One nobody picks adds nothing. One that attracts your strongest learners may mean the key is wrong or the item is ambiguous.
Two rules of thumb apply to rarely chosen options. Castle Worldwide says a distractor chosen by under 3 percent of candidates might not be plausible. A 2022 Frontiers in Medicine study describes a traditional rule of under 5 percent. The mean score on the other items of the learners who chose each option adds a second check, as the Washington report shows: stronger learners should be choosing the key.
| Item and option | Chosen by | Upper group | Lower group | Mean score on other items |
|---|---|---|---|---|
| Item 3, A (key) | 8 (40%) | 2 | 3 | 2.12 |
| Item 3, B | 4 (20%) | 1 | 2 | 2.50 |
| Item 3, C | 7 (35%) | 3 | 0 | 3.43 |
| Item 3, D | 1 (5%) | 0 | 1 | 1.00 |
| Item 4, A | 8 (40%) | 0 | 3 | 2.50 |
| Item 4, B (key) | 7 (35%) | 6 | 0 | 3.14 |
| Item 4, C | 0 (0%) | 0 | 0 | none |
| Item 4, D | 5 (25%) | 0 | 3 | 2.20 |
Item 3 is the warning. Option C drew three of the six upper learners and none of the lower ones, and its choosers average 3.43 on the other items against 2.12 for the key. Item 4 is healthier but has a dead option: 13 learners answered it wrongly and none chose C.
Items 1 and 5 also have unused options, but only three learners missed Item 1 and two missed Item 5, so that is expected. Build replacements from real learner errors, as in our guide to scenario-based multiple-choice questions for professional training.
Should you keep, revise or drop each item?
Check the key first, then compare difficulty with the item’s purpose, then discrimination, then distractors, and let a subject matter expert make the final call. The statistics flag items for review. This sequence is a framework built from the thresholds above, not a published standard.
- Check the key. A negative D or point-biserial, or a wrong option whose choosers outscore the key’s choosers, points to a mis-keyed item or a second defensible answer.
- Compare p with the item’s purpose. Mastery items can be easy, while separators should sit nearer the middle.
- Check discrimination. Look for clearly positive values, since very easy and very hard items tend to score low without being faulty.
- Scan the distractors, and note whether enough learners answered wrongly for an unused option to mean anything.
- Decide with an expert. The Florida module calls its guidelines not hard and fast rules.
| Item | What the numbers show | Decision | Why |
|---|---|---|---|
| Item 1 | p 0.85, D 0.50, point-biserial 0.21 | Keep | Easy, but inside the Arizona mastery range and positively discriminating. |
| Item 2 | p 0.50, D 1.00, point-biserial 0.37 | Keep | Mid difficulty, and every wrong option drew only lower-group learners. |
| Item 3 | p 0.40, D -0.17, point-biserial -0.34, option C chosen most by strong learners | Check key, then revise | Possible second defensible answer or wrong key. Have an expert answer it blind. |
| Item 4 | p 0.35, D 1.00, point-biserial 0.46, option C chosen by no one | Keep, replace option C | It separates learners well. One distractor does no work. |
| Item 5 | p 0.90, D 0.33, point-biserial 0.20, two options unused | Drop unless it is a must-pass check | Almost everyone answers it correctly, so it says little about differences. |
Pro tip: Label Each Item’s Purpose First. Before the test, tag each item in the bank as a must-pass mastery check or a separator. Then a p of 0.90 on a mastery item reads as success, and the same value on a separator reads as a candidate for replacement.
What can item analysis tell you with only 20 learners?
Less than a tidy table suggests. With 20 learners, each answer moves p by 0.05 and D by 0.17, and a five-item total is a crude yardstick. Use the statistics to decide what a subject matter expert reviews first, not to retire items on their own.
- One answer moves D by 0.17. Item 3’s D of -0.17 is exactly one answer away from zero. The flag stands on the pattern instead: the negative point-biserial and where option C’s choosers came from.
- p is imprecise. A standard 95 percent Wilson interval for 10 correct out of 20 runs from about 0.30 to 0.70.
- D of 1.00 is flattered. The groups are defined by the same five items, so good items separate them almost by construction. The point-biserials of 0.37 and 0.46 are the more modest reading.
- Reliability is low. KR-20 for this test is about 0.3. The Arizona guide says KR-20 above 0.50 is acceptable with at least 30 exam takers and 50 items. This test has neither.
The Washington office calls item analysis data tentative and influenced by the type and number of students tested. The Frontiers authors add that changes should almost always be verified by subject matter experts and stakeholders.
Pro tip: Pool Counts Across Cohorts. Store each option’s answer counts per cohort in the item’s bank record, and add them up across cohorts only while the item and its options are unchanged. Three cohorts of 20 give you 60 answers before you retire anything.
How do you run item analysis in a spreadsheet?
Score each answer as 1 or 0 against the key, then use a few standard functions in Excel or Google Sheets. The University of Florida module notes that Canvas calculates these statistics automatically, so check whether your LMS already does and which formula it uses.
- Score: compare each response with the key so every item column holds 1 or 0, then add a total column and a rest-score column (total minus that item).
- Difficulty: use AVERAGE on the item’s 0/1 column.
- Point-biserial: use CORREL(item column, rest-score column). CORREL returns the correlation of two ranges, which for a 0/1 column is the point-biserial.
- Upper-lower index: sort by total score, fix your group size and tie rule, then subtract the lower group’s AVERAGE from the upper group’s.
- Options: use COUNTIF on the raw answers for each of A, B, C and D, split by group.
What should you do next?
Pick one recent test with a full cohort of answers. Compute p, a discrimination measure and option counts for every item, then work through the five checks above. Send flagged items to a subject matter expert with the option counts attached, not just the numbers.
Record the outcome in your item bank, along with the cutoffs you adopted and whose rule of thumb they came from.
If you deliver tests on SimpliAssess, its features page lists item banking and analytics and reporting among its capabilities. It does not describe which item statistics the reports show, so confirm that before you rely on them.
Frequently asked questions
What is a good item difficulty index?
It depends on the item’s purpose. The University of Florida module suggests about 0.25 to 0.75 for most items. The University of Arizona guide accepts 0.80 to 1.00 for mastery items and 0.30 to 0.70 for items meant to separate learners. Treat these as rules of thumb from different sources, and judge each item against what it is supposed to do.
What is a good discrimination index?
The University of Florida module names 0.20 as the lowest recommended value, and the University of Arizona guide calls 0.30 or higher highly discriminating. For the point-biserial version, the University of Washington scoring office calls values above .30 good. Aim for clearly positive values, and expect very easy items to score low without being faulty.
What does a negative discrimination index mean?
It means the lower-scoring group got the item right more often than the upper group. The University of Washington scoring office says to examine such items and notes the cause can be a mis-keyed answer. Check the key and wording first. In a small class, also check whether one or two learners explain the result.
How many learners do you need for item analysis?
The guides reviewed for this post set no single minimum for item-level statistics. One university guide pairs a reliability figure with at least 30 takers and 50 items. With fewer, treat the statistics as prompts for expert review, and pool answers across several cohorts before retiring any item.
What is a non-functioning distractor?
It is a wrong option that few or no learners choose, so it does not help separate those who know the content from those who do not. Castle Worldwide flags options chosen by under 3 percent of candidates, and a Frontiers in Medicine study cites a traditional cutoff of 5 percent. With 20 learners, one choice is already 5 percent.
Is the point-biserial the same as the discrimination index?
They answer the same question but are not identical. The upper-lower index compares only the top and bottom groups. The point-biserial is the correlation between item score and total score, so it uses every learner. The University of Washington scoring office reports the correlation as its discrimination index. Check which one your tool shows.
