AI in assessment: 8 ways test design is changing

AI in assessment is changing test design in eight practical ways, from how items are drafted to how open-ended answers are scored and documented. Drafting and scoring are getting cheaper, so design choices, human review and evidence now decide test quality. This guide is for assessment authors, instructional designers and training providers who build tests.

It rests on studies, standards and regulator texts published between 2025 and 2026, dated in the text and current as of November 2026. We have not tested any AI tool for this post. Where the evidence is thin we say so, and where a point is our own editorial judgment we label it. Earlier posts on this site covered the promise in the rise or fall of AI in assessments and generative AI for test content. This one looks at what newer evidence supports.

Key takeaways

• AI-drafted items matched expert items on difficulty and discrimination in one 2026 study, but a 2025 preprint found AI items easier, so pilot your own items.
• When drafting is cheap, the blueprint, the difficulty target and the pilot become the main design work.
• AI scoring of extended writing agreed with human markers at human-level rates in one national review, and the reviewers still kept people on low-confidence and near-cut-score responses.
• Plan for learners who use AI, give every AI-assisted step a named human reviewer, and keep records, because standards and the EU AI Act now address this.

What does the evidence on AI in assessment actually show?

Recent studies show AI-drafted multiple-choice items can match expert-written items on difficulty and discrimination, and one national review found AI scoring of writing agreed with human markers about as well as two markers agree with each other. Every result comes with limits, summarized below as of November 2026.

Treat the vendor row as a claim, not a finding. The other three cover medicine, college courses and national literacy writing, not corporate training, so none proves the result for your courses.

Which eight changes is AI in assessment bringing to test design?

Eight changes recur across the sources: AI-assisted drafting, blueprint-led design, pilot-based difficulty, scored open responses, AI-aware tasks, measuring AI skills, designed human review and documentation. Each section below says what the published evidence supports and where it stops.

1. How does AI change who drafts the items?

Drafting shifts from a person writing every item to an expert reviewing drafts generated from approved source material. In the January 2026 npj Digital Medicine study, GPT-4o items and human-written items showed no significant difference in difficulty or discrimination.

Participants could not tell which items were AI-written, yet a human-in-the-loop review still corrected a few factual errors, so review does not go away. For the review steps themselves, see our guide to AI question generation for training companies.

2. Why does the blueprint matter more when drafting is cheap?

The blueprint becomes the main design document. When a draft takes seconds, the objective, cognitive level, source material and difficulty target you supply decide how much review each item needs. This is our editorial judgment, not a study finding.

The ITC/ATP Guidelines for Technology-Based Assessment, version 1.1 (July 2025), point the same way. They say assessment designers or content experts should evaluate the validity of the process an item-generation engine uses. The guidelines do not prescribe when to use technology; they inform your choices.

3. Why do difficulty targets need pilot data?

AI drafts can skew easier than you intend, so check difficulty against learner data. In a preprint by Isley and colleagues (August 2025), students answered about 60% of AI-generated items correctly versus 39% of benchmark items drawn from 2012 AP Statistics practice questions.

The authors’ pipeline kept the 10 hardest of 20 accepted questions, and the AI items still came out easier, though slightly more discriminating. The Linde study found near parity on difficulty, so the sources differ. They used different subjects, models and benchmarks, and neither result can be assumed for your course. Pilot items and run item analysis before you set a cut score.

4. Can AI score open-ended answers at scale?

AI scoring of extended writing reached human-level agreement in at least one national review, though people still check the edges. A February 2026 review of New Zealand’s NZQA scoring model reported a quadratic weighted kappa of 0.717 against human markers, compared in the report with about 0.6 typical between two humans.

The NZCER review covered literacy writing from the 2025 assessment events. It reported exact agreement of about 0.797 and no evidence of demographic bias across the gender, ethnicity and equity-index groups checked, although agreement varied slightly by group (0.739 for males, 0.674 for females). It recommended keeping human moderation, with people reviewing low-confidence essays and those near cut-score boundaries.

Pro tip: Send The Edges To People. Route every AI-scored response that lands near the pass mark to a human, whatever the model’s confidence. That is where one scoring error flips a result, and it extends NZCER’s advice about cut-score boundaries to any pass/fail course.

5. How do you design tasks when learners have AI?

Task design now assumes learners can use AI. The University of Minnesota’s Center for Educational Innovation says there are currently no reliable AI detectors, so it recommends redesigning assessments instead of policing them. Favor process over product, hands-on work and oral or reflective components.

Its redesign steps include tasks that are harder for AI to complete, drafts that show process, and tasks that ask learners to critique AI output. The page is undated and cites no evidence for the detector claim, so treat it as one institution’s guidance. Our judgment: the training equivalents are scenario items, observed practical tasks and short oral checks.

6. Are tests starting to measure how people use AI?

Some tests now measure how people work with AI, not only what they know. The OECD’s PISA 2029 cycle will include a Media and AI Literacy domain, which an ACER release of 1 April 2026 describes as a first-of-its-kind international assessment.

The ACER release says the domain will look at how students access, analyze and critically engage with digital information, including misinformation. It does not say whether AI tools will be used inside the tasks. For workplace training our takeaway is smaller: if the job involves AI tools, add items where the learner must check or correct an AI-generated output.

7. Who reviews AI-assisted work, and how?

Human review becomes a designed step with a named owner, a method and enough time. In February 2026 the assessment vendor Questionmark warned about automation bias, the tendency to over-rely on AI output, and argued that time pressure leads to cursory review.

That matches the Linde study, where reviewers caught factual errors in AI items. In practice, write review guidelines, calibrate reviewers on sample items, and limit how many items one person checks in a sitting. Questionmark’s article recommends guidelines and calibration. The sitting limit is our suggestion, not a published threshold.

8. What documentation does AI-assisted test design now need?

AI-assisted test design needs a paper trail: what the AI did, who reviewed it and what evidence supports the scores. The ITC/ATP guidelines say steps should be taken to keep scoring from seeming like a black box, and that potential bias in scoring technology should be evaluated and tested.

Regulation is moving too. Annex III of the EU AI Act lists AI systems intended to evaluate learning outcomes among high-risk uses in education. The Digital Omnibus on AI, Regulation (EU) 2026/1744, entered into force on 27 July 2026, and law-firm commentary from August 2026 reports that Annex III obligations now apply from 2 December 2027. Whether your tests are covered depends on how and where you use AI, so check with counsel. This is general information, not legal advice.

How should you split test design work between AI and people?

Match the amount of AI to the stakes. For practice quizzes and internal knowledge checks, AI can draft and people can spot-check. For certification or pass/fail decisions, AI can assist, but people own the blueprint, every item and every scoring decision. The table is our judgment, built from the sources above.

Pro tip: Judge Items On Data, Not Gut Feel. In the Linde study, participants identified an item’s origin at chance (0.49), so reviewers cannot spot AI items by instinct. Decide what stays in the bank from pilot difficulty and discrimination, not from a hunch about who wrote the item.

What should you do next?

Pick one stage from the table and pilot it before changing your whole process. Draft a small set of items from approved source material, have an expert review them, give them to a real cohort, and compare their difficulty and discrimination with items you already trust.

SimpliAssess’s AI page says questions can be created from existing tests, articles, eBooks and websites, and it describes automated psychometric checks. The page does not describe a human review step, so plan that step yourself. If you want to see how AI authoring fits your workflow, you can request a SimpliAssess demo.

Frequently asked questions

What is AI in assessment?

AI in assessment means using machine learning or generative AI at any stage of testing: drafting items, selecting or adapting questions, scoring responses, giving feedback, analyzing results or monitoring test sessions. The label covers very different risk levels. Drafting a practice quiz is low risk, while scoring a certification exam is not, so judge each use separately.

Can AI write good test questions?

Sometimes, with review. A January 2026 study found GPT-4o items matched human items on difficulty and discrimination, and an August 2025 preprint found AI items performed comparably to expert-written ones but were easier. Both used human checks and narrow settings, so pilot your own items before they count toward any result.

Can AI score open-ended answers reliably?

In one February 2026 national review, an automated model reached a quadratic weighted kappa of 0.717 against human markers, and the reviewers recommended keeping human moderation for low-confidence and near-cut-score essays. That is one system on one task. Any model you use needs its own validation against human scores, including subgroup checks.

Do AI-written items need a subject matter expert?

Yes. In the Linde study a human-in-the-loop review corrected a few factual errors in GPT-4o items, and the ITC/ATP guidelines say designers or content experts should evaluate the process an item-generation engine uses. An expert should check accuracy, alignment to the objective and the answer key before any learner sees an item.

Does the EU AI Act cover AI used in assessment?

Annex III of the EU AI Act lists AI systems intended to evaluate learning outcomes, and to monitor prohibited behavior during tests, as high-risk uses in education. Commentary from August 2026 says those obligations now apply from 2 December 2027. Coverage depends on your use and location, so ask counsel. This is general information, not legal advice.

How do you design tests when learners can use AI?

Redesign instead of relying on detection. The University of Minnesota’s teaching center says reliable AI detectors do not currently exist and recommends tasks that show process, such as drafts, reflections, oral components and hands-on work, plus tasks where learners critique AI output. Match each choice to the skill you need to certify.