Why the ‘None of the above’ answer is poor test design

A new study argues that some of the most common practices within a multiple choice test are not actually good for measuring learning.

For those of us who hate standardized tests, we may suspect that multiple choice questions are designed more to trip us up than to actually help us learn. And we may be right. Multiple choice tests are a popular assessment tool used to test the knowledge of students and professionals. But a new study argues that some of the most common practices within a multiple choice test are not actually good for measuring learning.

For the Journal of Applied Research in Memory and Cognition, Andrew Butler of Washington University in St. Louis reviewed the literature of test-taking and came away with evidence-based tips of what does and does not work. Top of his list of don’ts: complex question formats. He found that complex question-and-answer formats confuse participants and question formats should “challenge students but allow them to succeed often, and target specific cognitive processes that correspond to learning objectives.”

Avoid ‘none of the above’ if you want students to learn

When the correct answer to a test is “A and B, but not C,” students have to jump through a variety of hoops in their brain to reach the right answer. Students are more likely to get stumped by the options and can have more misunderstandings as a result. This makes the complex question format an unreliable tool to measure learning. “The variability in responding among test-takers reduces reliability, which is critical for assessment,” Butler writes.

One of the most interesting tips from Butler’s research was to avoid “None of the Above” or “All of the above” constructions if you want to help test-takers learn. “When ‘None of the Above’ is the correct response, test-takers are presented with only incorrect information,” he writes. If your goal is teaching a student, you don’t want to remind them of wrong answers, you want to reinforce right answers. “All of the Above” can also be confusing when it is the wrong answer, but it can be helpful when it is the correct answer, because the test-taker is “only exposed to correct information.”

You want a test that is not too easy and not too hard. Butler said that “the ideal difficulty level is a bit higher than the midpoint between chance and perfect performance.” The main takeaway is that test-takers do not need to go through A-J choices to be challenged enough to learn. Test-makers and graders can make their jobs easier by simplifying answer formats. “Keeping the process of answering multiple-choice items simple maximizes both effectiveness and efficiency,” Butler concludes.

The act of taking a test is not just assessing learning, it causes it too. Test-makers need to be sure they are modeling the right kind of learning they want test-takers to remember.

Monica Torres|is a reporter for Ladders and can be reached at mtorres@theladders.com.