Making the Most Out of Multiple-Choice Testing

Topic(s):

Topic: Measurement
Publication: International Journal of Selection and Assessment
Article: On minimizing guessing effects on multiple-choice items: Superiority of a two solutions and three distractors item format to a one solution and five distractors item format
Authors: K.D. Kubinger, S. Holocher-Ertl, M. Reif, C. Hohensinn, and M. Frebort
Reviewed By: Benjamin Granger

In addition to being popular among test takers, the multiple- choice format test is nearly ubiquitous in employee selection and assessment contexts and offers many advantages (e.g., easily quantified, easily scored, etc.) to organizations.

The most common multiple-choice item format includes a single correct answer and several (perhaps 3 or 4) wrong answers or “distractors.” But, this format leaves the door open to what we may call the “guessing effect.” The basic idea is that, in theory, a person with absolutely no knowledge of the content area can endorse some items correctly by luck (i.e., guessing correctly). In fact, many standardized multiple-choice tests have instruction books that discuss guessing strategies (e.g., ACT, GRE).

Acknowledging the utility of the format itself, Kubinger and colleagues (2010) explored a multiple-choice format with two correct answers as opposed to the single correct (or best) answer that is most commonly used. In order to correctly answer such an item, test takers must endorse BOTH correct answers and cannot endorse any of the distractors. Needless to say, this manipulation makes multiple-choice items substantially more difficult which is indeed what the authors found. In fact, the difficulty of this format was comparable to that of a free response format test of the same content (i.e., math).

However, compared to the traditional multiple-choice format with a single correct answer and five distractors, the two correct answer format drastically reduced the “guessing effect.”

Kubinger et al.’s study presents an interesting alternative to the multiple-choice response formats that we are accustomed to. Although they are significantly more difficult, items that require recognition of two correct answers among three distractors can dramatically reduce the occurrence of lucky guesses that can potentially impact important employment decisions.

Kubinger, K.D., Holocher-Ertl, S., Reif, M., Hohensinn, C., & Frebort, M. (2010). On minimizing guessing effects on multiple-choice items: Superiority of a two solutions and three distractors item format to a one solution and five distractors item format. International Journal of Selection and Assessment, 18(1), 111-115.