Topic: Measurement
Publication: International Journal of Selection and Assessment
Article: On minimizing guessing effects on multiple-choice items: Superiority of a two solutions and three distractors item format to a one solution and five distractors item format
Authors: K.D. Kubinger, S. Holocher-Ertl, M. Reif, C. Hohensinn, and M. Frebort
Reviewed By: Benjamin Granger
In addition to being popular among test takers, the multiple- choice format test is nearly ubiquitous in employee selection and assessment contexts and offers many advantages (e.g., easily quantified, easily scored, etc.) to organizations.
The most common multiple-choice item format includes a single correct answer and several (perhaps 3 or 4) wrong answers or “distractors.” But, this format leaves the door open to what we may call the “guessing effect.” The basic idea is that, in theory, a person with absolutely no knowledge of the content area can endorse some items correctly by luck (i.e., guessing correctly). In fact, many standardized multiple-choice tests have instruction books that discuss guessing strategies (e.g., ACT, GRE).
Acknowledging the utility of the format itself, Kubinger and colleagues (2010) explored a multiple-choice format with two correct answers as opposed to the single correct (or best) answer that is most commonly used. In order to correctly answer such an item, test takers must endorse BOTH correct answers and cannot endorse any of the distractors. Needless to say, this manipulation makes multiple-choice items substantially more difficult which is indeed what the authors found. In fact, the difficulty of this format was comparable to that of a free response format test of the same content (i.e., math).
However, compared to the traditional multiple-choice format with a single correct answer and five distractors, the two correct answer format drastically reduced the “guessing effect.”
Kubinger et al.’s study presents an interesting alternative to the multiple-choice response formats that we are accustomed to. Although they are significantly more difficult, items that require recognition of two correct answers among three distractors can dramatically reduce the occurrence of lucky guesses that can potentially impact important employment decisions.
Kubinger, K.D., Holocher-Ertl, S., Reif, M., Hohensinn, C., & Frebort, M. (2010). On minimizing guessing effects on multiple-choice items: Superiority of a two solutions and three distractors item format to a one solution and five distractors item format. International Journal of Selection and Assessment, 18(1), 111-115.
This appears to be a good idea to try, however I believe test items of this nature will increase adverse impact, especially if used for entry-level jobs.
This is a really interested point. It does seem as though this increases the cognitive load of the test items. It also seems that this would increase the variability in test scores considerably and ultimately would increase validity. Once again, the diversity-validity dilemma rears its ugly head…
As a practitioner, I believe that if you gave no points for correctly answering one out of two correct options, your test takers would really make a fuss. They would argue that you gave no points when they really got it half correct. And that you could have easily given partial credit (1 out of 2 points). I would expect an upheaval/group appeal. Separate issue: what about the following hypothetical scenario: subject A correctly answered one of the two correct alternatives on all questions. Since you must answer BOTH correctly, he failed miserably (zero points on whole test). Subject B also correctly answered half of all keyed responses, but his correct answers occur such that he ends up answering both correct answers in half the items. Subject B earned credit for half the test items while subject A correctly answered the same number of (collective) response options and is scored as missing every question. Do the authors address any of this?
The authors attack this issue from a purely “guessing” perspective and appear not to address this issue in their paper. In fact, in their general discussion, the authors make some very interesting (rather sweeping) statements about the use of traditional MC formats (one correct answer) vs. the format that they endorse.
I think your point brings up another important issue: how would applicants react to such a format?