What Do Situational Judgment Tests Really Measure?

Topic(s): selection
Publication: Journal of Applied Psychology
Article: How “Situational” Is Judgment in Situational Judgment Tests?
Authors: S. Krumm, F. Lievens, J. Hüffmeier, A.A. Lipnevich, H. Bendels, G. Hertel 
Reviewed by: Ben Sher

Situational judgment tests are often used during employee selection. They present the job applicant with a series of situations that may be encountered on the job. For example, one situation might include an anecdote about a coworker encouraging you to steal. For each situation, several different responses are listed. Applicants simply choose the response that seems most appropriate. Because these tests are (hopefully) designed by highly trained experts, certain answers are designed to reflect behavior that is consistent with good job performance. The more the applicant chooses these “good” answers, the more certain we are that the applicant will succeed on the job if hired.

The theory behind situational judgment tests is that applicants who score well are better equipped with knowledge that is very specific to the job. It is not that the high scoring applicant is necessarily smarter, or has greater social tact. Instead, we believe that the high-scoring applicant has a certain kind of knowledge or skill that is useful for succeeding at the very specific situations that will be encountered on the job. However, new research (Krumm et al., 2015) showed that our assumptions about situational judgment tests may be wrong. 

SITUATIONAL JUDGMENT TESTS: WHAT’S REALLY GOING ON

The researchers presented scenarios from real situational judgment tests, and investigated what would happen if they left out the situational anecdote and simply offered the behavioral responses. They wanted to know if people could identify the best answer without even knowing the question.

The researchers conducted several studies using real situational judgment tests. They found that in 43-71% of all scenarios (across different studies), it did not matter if the applicants were given the situational scenarios or not. They had an equal chance of getting the item correct whether or not the items were presented with the situational scenarios or by themselves. 

What does this mean? The authors explain that many items from these situational judgment tests are not measuring knowledge that is specific to the job, but are instead measuring broader knowledge or abilities that might work on any job, such as social-skills or intelligence. The authors did find two scenarios in which applicants benefited from having the situational scenarios instead of just the behavioral answers. The first is when the items measured skills or abilities that specifically related to the job in question, and the second is when the behavioral options or answers included actions that were very specific to the given scenario, and not just generally good or bad behavior. 

ORGANIZATIONAL IMPLICATIONS

This research shows that around half of all items on situational judgment tests are not measuring knowledge or skills that are needed for a specific job context. Instead, these rogue items seem to be measuring broader traits applicable to many jobs. Why does this matter? 

Organizations typically invest time and money into developing a situational judgment test that reveals which employees are best suited to a specific job. They usually convene a panel of subject-matter experts (SMEs) to rigorously develop the scenarios and behavioral possibilities for these tests. If the organization is content with measuring broad general traits useful for employees, they may be able to use generic “off-the-shelf” tests that have already been developed, instead of investing the resources into developing a situational judgment test for the specific job. The authors say that this may be the case for entry-level or other low-complexity jobs. 

Another alternative is to design a test that is specific to a job and simply omit the situational scenarios, providing the behavioral responses alone. This could also save the organization time and money, because the developmental process would be shorter. 

Finally, if organizations really want to have a situational judgment test tailor-made for a job, they can do themselves a favor and make sure that the questions really are specific to the job. Make sure that the items measure job-specific information, and make sure that the behavioral response options are very specific to the scenario. This will ensure that the situational judgment test is measuring what it intends to measure.

 

Krumm, S., Lievens, F., Hüffmeier, J., Lipnevich, A. A., Bendels, H., & Hertel, G. (2015). How “situational” is judgment in situational judgment tests? Journal of Applied Psychology, 100(2), 399-416.