What Do Situational Judgment Tests Really Measure?

Topic(s): job performance, selection
Publication: Personnel Psychology (2010)
Article: Situational judgment tests: Constructs assessed and a meta-analysis of their criterion-related validity
Authors: M.S. Christian, B.D. Edwards, J.C. Bradley
Reviewed by: Benjamin Granger

As part of an employee selection system, organizations may give situational judgment tests (STJs). These tests ask people to evaluate a realistic work situation and identify the best option for handling it (e.g., “A customer is complaining that his phone is no longer working and wants a full refund even though the warranty ran out yesterday. What do you do?”). They garner a lot of attention in selection contexts because of their effectiveness for predicting job performance, their ability to measure many performance predictors (constructs), and their tendency to reduce sub-group differences that are often found with pure cognitive ability tests.

However, some of the advantages of SJTs may also be disadvantages. For instance, because SJTs can measure a variety of job performance predictors, it is often unclear what they actually (or usually) measure. Another complicating factor is that SJTs come in many formats (e.g., paper-and-pencil, video-based, computer-based, verbally) and may not all have the same degree of effectiveness.

RESULTS OF THE RESEARCH STUDY

In an extensive meta-analysis (or statistical combination of many previous studies), researchers (Christian, Edwards, & Bradley, 2010) located a large number of SJT studies and organized them into categories based on what they are intended to measure. Here are the categories and the percentage of total studies that measured the particular category: job knowledge and skills (3%), interpersonal skills (13%), teamwork skills (4%), leadership skills (37%), personality (10%), and what they call “heterogeneous composites” (33%) which are basically pot-luck.

In general, all of the categories were found to predict job performance. It should be noted, however, that SJTs measuring teamwork skills and several personality characteristics tended to have the highest relationships with performance. Interestingly, video-based SJTs were found to relate more positively with performance than paper-and-pencil SJTs. But, the authors note that there were very few studies using video-based SJTs as well as those measuring teamwork skills (about 4% of total) so care should be taken when generalizing these findings.

PRACTICAL IMPLICATIONS FOR ORGANIZATIONS

Ultimately, this study provides clarification as to what SJTs are typically used to measure in employee selection contexts, which has been an elusive issue. The findings also confirmed that SJTs are effective predictors of job performance across the board, though teamwork and composite personality SJTs appear to have the strongest relations with performance. Finally, although more research is needed before we all jump on the video-based SJT bandwagon, video-based SJTs may predict performance better than paper-and-pencil SJTs.

 

Christian, M.S., Edwards, B.D., & Bradley, J.C. (2010). Situational judgment tests: Constructs assessed and a meta-analysis of their criterion-related validity. Personnel Psychology, 63(1), 83-117.