Do Situational Judgment Tests Work?… and more

Topic: Selection
Publication: Personnel Psychology (SPRING 2010)
ArticleSituational judgment tests: Constructs assessed and a meta-analysis of their
criterion-related validity

Authors: M.S. Christian, B.D. Edwards, and J.C. Bradley
Reviewed By: Benjamin Granger

A situational judgment test (STJs)
is a test which asks a person to evaluate a realistic work situation and identify the best option for handling it (e.g., “A customer is complaining that his phone is no longer working and wants a full refund even though the warranty ran out yesterday.  What do you do?”).  They garner a lot of attention in selection contexts because of their effectiveness for predicting job performance, their ability to measure many performance predictors (constructs) and their tendency to reduce sub-group differences that are often found with pure cognitive ability tests.  But some of the advantages of SJTs may also be disadvantages.

For instance, because SJTs can measure a variety of job performance predictors, it is often unclear what they actually (or usually) measure.  Another complicating factor is that SJTs come in many formats (e.g., paper-and-pencil, video-based, computer-based, verbally) and may not all have the same degree of effectiveness.

So you want answers huh?  Well, here they are…

In an extensive meta-analysis, Christian, Edwards, and Bradley (2010) located a large number of SJT studies and organized them into categories based on what they are intended to measure.  Here are the categories and the percentage of total studies that measured the particular category: job knowledge and skills (3%), interpersonal skills (13%), teamwork skills (4%), leadership skills (37%), personality (10%), and what they call “heterogeneous composites” (33%) which are basically pot-luck.

In general, all of the categories were found to predict job performance.  It should be noted, however, that SJTs measuring teamwork skills and multiple personality characteristics tended to have the highest relations with performance. Interestingly, video-based SJTs were found to relate more positively with performance than paper-and-pencil SJTs. But, the authors note that there were very few studies using video-based SJTs as well as those measuring teamwork skills (~4% of total) so care should be taken when generalizing these findings.

Ultimately, Christian and colleagues’ study provides clarification as to what SJTs are typically used to measure in employee selection contexts (which has been an elusive issue to say the least).  Their findings also confirmed that SJTs are effective predictors of job performance across the board, though teamwork and composite personality SJTs appear to have the strongest relations with performance.  Finally, although more research is needed before we all jump on the video-based SJT bandwagon, video-based SJTs may predict performance better than their paper-and-pencil SJTs.

Christian, M.S., Edwards, B.D., & Bradley, J.C. (2010). Situational judgment tests: Constructs assessed and a meta-analysis of their criterion-related validity. Personnel Psychology, 63, 83-117.