SJTs: They’re Not Perfect, but Gosh Darn it they Work!

Topic: Selection
Publication: Human Performance ( JAN 2010)
Article: Contextual effects on SJT responses: An examination of construct validity and mean differences across applicant and incumbent contexts
Authors: W.I. MacKenzie, R.E. Ployhart, J.A. Weekley, and C. Ehlers
Reviewed By: Benjamin Granger

A situational judgment test (SJT) is a commonly used employee selection tool which presents job applicants with realistic work situations. Job applicants are required to choose from several response options, which range in their effectiveness (as rated by subject matter experts).  While SJTs tend to predict future job performance rather well, there is still debate as to what SJTs actually measure (SJTs correlate with and likely measure cognitive ability, personality, job knowledge, and experience simultaneously) and how they operate in different contexts (e.g., job applicants vs. incumbents).

To further investigate the validity of SJTs, MacKenzie et al. (2010) addressed a few major issues common in SJT research. For example, much of the existing research on SJTs relies on samples of job incumbents.  However, SJTs are most frequently used in new employee selection and the authors speculate that the knowledge, skills, and abilities of incumbents may differ significantly from that of job applicants.  In other words, SJTs may operate differently for job incumbents and applicants.

Impressively, MacKenzie et al. collected SJT data from applicant and incumbent samples from six different organizations, arriving at a total sample of over 40,000 people. Because the SJTs were  tailored to the individual organizations using them, six different SJTs were employed. However, all six were similarly designed (Each situation was accompanied by five response options and participants chose the “best” and “worst” responses).

For all six organizations, job incumbents tended to have higher SJT scores than applicants. More importantly, the findings suggest that SJTs do function differently for job applicants and incumbents.  For all six organizations, cognitive ability was more strongly related to SJT scores for job incumbents than for applicants.

MacKenzie et al.’s findings suggest that practitioners should take care when interpreting the results of SJT research conducted on job incumbents, especially if the SJTs are intended for use in new employees in selection settings.  Nevertheless, SJTs enhance our ability to make predictions about future job performance and are even known to reduce adverse impact against racial minorities, which is a common by product of using pure cognitive ability tests in selection.

MacKenzie, W.I., Ployhart, R.E., Weekley, J.A., & Ehlers, C. (2010). Contextual effects on SJT responses: An examination of construct validity and mean differences across applicant and incumbent contexts. Human Performance, 23(1), 1-21.MacKenz