Automated video interviews (AVIs) are job interviews that use computers to assess candidates. These AVIs extract behavioral information from video interviews including verbal (e.g., word count), nonverbal (e.g., facial expressions), and paraverbal (e.g., pitch). They use statistical models to make predictions about the job-relevant characteristics of a candidate. AVIs are becoming increasingly popular because they can help organizations save time and money during the hiring process. However, we still know little about the psychometric properties (e.g., reliability, validity, generalizability) of AVIs used in selection.
PSYCHOMETRIC PROPERTIES OF VIDEO INTERVIEWS
In this study, the researchers (Hickman et al., 2022) developed statistical models to predict both self-reported and interviewer-reported personality traits using mock video interviews. Results showed that AVIs trained on interviewer-reports of personality generally had better psychometric properties (in advanced terms, better construct validity and generalizability to other interview questions) compared to AVIs trained on self-reports of personality. However, the consistency of the test across multiple administrations (test-retest reliability) had mixed results depending on the trait of interest; this finding applied to both models – those trained on self-report and those trained on interviewer-reports.
The researchers also found that the verbal, paraverbal, and nonverbal behaviors extracted and used as predictors in the statistical models tended to be conceptually relevant to the trait of interest. Models trained on interviewer-reports also were able to predict academic outcomes. Therefore, these findings support the idea that AVIs can accurately tap into the content of personality traits.
PRACTICAL APPLICATIONS FOR ORGANIZATIONS
Overall, this study shows mixed evidence regarding the psychometric properties of AVIs. Still, models trained on interviewer-reports of personality tended to fare better compared to models trained on self-reports of personality. For example, AVIs trained on interviewer-reports of personality were good at being able to discriminate between different psychological concepts, which has traditionally been a challenge for ordinary job interviews. Therefore, using AVIs may be helpful when trying to isolate specific job-relevant behaviors.
The results also provide initial evidence that the psychometric properties of AVIs trained on one set of interview questions may stay relatively the same when the same statistical model is used on video interviews that used a different set of question. This suggests that although standardizing the questions used in AVIs can improve psychometric properties, organizations may be justified in using different interview questions should they choose to do so.
Although some of the evidence found in this research is promising, the authors caution that much more evidence is needed before organizations can fully justify the use of AVIs in hiring.
Hickman, L., Bosch, N., Ng, V., Saef, R., Tay, L., & Woo, S. E. (2022). Automated video interview personality assessments: Reliability, validity, and generalizability investigations. Journal of Applied Psychology. Advance online publication.