Researchers (Cucina et al., 2012) recently conducted a study to explore the best method of scoring biographical data (biodata) measures. Biodata can be scored using empirical keying methods in which the assessor weights item responses based on the objective relationship between the item and performance; rational keying, in which scorers come up with subjective estimates of that relationship based on theory; and a hybrid approach, in which assessors use both rational and empirical keying. In this study, the authors answered seven research questions. We’ve provided you with a summary of the questions and answers:
(1) Which scoring approach (i.e., rational, empirical, or hybrid) yields the highest criterion-related validity?
With sample sizes of 1600 cases or less, empirical keying with unit weights has the highest validities. With smaller sample sizes, hybrid keying with unit weights resulted in less variability.
(2) Do the different empirical keying procedures (e.g., vertical percent, point biserial, mean criterion, etc.) have different criterion-related validities?
Not really – they’re quite similar.
(3) Which biodata scoring procedures should practitioners use (considering factors such as validity, feasibility, and legal defensibility)?
The different methods are very similar, so practitioners can use any of them. The simplest method is the point biserial raw weights method, which can be done in many statistics programs.
(4) Do the different biodata scoring procedures yield similar (i.e., highly correlated) scores?
Yes, very.
(5) Does sample size impact the validity of the different biodata scoring procedures?
Yes. For the empirical and hybrid keying approaches with unit weighting of items, the relationship is strongly positive up to about 500 cases, but then it levels off. When items are weighted with stepwise regression, the relationship is positive but diminishes after about 1600 cases. With rational keying using unit weights, sample size didn’t really affect validity.
(6) What are the sample size requirements for empirical and hybrid keying?
It depends on the power and the specific method that you use. The researchers provide some general guidelines though.
(7) Does hybrid keying decrease the sample size requirements?
Only when using stepwise regression and over 1600 cases.
Cucina, J. M., Caputo, P. M., Thibodeaux, H. F., & Maclane, C. N. (2012). Unlocking the key to biodata scoring: A comparison of empirical, rational, and hybrid approaches at different sample sizes. Personnel Psychology, 65, 385-428.
Image credit: Unsplash+