Topic: Selection, Assessment
Publication: Personnel Psychology (SUMMER 2012)
Article: Unlocking the key to biodata scoring: A comparison of empirical, rational, and hybrid approaches at different sample sizes
Authors: J. M. Cucina, P. M. Caputo, H. F. Thibodeaux, & C. N. Maclane
Reviewed by: Alexandra Rechlin
Cucina and his colleagues recently conducted a study to explore the best method of scoring biographical data (biodata) measures. Biodata can be scored using empirical keying methods in which the assessor weights item responses based on the objective relationship between the item and performance; rational keying, in which scorers come up with subjective estimates of that relationship based on theory; and a hybrid approach, in which assessors use both rational and empirical keying). In this study, the authors answered seven research questions. We’ve provided you with the questions and answers for your own biodata-scoring enjoyment.
1. Which scoring approach (i.e., rational, empirical, or hybrid) yields the highest criterion-related validity?
With sample sizes of 1600 cases or less, empirical keying with unit weights has the highest validities. With smaller sample sizes, hybrid keying with unit weights resulted in less variability.
2. Do the different empirical keying procedures (e.g., vertical percent, point biserial, mean criterion, etc.) have different criterion-related validities?
Not really – they’re quite similar.
3. Which biodata scoring procedures should practitioners use (considering factors such as validity, feasibility, and legal defensibility)?
The different methods are very similar, so you could use any of them. The simplest method is the point biserial raw weights method, which you can do in many statistics programs.
4. Do the different biodata scoring procedures yield similar (i.e., highly correlated) scores?
5. Does sample size impact the validity of the different biodata scoring procedures?
Yes. For the empirical and hybrid keying approaches with unit weighting of items, the relationship is strongly positive up to about 500 cases, but then it levels off. When items are weighted with stepwise regression, the relationship is positive but diminishes after about 1600 cases. With rational keying using unit weights, sample size didn’t really affect validity.
6. What are the sample size requirements for empirical and hybrid keying?
It depends on the power and the specific method that you use (it always depends, but isn’t that just like psychology!). The researchers provide some general guidelines though.
7. Does hybrid keying decrease the sample size requirements?
Only when using stepwise regression and over 1600 cases.
So, when all of the methods work, it becomes important to tailor your scoring to the size and data that you have. Results may be similar across methods, but better methods give you better results. One size doesn’t fit all method types!
Cucina, J. M., Caputo, P. M., Thibodeaux, H. F., & Maclane, C. N. (2012). Unlocking the key to biodata scoring: A comparison of empirical, rational, and hybrid approaches at different sample sizes. Personnel Psychology, 65, 385-428. doi: 10.1111/j.1744-6570.2012.01244.x
human resource management, organizational industrial psychology, organizational management