Who Knows Better, Job Analysts or Job Incumbents?

Topic(s): job performance
Publication: International Journal of Selection and Assessment (2012)
Article: Only Incumbent Raters in O*NET? Oh Yes! Oh No!
Authors: P.T. Walmsley, M.W. Natali, J.P. Campbell
Reviewed by: Megan Leasher

The U.S. Department of Labor hosts and updates a massive database for a growing number of occupations. This web-based database, O*NET, is available for free to the public and serves as the nation’s leading resource for job information. O*NET is a bastion of knowledge that can be leveraged for many purposes and audiences. Human resource professionals can use it to create job descriptions, develop criteria for selection and performance appraisal systems, and structure compensation systems. Students and job seekers will find value in researching future roles and career paths.

USING THE O*NET DATABASE

The occupational information in the O*NET database is based on a series of descriptors that identify work skills, abilities, activities, and the context in which work takes place. Descriptor data comes from two ratings sources: incumbents who are currently employed in the job as well as occupational analysts, commonly industrial and organizational psychologists who are expertly trained in job analysis. Although most agree that ratings from each of these data sources provide unique job information, others have proposed eliminating the analysts’ ratings in an effort to conserve resources. But instead of simple elimination, some have suggested that a series of statistical calculations could accurately estimate how the analyst would rate each work descriptor, thus substituting an imputed “rating” for what a real, live human analyst actually observing the job would rate.

This research tested this idea, to see if the would-be ratings of an analyst could be accurately computed. If so, it would suggest that analysts would no longer be needed to rate job descriptors, as we could simply calculate what their ratings would be. Who needs more humans when you have stats to compute their thoughts? The authors tested multiple statistical methods, all combinations of work descriptors, and substituted calculated ratings in both directions (incumbent for analyst and vice versa). Overall, they found that it was not possible to accurately substitute one group’s rating for another, nor accurately calculate the would-be ratings of analysts. We need both analyst and incumbent viewpoints for the most accurate O*NET database.

Said another way, the analyst is a job watcher. Peering in from the outside, the job watcher is trained in the art of observation and the science of how to operationalize and measure it.  Then you have the incumbent, the job do-er.  The job do-er is trained in the mental and physical steps required to accomplish the tasks of the job. Both the do-er and the watcher capture valuable, meaningful aspects of the job that the other cannot. They are not mathematically substitutable. They are not imputable from a series of formulas. They are a necessary pair, each unique in the value they bring.

SIMILARITIES WITH 360 DEGREE FEEDBACK

This article sounds very much like the same debates over using 360° feedback tools for development. People have different perspectives for a variety of reasons. These perspectives have been found, for the most part, to be unique enough (from a statistical perspective) that we should pay attention to the different viewpoints. By doing this, we are able to see a more complete, holistic view of a person in order to provide every angle of unique developmental feedback possible. O*NET descriptor ratings appear to follow suit; we want to tell the best, most accurate story possible, and the moral of the story is that one storyteller simply cannot accomplish this alone.

Walmsley, P. T., Natali, M. W., & Campbell, J. P. (2012). Only incumbent raters in O*NET? Oh yes! On no!.  International Journal of Selection and Assessment, 20, 283-296.