Category Archives: Research Methodology
Self-Starting Research on a Test of Personal Initiative
Topic: Research Methodology
Publication: Personnel Psychology (SUMMER 2009)
Article: A situational judgment test of personal initiative and its relationship to performance.
Author: R. Bledow, M. Freese
Reviewed By: Katie Bachman
Predicting performance is the Holy Grail of Industrial/Organizational Psychology. I hate to whine, but how can we possibly measure performance when those darn applicants don’t take our perfectly designed tests properly? Sarcasm aside, self-report measures using Likert scales (e.g., 5-point rating scales, where 1 = Low and 5 = High) tend to elicit generalized responses based on previous experience or self-concept, both of which might not represent reality.
Consider the following example:
Question: To what extent are you likeable?Answer: 5 = extremely likeableReality: Not really
In the present research, Bledow and Frese have attempted to get past the usual barriers presented in self-report measures by creating a behavioral construct-specific situational judgment test (SJT). An SJT is a measure that gives respondents scenarios and responses and then asks them which response they would be most and least likely to perform. An individual’s level on the desired trait is inferred from the response. In this article, the trait of interest was personal initiative. Their test was effective at predicting performance and did so differently than did the self-report measure.
For employers, this type of test could prove useful when trying to pin down relevant traits for potential employees. Rather than asking how employees would act or did act in a given situation, we can instead ask how they are acting.
Bledow, R., & Frese, M. (2009). A situational judgment test of personal initiative
and its relationship to performance. Personnel Psychology, 62, 229-258.
Friends don’t let friends use unsubstantiated selection and development methods
Topic: Assessment, Research Methodology
Publication: International Journal of Selection and Assessment
Article: HR professionals’ beliefs about, and knowledge of, assessment techniques and psychometric tests.
Blogger: Rob Stilson
Some academics slave over piles of data and spend months of their lives trying to determine the best selection and development methods available for the work place. Makes you wonder; is anybody paying attention? If your life’s work involves developing the absolute, most reliable and valid methods for selection and development, you may want to stop reading now.
For this study, the authors collected survey data from 255 professionals in the personnel selection and development field (mainly in the UK and Europe). Questions concerned different techniques of selection and development (e.g., interviews, references, peer ratings, cognitive ability tests, personality tests, work samples, job knowledge, educational qualifications, 360s assessment centers, biodata, and personal hunches) and specific questions on personality tests and aptitude tests. The survey inquired about how the participants felt about validity, cost, practicality and the legality of the different methods.
For the personality and aptitude tests, the questions included;
· have you heard of this test?
· have you completed this test?
· does your organization use this test?
· how valid do you rate this test? 1-10
· how useful is this test? 1-10
· how useful is this test for selection? 1-10
· how useful is this test for development? 1-10
I won’t bore you with how it all shook out, but some of the interesting findings of this survey included the fact that some tests that have very little research behind them or have been bashed psychometrically still remain very popular among practitioners. I encourage you to read this article if you know anybody in the applied world.
Is interrater correlation really a proper measurement of reliability?
Topic: Measurement, Research Methodology, Statistics
Publication: Human Performance
Article: Exploring the relationship between interrater correlations and validity of peer ratings
Blogger: Rob Stilson
Interrater reliability (still with me?, Ok good) is often used as the main reliability estimate for the correction of validity coefficients when the criterion is job performance. Issues arise with this practice when one considers that the errors present between raters may not be random, but due to bias, while agreement between raters may also stem from bias instead of actual consistency. In this study, the authors’ main goal was to explore the relationship between interrater correlations and validity and also to explore the relationship between the number of raters and validity.
In order to do this, the authors gathered information from 3072 Israeli policemen from 281 work teams who took part in peer rating. The average size of each of these work teams averaged about 12 people and ranged from 5 all the way to 33. The measure used was overall performance (on a 7-point Likert scale). The predictor employed in this study was the ICC (C,k) model, which is equivalent to Cronbach’s alpha. Measurement indices were computed on the team level as rating only took place within work teams.
The predicted variable for the study was the validity coefficient for each work team. This is the part of the study where you could really feel the sweat involved. Here the authors gathered information on
supervisor evaluations, absenteeism data, and discipline data collected over several years (for over 3000 policemen)! The authors then converted this information into z scores with higher scores indicating better performance.
Results showed a weak positive linear relationship between interrater correlations and the various validity indexes. This is not what you want to hear if you are doing peer rated performance evaluations. The authors’ stipulate that the correlation between raters is a conglomeration of factors
having different theoretical relationships with validity (i.e. bias and other idiosyncrasies).
Practical implications from the information gleaned here include the adjustment of validity due to attenuation. If the measurements used in the calculation included non random error estimates, the ensuing calculations will be off. A positive finding for the work world was validity in small units (less than 10 people) was about the same as those for larger units. The authors’ believe this finding may be due to observation opportunity level, which is seemingly greater in smaller work units.
It’s the simple things in life…
Topic: Research Methodology
Publication: Organizational Research Methods
Article: Grounded Theory Method in management research: Users’ perspectives.
Blogger: James Grand
Qualitative research is the “art” of taking real-time narratives or observations of human behavior/cognitions and systematically analyzing them for themes, theories and indicators of what is occurring in a given situation. In truth, such data is collected all the time in organizations.
For example, employees are often asked to provide HR departments with their goals for the year, provide feedback on new company policies, or summarize the information they learned in a training seminar. The real challenge for organizations (and researchers as well) is figuring out how to use these observations to understand their employees and affect meaningful changes just as they would had people provided survey data (quantitative). Grounded Theory Method (GTM, Glaser & Strauss, 1967) provides one such procedure. In brief, it involves a series of stepsin which the transcribed responses of individuals are reduced into increasinglysuccinct bits of information until common, thematic keywords can be identifiedthat describe the nature of the phenomenon. If this sounds like an onerous process, that’s because it is.
However, the most recent issue of Organizational Research Methods (2008, Vol. 11, Iss. 3) presents an article by Fendt and Sachs that provides a number of recommendations for flattening that learning curve a bit, including tips on how to deal with nonverbal information, researcher bias, and making sense of your data. While these are difficult concepts to describe without a better understanding of GTM, such information is invaluable to researchers and practitioners hoping to use observational data without getting swamped in their analytic efforts. And that simplicity is something we can all appreciate.
Fendt, J., & Sachs, W. (2008). Grounded Theory Method in management
research: Users’ perspectives. Organizational Research Methods, 11(3), 430-455.
Glaser, B. G., & Strauss, A. L. (1967). The
discovery of grounded theory: Strategies for qualitative research. New York: De Gruyter.
It’s The Simple Things In Life…
Topic: Research Methodology
Publication: Organizational Research Methods
Article: Grounded Theory Method in management research: Users’ perspectives.
Blogger: James Grand
Qualitative research is the “art” of taking real-time narratives or observations of human behavior/cognitions and systematically analyzing them for themes, theories and indicators of what is occurring in a given situation. In truth, such data is collected all the time in organizations.
For example, employees are often asked to provide HR departments with their goals for the year, provide feedback on new company policies, or summarize the information they learned in a training seminar. The real challenge for organizations (and researchers as well) is figuring out how to use these observations to understand their employees and affect meaningful changes just as they would had people provided survey data (quantitative).
Grounded Theory Method (GTM, Glaser & Strauss, 1967) provides one such procedure. In brief, it involves a series of steps in which the transcribed responses of individuals are reduced into increasingly succinct bits of information until common, thematic keywords can be identified that describe the nature of the phenomenon. If this sounds like an onerous process, that’s because it is.
However, the most recent issue of Organizational Research Methods (2008, Vol. 11, Iss. 3) presents an article by Fendt and Sachs that provides a number of recommendations for flattening that learning curve a bit, including tips on how to deal with nonverbal information, researcher bias, and making sense of your data.
While these are difficult concepts to describe without a better understanding of GTM, such information is invaluable to researchers and practitioners hoping to use observational data without getting swamped in their analytic efforts. And that simplicity is something we can all appreciate.
Fendt, J., & Sachs, W. (2008). Grounded Theory Method in management
research: Users’ perspectives. Organizational Research Methods, 11(3), 430-455.
