Category Archives: Research Methodology

Self-Starting Research on a Test of Personal Initiative

Reply

Topic: Research Methodology
Publication: Personnel Psychology (SUMMER 2009)
Article: A situational judgment test of personal initiative and its relationship to performance.
Author: R. Bledow, M. Freese
Reviewed By: Katie Bachman

Predicting performance is the Holy Grail of Industrial/Organizational Psychology. I hate to whine, but how can we possibly measure performance when those darn applicants don’t take our perfectly designed tests properly? Sarcasm aside, self-report measures using Likert scales (e.g., 5-point rating scales, where 1 = Low and 5 = High) tend to elicit generalized responses based on previous experience or self-concept, both of which might not represent reality.

Consider the following example: 

Question:  To what extent are you likeable?Answer:  5 = extremely likeableReality:  Not really

In the present research, Bledow and Frese have attempted to get past the usual barriers presented in self-report measures by creating a behavioral construct-specific situational judgment test (SJT). An SJT is a measure that gives respondents scenarios and responses and then asks them which response they would be most and least likely to perform. An individual’s level on the desired trait is inferred from the response. In this article, the trait of interest was personal initiative. Their test was effective at predicting performance and did so differently than did the self-report measure.

For employers, this type of test could prove useful when trying to pin down relevant traits for potential employees. Rather than asking how employees would act or did act in a given situation, we can instead ask how they are acting.

Bledow, R., & Frese, M. (2009). A situational judgment test of personal initiative
and its relationship to performance. Personnel Psychology, 62, 229-258.

Friends don’t let friends use unsubstantiated selection and development methods

Reply

Topic: Assessment, Research Methodology
Publication: International Journal of Selection and Assessment
Article: HR professionals’ beliefs about, and knowledge of, assessment techniques and psychometric tests.
Blogger: Rob Stilson

Testing Some
academics slave over piles of data and spend months of their lives trying to
determine the best selection and development methods available for the work
place. Makes you wonder; is anybody paying attention? If your life’s work
involves developing the absolute, most reliable and valid methods for selection
and development, you may want to stop reading now.

For
this study, the authors collected survey data from 255 professionals in the
personnel selection and development field (mainly in the UK and Europe).  Questions concerned different techniques
of selection and development (e.g., interviews, references, peer ratings,
cognitive ability tests, personality tests, work samples, job knowledge,
educational qualifications, 360s assessment centers, biodata, and personal
hunches) and specific questions on personality tests and aptitude tests. The
survey inquired about how the participants felt about validity, cost,
practicality and the legality of the different methods.

For
the personality and aptitude tests, the questions included;

·      have you heard of this test?
·      have you completed this test?
·      does your organization use this test?
·      how valid do you rate this test? 1-10
·      how useful is this test? 1-10
·      how useful is this test for selection? 1-10
·      how useful is this test for development? 1-10     

I
won’t bore you with how it all shook out, but some of the interesting findings
of this survey included the fact that some tests that have very little research
behind them or have been bashed psychometrically still remain very popular
among practitioners. I encourage you to read this article if you know anybody
in the applied world.

Furnham, A.
(2008) HR professionals’ beliefs about, and knowledge of, assessment techniques
and psychometric tests. International Journal of Selection and Assessment.
16(3), 300-305.

Is interrater correlation really a proper measurement of reliability?

Reply

Topic: Measurement, Research Methodology, Statistics
Publication: Human Performance           
Article: Exploring the relationship between interrater correlations and validity of peer ratings
Blogger: Rob Stilson

Is inter Interrater reliability (still with me?, Ok good) is often
used as the main reliability estimate for the correction of validity
coefficients when the criterion is job performance. Issues arise with this
practice when one considers that the errors present between raters may not be
random, but due to bias, while agreement between raters may also stem from bias
instead of actual consistency. In this study, the authors’ main goal was to
explore the relationship between interrater correlations and validity and also
to explore the relationship between the number of raters and validity.

In order to do this, the authors gathered information from
3072 Israeli policemen from 281 work teams who took part in peer rating. The
average size of each of these work teams averaged about 12 people and ranged
from 5 all the way to 33. The measure used was overall performance (on a
7-point Likert scale). The predictor employed in this study was the ICC (C,k)
model, which is equivalent to Cronbach’s alpha. Measurement indices were
computed on the team level as rating only took place within work teams.

The predicted variable for the study was the validity
coefficient for each work team. This is the part of the study where you could
really feel the sweat involved. Here the authors gathered information on
supervisor evaluations, absenteeism data, and discipline data collected over
several years (for over 3000 policemen)! The authors then converted this
information into z scores with higher
scores indicating better performance.

Results showed a weak positive linear relationship between
interrater correlations and the various validity indexes. This is not what you
want to hear if you are doing peer rated performance evaluations. The authors’
stipulate that the correlation between raters is a conglomeration of factors
having different theoretical relationships with validity (i.e. bias and other
idiosyncrasies).

Practical implications from the information gleaned here
include the adjustment of validity due to attenuation. If the measurements used
in the calculation included non random error estimates, the ensuing
calculations will be off. A positive finding for the work world was validity in
small units (less than 10 people) was about the same as those for larger units.
The authors’ believe this finding may be due to observation opportunity level,
which is seemingly greater in smaller work units.

Kasten,
R., and Nevo, B. (2008) Exploring the relationship between interrater
correlations and validity of peer ratings. Human Performance, 21(2), 180-197.

It’s the simple things in life…

Reply

Topic:  Research Methodology
Publication: Organizational Research Methods
Article: Grounded Theory Method in management research: Users’ perspectives.
Blogger: James Grand

Its the simple things in life Qualitative research is the “art” of taking real-time narratives
or observations of human behavior/cognitions and systematically analyzing them
for themes, theories and indicators of what is occurring in a given
situation.  In truth, such data is
collected all the time in organizations. 
For example, employees are often asked to provide HR departments with
their goals for the year, provide feedback on new company policies, or
summarize the information they learned in a training seminar. 

The real challenge for organizations (and researchers as well) is
figuring out how to use these observations to understand their employees and
affect meaningful changes just as they would had people provided survey data
(quantitative).  Grounded Theory
Method (GTM, Glaser & Strauss, 1967) provides one such procedure.  In brief, it involves a series of steps
in which the transcribed responses of individuals are reduced into increasingly
succinct bits of information until common, thematic keywords can be identified
that describe the nature of the phenomenon.

If this sounds like an onerous process, that’s because it is.  However, the most recent issue of Organizational Research Methods (2008,
Vol. 11, Iss. 3) presents an article by Fendt and Sachs that provides a number
of recommendations for flattening that learning curve a bit, including tips on
how to deal with nonverbal information, researcher bias, and making sense of
your data.  While these are
difficult concepts to describe without a better understanding of GTM, such
information is invaluable to researchers and practitioners hoping to use
observational data without getting swamped in their analytic efforts.

And that simplicity is something we can all appreciate.

Fendt, J., & Sachs, W. (2008). Grounded Theory Method in management
research: Users’ perspectives. Organizational
Research Methods, 11(3), 430-455.

Glaser, B. G.,
& Strauss, A. L. (1967). The
discovery of grounded theory: Strategies for qualitative research. New
York: De Gruyter.

It’s The Simple Things In Life…

Reply

Topic: Research Methodology 
Publication: Organizational Research Methods
Article: 
Grounded Theory Method in management
research: Users’ perspectives. 
     
Blogger: James Grand

Qualitative research is the “art” of taking real-time narratives
or observations of human behavior/cognitions and systematically analyzing them
for themes, theories and indicators of what is occurring in a given situation.  In truth, such data is collected all
the time in organizations.  For
example, employees are often asked to provide HR departments with their goals
for the year, provide feedback on new company policies, or summarize the
information they learned in a training seminar. 

The real challenge for organizations (and researchers as well) is
figuring out how to use these observations to understand their employees and
affect meaningful changes just as they would had people provided survey data
(quantitative).  Grounded Theory
Method (GTM, Glaser & Strauss, 1967) provides one such procedure.  In brief, it involves a series of steps
in which the transcribed responses of individuals are reduced into increasingly
succinct bits of information until common, thematic keywords can be identified
that describe the nature of the phenomenon.

If this sounds like an onerous process, that’s because it is.  However, the most recent issue of Organizational Research Methods (2008,
Vol. 11, Iss. 3) presents an article by Fendt and Sachs that provides a number
of recommendations for flattening that learning curve a bit, including tips on
how to deal with nonverbal information, researcher bias, and making sense of
your data.  While these are
difficult concepts to describe without a better understanding of GTM, such
information is invaluable to researchers and practitioners hoping to use
observational data without getting swamped in their analytic efforts.

And that simplicity is something we can all appreciate.

Fendt, J., & Sachs, W. (2008). Grounded Theory Method in management
research: Users’
perspectives. Organizational Research
Methods, 11(3), 430-455.