Category Archives: Measurement

Predicting Job Performance with Implicit Words Games?

3

Topic: Personality, Measurement, Job Performance
Publication: Personnel Psychology (SPRING 2010)
Article: We (sometimes) know not how we feel: Predicting job performance with an implicit measure of trait affectivity
Authors: R.E. Johnson, A.L. Tolentino, O.B., Rodopman, and E. Cho
Reviewed By: Benjamin Granger

In the world of emotions, trait affect refers to the predisposition some people have to generally experience positive or negative emotions.

Trait affect is often broken up into Negative Affect (NA) and Positive Affect (PA). While high levels of NA are associated with negative emotions such as fear and anxiety, high levels of PA are associated with positive emotions such as excitement and joy.  It should not come as a surprise that PA tends to relate favorably to work performance whereas the opposite is true for NA.

Recently, Johnson, Tolentino, Rodopman, and Cho (2010) suggested that because trait affect (e.g., PA & NA) operates outside of employees’ conscious awareness, it is more appropriate to measure it at the unconscious or implicit level.  This is in stark contrast to the self-report, explicit measurement of trait affect that is typically used when explicitly asking people to rate the extent to which they feel certain emotions across many different situations.

But how in the heck would you measure trait affect implicitly?  Johnson et al. used a word completion task that presented word fragments to employees for which they were required to complete to create a meaningful English word.  The following are actual examples of word fragments used by Johnson and colleagues:

F E _ _ (NA = FEAR, or neutral = FEEL, FEED) S M _ _ _ (PA = SMILE, or neutral = SMART, SMOKE)

A person’s level of trait NA and PA were determined by the relative amount of NA-related and PA-related word fragments completed by employees, respectively.  But, don’t worry if you are a bit skeptical; this is not exactly your everyday personnel survey!

Nevertheless, Johnson and colleagues conducted two independent pilot studies that supported the validity of their word fragment approach. Ultimately, Johnson and colleagues demonstrated that implicit measures of trait affect are important predictors of task performance, organizational citizenship behaviors (OCBs) and counterproductive work behaviors (CWBs), even more so than the conscious/explicit measures that we are more accustomed to. Johnson et al.’s study highlights an interesting way to measure employees’ predispositions to experience positive and negative emotions.

Moreover, while employees can easily misrepresent themselves on explicit personality measures, this is likely not possible for implicit measures.

Johnson, R.E., Tolentino, A.L., Rodopman, O.B., & Cho, E. (2010). We (sometimes) know not how we feel: Predicting job performance with an implicit measure of trait affectivity. Personnel Psychology, 63 (1), 197-219.

 

Making the Most Out of Multiple-Choice Testing

4

Topic: Measurement
Publication: International Journal of Selection and Assessment
Article: On minimizing guessing effects on multiple-choice items: Superiority of a two solutions and three distractors item format to a one solution and five distractors item format
Authors: K.D. Kubinger, S. Holocher-Ertl, M. Reif, C. Hohensinn, and M. Frebort
Reviewed By: Benjamin Granger

In addition to being popular among test takers, the multiple- choice format test is nearly ubiquitous in employee selection and assessment contexts and offers many advantages (e.g., easily quantified, easily scored, etc.) to organizations.

The most common multiple-choice item format includes a single correct answer and several (perhaps 3 or 4) wrong answers or “distractors.” But, this format leaves the door open to what we may call the “guessing effect.” The basic idea is that, in theory, a person with absolutely no knowledge of the content area can endorse some items correctly by luck (i.e., guessing correctly). In fact, many standardized multiple-choice tests have instruction books that discuss guessing strategies (e.g., ACT, GRE).

Acknowledging the utility of the format itself, Kubinger and colleagues (2010) explored a multiple-choice format with two correct answers as opposed to the single correct (or best) answer that is most commonly used. In order to correctly answer such an item, test takers must endorse BOTH correct answers and cannot endorse any of the distractors. Needless to say, this manipulation makes multiple-choice items substantially more difficult which is indeed what the authors found. In fact, the difficulty of this format was comparable to that of a free response format test of the same content (i.e., math).

However, compared to the traditional multiple-choice format with a single correct answer and five distractors, the two correct answer format drastically reduced the “guessing effect.”

Kubinger et al.’s study presents an interesting alternative to the multiple-choice response formats that we are accustomed to. Although they are significantly more difficult, items that require recognition of two correct answers among three distractors can dramatically reduce the occurrence of lucky guesses that can potentially impact important employment decisions.

Kubinger, K.D., Holocher-Ertl, S., Reif, M., Hohensinn, C., & Frebort, M. (2010). On minimizing guessing effects on multiple-choice items: Superiority of a two solutions and three distractors item format to a one solution and five distractors item format. International Journal of Selection and Assessment, 18(1), 111-115.  

The Muddy Waters of Measuring Executive Coaching

1

Topic: Coaching, Measurement, Training
Publication: Consulting Psychology Journal (JUN 2009)
Article: Measuring and Maximizing the Business Impact of Executive Coaching
Author: A. Levenson
Reviewed by: Lit Digger

Given the amount of money organizations invest in executive coaching programs, it would be refreshing if someone could come up with a reliable and fool-proof way to measure their effectiveness.

Organizations are complex entities, so developing a measurement tool like this would be a notable challenge.  Levenson (2009) explored a dozen coach-coachee pairs to contribute to this ongoing conversation and shed some light on this measurement puzzle.  Given the constraints of the study, Levenson cautioned that we should interpret his findings lightly.

To recap, studies already exist measuring coaching’s effect on:

·   The executive’s actual changes in behavior

·   The degree to which those around the executive perceive increased effectiveness of the executive

·   Changes in what Levenson calls “hard” performance measures (e.g., unit productivity, number of tasks completed, ability to meet goals, etc.)

But how can we measure business impact of executive coaching? Levenson suggests that we should “start with the organization’s strategy” (p.110). He recommends that we should determine whether the business impact we care to measure most is strategic or financial. For example, if a company has a strategic aim to increase sales to a certain demographic group, then the outcome should be designed to target that strategy – not a more distal, less-related financial goal.

Levenson also warns that we should consider the complexity of the executive’s job in relationship to the functioning of the organization.  Take the above sales example for instance. If the executive’s primary role is to make decisions and cultivate a productive working environment, then he/she may not actually have all that much impact on increasing sales to the target demographic group.  It would be difficult to evaluate the business impact of coaching if the executive’s role has little business impact to begin with.

Levenson reminds us that if other needed training programs or selection systems are being implemented around the time that executive coaching takes place, then you will be much more likely to see organizational changes in the direction desired. Systemic changes often will have more business impact than executive coaching alone.

Finally, is executive coaching always the answer to our organizational problems? No! Levenson cautions that the intervention needed will depend on the issue at hand. An executive might be better off gaining critical skills from a stretch assignment if the key issue is professional development. Or if team performance is slacking, perhaps a team building activity would be best.

You’re more likely to see bang for your buck if the interventions you select are targeted appropriately. Now we just need to figure out how to effectively measure the “bang”.

Levenson, A. (2009). Measuring and maximizing the business impact of executive coaching. Consulting Psychology Journal, 61 (2), 103-121.

Book Review

Reply

Topic: Book Reviews, Strategic HR, Measurement
Book Title: Investing in what matters: linking employees to business outcomes
Authors: Scott Mondore, Ph.D. and Shane Douthitt, Ph.D.


In SHRM’s recently published the book, “Investing in What Matters,” Scott Mondore, Ph.D. and Shane Douthitt, Ph.D. offer a process to understand the links between HR strategy  and business outcomes.  Below is Dr. Mondore’s overview of the book.

Organizations collect vast amounts of data from operations to people, but rarely do organizations bring this data together to discover how these data relate to each other. In addition, current economic conditions are demanding deep budget cuts—leaving HR departments with few tools to figure out where to cut and where to invest. “Investing in What Matters” provides HR leaders with a straightforward process of six steps, that they can immediately implement, to allow them to create an HR strategy that is business-focused and based on expected ROI. With these steps, HR leaders will learn how to discover key business outcomes, show the link between HR data (training, surveys, competencies etc) to those business outcomes, execute programs that have an expected ROI and create a culture of measurement, analysis and adjustment going forward.

The six steps in the Business Partner RoadMap process are:

1. Determine Critical Outcomes (conduct stakeholder interviews with senior/functional leaders, examine the organization’s scorecards)

2. Create a Cross-Functional Data Team (bring together data owners from across the
organization to set up the analyses)

3. Assess Outcomes Measures (make sure the data being looked at is measured at the
same frequency and level within the organization)

4. Analyze Data (use advanced statistical to show cause-and-effect relationships between
HR data and business outcomes)

5. Build Programs & Execute (create initiatives around the drivers of business outcomes—
based on expected ROI calculations)

6. Measure and Adjust (re-analyze data on a regular basis to discover new drivers of
business outcomes or tweak current programs)

In addition, the book provides ten key principles for HR leaders to adopt during this process as it is not always easy and it needs to stay completely focused on business outcomes:

1. Organizations already spend significant amounts of money on their people….they just
don’t spend it on the right things.

2. Organizations make investments in people without any data or with the wrong data.

3. Employee engagement in itself is not a business outcome.

4. People and organizations are complex. The linkages between attitudes and outcomes
have to be understood within your organization using your data.

5. The people data and outcome data do exist—you just have to go and get it.

6. The organization’s data exist in silos.

7. There will be obstacles and barriers to obtaining the data (e.g. politics, turf battles).

8. Once a connection/linkage is made with the data—accountability is unavoidable (and
that’s a good thing).

9. Don’t assume a link between employee data and business outcomes—define it and
understand why or why not.

10. Perceptions alone do not show up on the profit and loss statement.

Mondore, S.P. & Douthitt, S.S. (2009). Investing in what matters: Linking employees to business outcomes. Society for Human Resource Management, Alexandria, Virginia.

Click to learn more: http://shrm.org/Publications/Books/Pages/InvestinginWhatMatters.aspx

Internet-based Data Collection: Just Do It Already!

Reply

Topic: Measurement, Statistics
Publication: Computers in Human Behavior
Article: From paper to pixels: A comparison of paper and computer formats in psychological assessment.
Author: M.J. Naus, L.M. Phillipp, M.Samsi
Featured by: Benjamin Granger

Internet Although many organizations
have jumped onto the internet-data collection bandwagon, several issues still
need to be addressed.  For example,
are paper-pencil and internet-based tests of the same trait (e.g., personality
questionnaire) or ability (e.g., cognitive ability test) really equivalent?  Similarly, are there any reasons to
believe that employees respond to internet-based tests differently than they
would a paper-pencil test of the same trait or ability?

Naus, Philipp, and Samsi
(2008) set out to investigate these questions using three commonly used
psychological scales (Beck Depression Inventory, Short Form Health Survey, and
the Neo-Five Factor Inventory).

Although Naus et. al found
that the paper-pencil and internet-based survey formats performed equivalently
for the Beck Depression Inventory and the Short Form Health Survey, there were
differences for Neo-Five Factor Inventory (a commonly used personality
assessment tool).  What’s going on
here?

One possibility is that
responses were more socially desirable for the paper-pencil format, since a
researcher was present at the time.  That is, in the presence of an authority figure (i.e.,
researcher) participants may have responded in order to appear more self-controlled
and self-focused.  This is likely
much less of a concern when completing the same survey on a computer at home
(in PJs!).

Overall, respondents
perceived the internet-based format to be convenient, user-friendly,
comfortable and secure (All great things!).

So what can we conclude
about these findings?  Although
internet-based data collection methods have some advantages over paper-pencil
methods, there are some caveats to their use.  In some cases, the tests may operate differently due to the
particular format.  Unfortunately,
not much is known about how they might differ.  However, Naus et al.’s findings suggest internet-based
methods receive good reactions from employees and can save an organization time
and money!

Is interrater correlation really a proper measurement of reliability?

Reply

Topic: Measurement, Research Methodology, Statistics
Publication: Human Performance           
Article: Exploring the relationship between interrater correlations and validity of peer ratings
Blogger: Rob Stilson

Is inter Interrater reliability (still with me?, Ok good) is often
used as the main reliability estimate for the correction of validity
coefficients when the criterion is job performance. Issues arise with this
practice when one considers that the errors present between raters may not be
random, but due to bias, while agreement between raters may also stem from bias
instead of actual consistency. In this study, the authors’ main goal was to
explore the relationship between interrater correlations and validity and also
to explore the relationship between the number of raters and validity.

In order to do this, the authors gathered information from
3072 Israeli policemen from 281 work teams who took part in peer rating. The
average size of each of these work teams averaged about 12 people and ranged
from 5 all the way to 33. The measure used was overall performance (on a
7-point Likert scale). The predictor employed in this study was the ICC (C,k)
model, which is equivalent to Cronbach’s alpha. Measurement indices were
computed on the team level as rating only took place within work teams.

The predicted variable for the study was the validity
coefficient for each work team. This is the part of the study where you could
really feel the sweat involved. Here the authors gathered information on
supervisor evaluations, absenteeism data, and discipline data collected over
several years (for over 3000 policemen)! The authors then converted this
information into z scores with higher
scores indicating better performance.

Results showed a weak positive linear relationship between
interrater correlations and the various validity indexes. This is not what you
want to hear if you are doing peer rated performance evaluations. The authors’
stipulate that the correlation between raters is a conglomeration of factors
having different theoretical relationships with validity (i.e. bias and other
idiosyncrasies).

Practical implications from the information gleaned here
include the adjustment of validity due to attenuation. If the measurements used
in the calculation included non random error estimates, the ensuing
calculations will be off. A positive finding for the work world was validity in
small units (less than 10 people) was about the same as those for larger units.
The authors’ believe this finding may be due to observation opportunity level,
which is seemingly greater in smaller work units.

Kasten,
R., and Nevo, B. (2008) Exploring the relationship between interrater
correlations and validity of peer ratings. Human Performance, 21(2), 180-197.