Category Archives: Assessment

Team Intimacy and Organizational Interventions: Emphasizing Team Cohesion May be More Effective (IO Psychology)

Reply

Topic: Teams, Development
Publication: Human Resource Management Review (JUN 2012)
Article: Too Close for Comfort? Distinguishing Between Team Intimacy and Team Cohesion
Authors: Rosh, L., Offermann, L. R., & Van Diest, R.
Reviewed By: Thaddeus Rada

Within IO psychology, research on teams has become increasingly important in recent years. As organizations have begun to use teams for a wider variety of roles and purposes, it has become necessary for both researchers and practitioners to gain a better understanding of how teams work and how they can be designed to operate most effectively. Two constructs that have received research attention in the realm of teams include team intimacy and team cohesion. Although these constructs may appear to be very similar from the outside, Lisa Rosh and colleagues argue that there are important differences between these constructs, and that they are best conceptualized as distinct constructs.

Rosh and her colleagues explored the literature surrounding group intimacy and group cohesion, noting that, although there are areas of overlap between the constructs (e.g. interpersonal attraction), there are also key differences between them. Specifically, group intimacy necessarily requires some level of group cohesion, while a group may have high levels of cohesion without the added elements (e.g. interpersonal affection) of intimacy.

The authors suggest that, to date, many team-building initiatives in organizations have been designed to foster team intimacy, not cohesion. Noting that the link between intimacy and team performance has not been well-established, the authors suggest that practitioners shift their focus towards team-building interventions that focus, not on intimacy, but rather on the “work-focused” purpose of the group, such as the group’s commitment to their task and the task-based collective efficacy of the group. However, the authors do not discount the importance of intimacy entirely; indeed, they note that group intimacy is likely to become more common as teams take on more and more sophisticated projects in organizations; as such, they argue that additional research and examination of group intimacy is needed so that practitioners will be equipped to address this component of life in teams.

Rosh, L., Offermann, L. R., & Van Diest, R. (2012). Too close for comfort? Distinguishing between team intimacy and team cohesion. Human Resource Management Review, 22, 116-127.

human resource management, organizational industrial psychology, organizational management

 

source for picture: http://www.freedigitalphotos.net/images/Business_People_g201-Multiethnic_Team_p66538.html

Predicting someone’s propensity to morally disengage (IO Psychology)

Reply

Topic: Assessment, Personality, Ethics, Counter-Productive Work Behavior, Workplace Deviance
Publication: Personnel Psychology (SPRING 2012)
Article: Why employees do bad things: Moral disengagement and unethical organizational behavior
Authors: Celia Moore, James R. Detert, Linda Klebe Treviño, Vicki L. Baker, & David M. Mayer
Reviewed by: Alexandra Rechlin

Organizations obviously want their employees to be ethical. While there are existing measures that are used to predict who will act immorally, the authors of this paper proposed a new construct that they called an individual’s propensity to morally disengage – an individual difference in how people think about ethical decisions and behavior that allows them to act unethically without feeling bad about it.

Celia Moore and her colleagues developed a measure of an individual’s propensity to morally disengage. In a series of studies, they then validated the measure for working adults by showing that the propensity to morally disengage was positively related to unethical behavior after accounting for a number of other related traits, orientations, and emotions. Predicted outcomes included self-, supervisor-, and coworker-reported unethical behavior, decisions to commit fraud, and self-serving decisions in the workplace.

You may be wondering how this paper is relevant to practitioners. This new measure of the propensity to morally disengage predicts unethical behavior, and it is short – it only includes eight items. While it has yet to be validated for employee selection, this measure certainly shows promise for its ability to predict unethical behavior. The authors also found that this measure had a low correlation with social desirability, so it seems to be fairly resistant to test-takers faking their responses to receive a good score. If your organization is using a lengthy integrity test in the selection process for the sole purpose of predicting those who would conduct unethical behavior, then this new measure may be something your organization might want to consider using instead.

Moore, C., Detert, J. R., Treviño, L. K., Baker, V. L., & Mayer, D. M. (2012). Why employees do bad things: Moral disengagement and unethical organizational behavior. Personnel Psychology, 65, 1-48. doi: 10.1111/j.1744-6570.2011.01237.x

human resource management, organizational industrial psychology, organizational management

 

source for picture: http://www.freedigitalphotos.net/images/Business_People_g201-Business_Team_Thinking_p46925.html

Are cognitive ability tests insulting your applicants? (IO Psychology)

Reply

Topic: Organizational Justice, Fairness, Interviewing, Assessment, Selection
Publication: Personnel Psychology (WINTER 2011)
Article: Status and organizational entry: How organizational and individual status affect justice perceptions of hiring systems
Authors: Sumanth, J. J., & Cable, D. M.
Reviewed by: Alexandra Rechlin

It is well known in the field of IO psychology that cognitive ability tests are very predictive of employee performance.  However, applicants often see them as unfair and do not like taking them; more informal and much less valid methods (like informal interviews) tend to be preferred by applicants. In this study, Sumanth and Cable (2011) investigated the effect that the status of the organization and the career status of the applicant would have on applicants’ perceptions of the selection system’s fairness.

In this quasi-experimental study, the authors tested their hypotheses with two samples of adults (one of MBA alumni in the United States, one of executives in the United Kingdom). All participants were told that the hiring organization would use behavioral interviews, and half of the participants were told that they would also need to complete a cognitive ability test.

Organizational status is the reputation of an organization; a high-status organization (e.g., Google) is seen as being very reputable and prestigious. High-status organizations tend to be known for the rigor of their selection systems. The authors found that when low-status organizations (as opposed to high-status organizations) used cognitive ability tests as part of their selection system, applicants were less likely to view the organization attractively.

Individual career status refers to one’s beliefs about his or her career accomplishments and status (i.e., respect, prominence) within and outside of an organization. It is plausible that applicants with high status might be insulted by having to take a cognitive ability test when applying for a job; for example, they may think that their accomplishments should speak for themselves. Procedural justice is the fairness of the procedures used to make decisions. The authors found that when an individual’s status was high and cognitive ability tests were included, that person was more likely than low-status individuals to have lower perceptions of procedural justice. In other words, the process was seen as unfair. High-status individuals also were more insulted by the inclusion of a cognitive ability test.

These results indicate that even though cognitive ability tests are highly valid, they may repel high-status applicants who may see the process as unfair and insulting. My recommendation based on these findings would be to make it clear to applicants why cognitive ability tests are being used (i.e., their high validity) and that the exact same procedure will be used for all applicants.

Sumanth, J. J., & Cable, D. M. (2011). Status and organizational entry: How organizational and individual status affect justice perceptions of hiring systems. Personnel Psychology, 64, 963-1000. doi: 10.1111/j.1744-6570.2011.01233.x

human resource management, organizational industrial psychology, organizational management


source for picture: http://www.freedigitalphotos.net/images/Business_People_g201-Businessman_Holding_Hands_Up_p65923.html

 

The Peril of Excess: Why Moderate Levels of Many Traits Might be Best (Human Resource Management)

Reply

Topic: Assessment, Performance, Selection, Human Resource Management
Publication: Perspectives on Psychological Science (JAN 2011)
Article: Too Much of a Good Thing: The Challenge and Opportunity of the Inverted U
Authors: Grant, A. M., & Schwartz, B.
Reviewed By: Thaddeus Rada

A common assumption in personnel selection practice (and research) in IO psychology is that increasingly high levels of desirable traits are always a good thing. For instance, the Big 5 personality trait conscientiousness has been found to be a good predictor of job performance, such that highly-conscientiousness employees tend to be the best performers. As such, our selection systems are typically designed to identify applicants who are highest on these positive traits, so that they can be selected into the organization.

However, some authors have suggested that very high levels of some traits may actually produce “diminishing returns” and be detrimental to performance. This is the position taken up by Adam Grant and Barry Schwartz in a recent paper that evaluates this phenomenon. Expanding on arguments that can be traced back to Aristotle, the authors point out that for many traits and characteristics, extremely low or high levels of such qualities (such that courage becomes cowardice or recklessness, etc.) can be detrimental to performance and optimal functioning.

Although Grant and Schwartz do not focus specifically on organizations or personnel selection, it is clear that their main ideas have relevance to human resource management. These ideas point to the fact that we may want to consider the possibility of curvilinear relationships between traits and performance, such that the highest performers fall somewhere in the middle of the continuum from low to high possession of a trait.

Grant, A. M., & Schwartz, B. (2011). Too much of a good thing: The challenge and opportunity of the inverted U. Perspectives on Psychological Science, 6, 61-76.

human resource management,organizational industrial psychology, organizational management

Don’t Dump the Dimensions: A New Model for Evaluating Assessment Center Participants

Reply

Topic: Assessment
Publication: Personnel Psychology (SUMMER 2011)
Article: Exercises and Dimensions are the Currency of Assessment Centers
Authors: Hoffman, B. J., Melchers, K. G., Blair, C. A., Kleinmann, M., & Ladd, R. T.
Reviewed By: Thaddeus Rada

Assessment centers (ACs) remain a popular, and often effective, way for organizations to evaluate candidates, both in hiring and promotion settings. One choice that confronts users of assessment centers concerns the type of information that is gathered about candidates. A traditional practice with ACs has been to use multiple exercises to measure multiple job-relevant dimensions of candidate performance. However, some research has suggested that task-performance ratings are a more effective way to assess candidates. Some authors have even advocated for the abandonment of dimension ratings in AC practice.

Taking these differing perspectives into account, Brian Hoffman and colleagues conducted a study to identify a model that would effectively describe the structure of AC ratings. In their study, Hoffman et al. developed and tested a structure for assessment center ratings that included one general performance factor, as well as multiple broad performance dimensions and multiple exercise ratings.

The authors suggest that the results of their work provide a framework by which future ACs might be evaluated. They also argue that their results demonstrate the continued effectiveness of using dimensions in ACs, suggesting that it is unwise to discontinue their use.

Hoffman, B. J., Melchers, K. G., Blair, C. A., Kleinmann, M., & Ladd, R. T. (2011). Exercises and dimensions are the currency of assessment centers. Personnel Psychology, 64, 351-395.

 

Test Order Can Matter: A New Application of Order of Operations

1

Topic: Assessment
Publication: European Journal of Psychological Assessment (SPRING 2011)
Article: Context Effects on Test Performance: What about Test Order?
Authors: L. Khorramdel & M. Frebort
Reviewed By: Thaddeus Rada

Many people are familiar with the mathematical concept of order of operations. Sometimes shortened to the acronym PEMDAS, order of operations instructs us to solve in parentheses (P) before we deal with exponents (E), to multiply (M) before we divide (D), etc. The reason for working through math problems in this way is to lead us to the correct answer; if we solve the problem in any other order, we run the risk of coming up with an incorrect answer. It now appears that this same principle might apply to test administration. Previous studies have assessed how the order of items in a single test might impact individuals’ responses to the items. Now, a new study by Lale Khorramdel and Martina Frebort suggests that the order in which multiple tests in a testing battery are administered may impact individuals’ responses on the tests.

The current study examined these test order effects using two broad classes of tests: objective personality and cognitive ability. More specifically, Khorramdel & Frebort hypothesized that the administration of either test type as the first test would impact participants’ performance on the second test (e.g. working through the objective personality tests would have an impact on performance on a subsequently-administered cognitive ability test, and vice versa).

Using a sample of 66 members of upper management from an Austrian automotive supplier, the authors found support for their hypothesis that taking cognitive ability tests first impacted later performance on objective personality tests. However, they did not find support for their other hypothesis: that is, taking objective personality tests first did not significantly impact performance on cognitive ability tests.

While the somewhat small and specific nature of the sample used in this study might limit the generalizability of the study’s findings, the study still demonstrates that it is possible for test order to impact test performance. This knowledge leads to a somewhat obvious conclusion: the fact that test order can have an impact on test performance should be considered when implementing a testing battery. It may be the case that, if the impact of test order in a personnel selection or development scenario is unknown, it would be most prudent to follow the advice of the order of operations, and administer the battery in the same order to all participants, so that the order of the tests would not affect certain applicants differently than others. Or, the organization might take a more active approach and utilize an I/O psychologist to investigate possible test order effects!

Khorramdel, L. & Frebort, M. (2011). Context effects on test performance: What about test order? European Journal of Psychological Assessment, 27, 103-100.

Managing Assessors’ Workloads in Assessment Centers

Reply

Topic: Assessment, Staffing
Publication: International Journal of Selection and Assessment (SEP 2010)
Article: Do assessors have too much on their plates? The effects of simultaneously rating multiple assessment center candidates on rating quality
Authors: K.G. Melchers, M. Kleinmann, and M.A. Prinz
Reviewed By: Benjamin Granger

Assessment centers (ACs) usually consist of several job-related exercises that tap competencies necessary for the job.  ACs are most often used by organizations to select, promote and develop their employees.  Like many employee selection and assessment methods (e.g., interviews), ACs require a scorer or assessor to provide an evaluation of candidates’ performance.  But here’s where it gets tricky.

In two studies, Melchers et al. found that assessor ratings of candidates decrease in accuracy as the number of candidates they assess simultaneously increases.  While this finding may seem like a “no-brainer”, meta-analytic research (Woehr & Authur, 2003) has demonstrated that ACs with higher candidate-to-assessor ratios (i.e., fewer assessors evaluating more candidates) tend to be more valid than ACs with lower candidate-to-assessor ratios.  However, as Melchers et al. point out, the candidate-to-assessor ratio in an AC is not the issue here, its how many candidates an assessor must observe and evaluate simultaneously that apparently decreases rating accuracy.

Unfortunately, Melchers et al. do not provide specific recommendations for the ideal number of candidates assessors should be assigned to assess simultaneously in an AC exercise. 

Continue reading →

Using Simulations to Study, Assess, and Grow Managers

Reply

Topic: Assessment, Training

Publication: American Psychologist

Article: Developing Managerial Talent
Through Simulation

Authors: G. C. Thornton, J. N. Cleveland

Reviewed By: Rachel Marsh

 


User-group Simulations
are replications of essential parts of a job and have been utilized by
organizations for over 55 years. They are used to study, assess and develop
talent, especially managerial talent, and offer more information about
assessees than questionnaires.  Job simulations can range from being low
fidelity and very simple (e.g.,  asking
employees what they would do in certain situations, to very high fidelity and
quite complicated (e.g., behavioral simulations that include analyzing many
different aspects of company information). 

Standardized
simulations are often performed by trained assessment center personnel who can
quantify the actions of employees into usable data that is based upon the
performance of managers or managerial candidates who have gone through these
simulations in the past.  Because trained assessors perform the
simulations in a controlled setting, there is also more control over the
different variables the managers must deal with in the job simulation.  This allows the organization to make a
simulation very specific for each position or employee/applicant.     

The general
assumption regarding simulations as an assessment tool is that employees will behave
similarly in real work situations as they do in the simulation, thus helping
companies make better hiring/promotion decisions.  When used in a training
and development context, simulations help identify competencies that need
improvement so that a specific training plan can be developed for the employee,
saving the company time and money.   

In their
review article, Thornton and Cleveland suggest that in order to best utilize
simulations for training they should be used in conjunction with other training
methods such as lectures and demonstrations, and that employees should start
with the simplest methods or training, then move to more complex methods. 
Constant feedback should also be included in the training process to ensure
employees’ optimal learning from the training.  In order to best utilize
simulations for research and assessment, the authors suggest that several
different methods of simulations should be used, as this will ensure companies
get the most accurate employee information.      

Thornton,
G. C., & Cleveland, J.N.  (1990) Developing managerial talent through
simulation. American Psychologist, 45, 190-199.

Is What We Think We Know, What We Actually Know?

1

Topic: Assessment, Training

Publication: Academy of Management Learning
& Education (JUN 2010)

Article: Self-assessment of knowledge: A
cognitive learning or affective measure?

Authors: T. Sitzmann, K.E. Ely, K.G. Brown
and K.N. Bauer

Reviewed By: Benjamin Granger

 


Self_assessment Evaluating the effectiveness of an organizational
training program is a necessary but expensive process.  Oftentimes, the success of a training
program is evaluated by how much trainees learn or how much they know after
completing the program.  The classic
post-training test/exam is a great way to do this.  But, because developing and administering well-constructed
learning measures can be costly, one option is to simply ask trainees how much
they have learned.  But how “good”
are trainees’ self-assessments of their learning/knowledge?  That is, how well do self-assessments really measure actual learning/knowledge
gain?

In an extensive meta-analysis that
included a total of over 41,000 study participants from 166 studies, Sitzmann
et al. (2010) found that self-assessments of knowledge and learning are
actually more highly related to motivation and satisfaction with training than with
actual knowledge/learning.  In
fact, even in training programs that provide trainees with feedback and give
trainees access to information about their learning/knowledge, self-assessments
of learning are more strongly related to motivation and satisfaction than actual
learning.  

Additionally, Sitzmann et al. found
that self-assessments of one’s current knowledge level (How much do I know) are
much more strongly related to actual learning than self-assessments of
knowledge gain (how much have I learned). 
In fact, self-assessments of knowledge gain were unrelated to actual learning.


Yet another interesting finding is
that self-assessments of learning are more accurate when training is classroom
based v. web-based. Sitzmann et al. suggest that classroom-based and blended
training allow for trainees to observe others and thus gauge their knowledge or
mastery of the topic being trained. 
Such information is not as readily available in online training.

Though self-assessments of learning
are cheap to develop and generally easy to administer, they are not appropriate
alternatives to tried and true measures of knowledge gain (e.g., written tests,
skill assessments).  Decisions made
about training programs (e.g., “It’s a success!”) that are based on trainees’
self-assessments of their own learning may be misinformed, because what
trainees think they know, may not be what they actually know!

 

Sitzmann,
T., Ely, K.E., Brown, K.G., & Bauer, K.N. (2010). Self-assessment of
knowledge: A cognitive learning or affective measure? Academy of Management
Learning & Education, 9(2), 169-191.

Assessment centers for personnel selection: strengths, inconsistencies, and best practices

Reply

Topic: Assessment

Publication: Human Resource Management Review (September,
2009)

Article: Validity of assessment centers
for personnel selection

Authors: Thornton, G. C., & Gibbons, A.
M.

Reviewed By: Bobby Bullock

 


Images Assessment
centers (ACs) have
been used to aid the process of external and internal selection of employees
and high potentials, certification, and promotion for over 50 years.  At ACs, multiple assessors observe the
behavior of assesses as they engage in organizational simulations designed to
test their ability to perform new, relevant assignments.  Historically, overall assessment
ratings (OARs) and
dimensional scores have been shown to predict a range of relevant selection
outcomes.   

Compared
with other selection tools (such as cognitive ability or personality), ACs are
stronger predictors of future effectiveness.  OARs
have been found to predict promotion rates, salary progress, job performance
ratings, performance in training, and other performance criteria.  ACs exhibit less group bias (in terms of sub-group differences)
compared to cognitive ability test scores, but their fairness should not be
blindly assumed.  Also, although
the use of an AC can be fairly expensive, their overall utility is positive.  In fact, some studies have found the value of using an AC
can range from thousands of dollars per hire to over 1 million overall! 

Although
ACs have been found to be generally effective and worthwhile, research over the
past few decades has highlighted various controversies and conflicting
recommendations.  Thornton and
Gibbons (2009) have suggested that more research needs to be undertaken in
order to shed some light on various issues such as:  which type of dimensions to assess (i.e., cognitive,
interpersonal, etc.), the use of dimension-based ACs vs. exercise-based ACs
(which they believe depends on the target position), participant impression
management and faking, and how to integrate ratings from multiple raters and
exercises.  However, the authors
noted that certain best practices can improve the AC process.  These include:  

·     
         Highly
trained assessors (using psychologists is particularly effective)

·     
         Clear
and easy to use assessment tools

·     
         Use of
exercises identified through thorough job analysis to be relevant to the                   target job

·     
       Proper
thought given to:  the participant
rating system, insight into the variance of          possible responses to exercises,
and even the very instructions given to                              participants

Not all
hope is lost!  The authors
summarized their findings best by stating that “research and practice suggest
that ACs are valid, fair, legally defensible, and acceptable to candidates and
other stakeholders in a wide variety of jobs” (Thornton & Gibbons, 2009, p. 183).

Thornton, G. C., & Gibbons, A. M. (2009).
Validity of assessment centers for personnel selection. Human Resource
Management Review, 19(3), 169-187.