Category Archives: Assessment
When Reading Research Leads to a Brain Full of “What?!” (IO Psychology)
When you read scientific research, you should be left feeling as though you gained knowledge and/or have something new and shiny that can be applied to the real world. But once in a while you finish an article and there is nothing but unpoppable “What did I just read?!” bubbles floating in your brain.
This article focused on how applicants’ personality types might impact their reactions to assessment tests within a hiring process. Specifically, candidates for firefighter, dispatcher, and rescue management roles had to complete a series of personality and cognitive assessments as a part of the selection process. Immediately after, they were asked to complete a voluntary survey asking about their reactions to the tests. The researchers found that personality types had no impact on applicants’ perceptions that the assessments were related to the job and that the tests could predict future job performance. One personality type did perceive the tests as less fair than those with other personality types, but the difference may not have been large enough to have real meaning.
As I kept reading the article, I kept wondering how this information would be applied, or even how it would be useful. I kept wondering this because the authors never told me. The authors briefly mention previous research stating that applicant reactions can impact whether or not a candidate might accept a job offer and/or impact their future performance on the job. Yet they never relate their own findings to this previous research. I was left hanging.
The study also had a number of confounds, a few of which the authors acknowledged. Looking solely at rescue applicants isn’t representative of most jobs and applicants. Candidates had to first pass a physical test before they were allowed to begin the personality and cognitive assessments. The reactions survey only asked for their reactions to the personality and cognitive tests, but wouldn’t their perceptions of the physical test muck up their thoughts a bit?
Also, participants voluntarily completed the reactions survey, and not everyone completed it. Wouldn’t the thoughts of those who did NOT want to share their reactions be critical? Finally, their research found different reactions to the assessments based on gender and age, but they never investigated further, which I found disappointing.
Now I have to be fair and say that no research is perfect. All research has confounds. But when you feel as though you don’t get the “so what?” of the entire study and there are also lots of confounds, how are you supposed to react?
After reading this article I was left feeling a little icky inside. But it reminded me that reading research with a discerning amount of skepticism is not only healthy, it is mandatory. It reminded me of a wonderful quote by the philosopher George Santayana: “Skepticism, like chastity, should not be relinquished too readily.”
Random Thoughts While Taking a Personality Test (IO Psychology)
Have you ever taken a personality test as part of a hiring process and found your mind wandering? Where did it go? Did you wonder how to best answer the questions to secure the job? Or did you ponder why puppies are just so damn cute? Or why the Flock of Seagulls hairstyle never took off?
In this study, researchers interviewed test takers to learn what they were thinking while taking a personality test. I was excited about this, because in industrial psychology we tend to fixate on how well personality tests succeed in identifying rockstar candidates, but we should also think about how candidates react when completing them. Here are a few of the findings:
- Some test-takers had a difficult time answering the questions, as no context was provided. For example, if they had to rate how likely they are to “start a conversation with a stranger” (with no other context), they felt as though their answer would differ if asked in a work setting versus their personal life. I can see this one.
- A few didn’t feel as though they knew themselves well enough to answer the questions. This one really threw me; I don’t know about you, but I’m not sure I would want to hire someone who lacks enough self-awareness to answer questions on a basic personality test.
- A number of individuals answered the questions in a way to make them look more favorable, to help secure the job in question. Others tried to answer as honestly as possible, thinking the test would catch them and figure out their devious plan. I see both reactions here; some assessment tests can, in fact, detect if someone is trying to cast themselves in a more favorable light; other tests do not. This is one of the great mysteries when taking an assessment test…Will the stats police uncover my lies?
- Many test-takers tried to figure out how their answers would be analyzed and interpreted. We all have this natural curiosity, but voodoo-magician test publishers keep this secret close. (Cueing “something D-O-O economics…”)
- Most individuals noticed questions that seemed to be similar and/or repeating, but assumed this was a way to detect how consistently they answered the questions. This is true; it helps test creators measure the consistency of your answers, see if you are paying attention whatsoever, or reveal if you are just having your cat select the answers for you (we all know how much cats love to sprawl out on keyboards).
So the moral of the story is: our reactions to personality tests are as different as our personalities themselves, which might mean that I am the only person out there that wants the Flock of Seagulls hairstyle to make a comeback.
How Should I Measure That?
Topic: Selection, Assessment
Publication: Human Performance (2009)
Article: Not much more than g? An examination of the impact of intelligence on NFL performance
Authors: B.D. Lyons, B.J. Hoffman, & J.W. Michel
Reviewed By: Scott Charles Sitrin, M.A.
In most work, intelligence is a key a predictor of job performance. But what about when your job involves physically assaulting your opponent and not letting him say “uncle” until you have successfully moved a pigskin 100 yards into his end zone? For a football player, does intelligence predict performance? Do you, as an owner of a National Football team, select the genius in the tweed jacket with an Ivy League smile?
In investigating the relationship between intelligence and performance, Lyons, Hoffman, and Michel examined the general mental ability and performance of 762 football players drafted during the 2002, 2003, and 2004 NFL Drafts. Since there are relatively few kickers and punters, they were excluded. Cognitive ability was evaluated using the Wonderlic a performance was assessed three ways: future NFL performance, including position specific criterion such as total tackles and touchdowns over a three-year period; draft position, referring to when the player was selected by a professional team in the NFL Draft; and number of games started. Shockingly enough, cognitive ability did not relate to performance.
So, general mental ability does not appear to predict performance of professional players. That’s not to say that intelligence is not related to the performance of professional football players. (Confused yet?) Rather, this study shows that intelligence as measured by the Wonderlic does not relate to football performance. It is possible that intelligence as measured by another instrument could show a relationship to athletic performance. As a take home message, be thoughtful in selecting the tool that you use to measure something. The Wonderlic may be a helpful measure of intelligence in business settings, but its use in alternative contexts such as the world of the NFL may not be as valid.
human resource management, organizational industrial psychology, organizational management
Size Matters in Court? Determinations of Adverse Impact Based on Organization Size (IO Psychology)
Topic: Assessment, Discrimination, HR Policy, Statistics
Publication: Journal of Business Psychology (JUN 2012)
Article: Unintended consequences of EEO enforcement policies: Being big is worse than being bad
Authors: R. Jacobs, K. Murphy, and J. Silva
Reviewed By: Megan Leasher
Adverse impact occurs when neutral-appearing employment practices have an unintentional, discriminatory effect on a protected group. The Equal Employment Opportunity Commission is charged with enforcing all federal legislation related to employment discrimination and adheres to the 1978 Uniform Guidelines on Employee Selection Procedures for “rules of thumb” on inferring whether adverse impact is present.
However, it’s tricky for a plaintiff to present conclusive evidence that adverse impact is present in an organization’s practices. Statistical evidence is needed to demonstrate whether employment practices are truly discriminating against a protected group. Jacobs and colleagues investigate a common statistical method, known as “significance testing,” which is often used in courts to demonstrate evidence of adverse impact. Significance testing compares the difference between the proportion of majority candidates selected and the proportion of protected class candidates selected in an employment decision. If the test finds the difference between these proportions to be “statistically significant,” courts generally interpret this to mean that adverse impact is present.
This method seems to make sense from a high level, but problems arise when you look under the surface. The outcome of significance testing is greatly influenced by the number of people who are included in the analysis. Specifically, the more people you include in a significance test, the greater the likelihood of finding a statistically significant difference between groups. So, if you have an organization with many people included in the analysis, you are much more likely to yield a significant difference between majority and protected groups than you would with a smaller organization with fewer people to include in the exact same analysis. This is the primary argument of the authors; why do courts use significance testing to demonstrate adverse impact when, by nature of the test, the results would almost always find that big organizations are discriminating and smaller ones are not?
The authors conducted a series of studies to determine sources of differences in adverse impact significance testing. They found that the number of people included in the analysis was the strongest predictor in whether or not a statistically significant difference was found between groups. Size accounted for 49% of the final outcome of the analysis, which was almost five times greater than what any other factor (e.g., score differences on assessment in question, proportion in each group selected, etc.) accounted for. They also discovered an interesting threshold: When an adverse impact significance test is conducted with 500 or more people in the analysis, very small differences between the groups’ selection proportions will be statistically significant (yet below 500 these same comparisons would not be significantly different).
These findings support the powerful impact of sample size on determinations of adverse impact via significance testing, but they do not tell us if members of majority and protected class groups are really experiencing systemic, differential outcomes in employment practices. Unless a statistical method can accurately assess the latter issue, it is meaningless. This oversimplification leads us to believe that virtually all larger organizations are guilty of discrimination and virtually all smaller organizations are not. This common practice in courts only serves to make small organizations feel impervious and invincible and leave large organizations running in fear.
The authors close by asserting that regulatory standards should always reflect current scientific knowledge, yet the Uniform Guidelines on Employee Selection Procedures still reflect the science of decades past. They advocate for not only alternative methods to more appropriately measure adverse impact, but also for a more dynamic definition of adverse impact; one that considers multiple, interactive factors before a determination can be made. Current practice is supporting the message that to be big is to be bad and to be small is to be nice, which goes directly against the spirit of anti-discrimination legislation.
human resource management, organizational industrial psychology, organizational management
source for picture: http://www.freedigitalphotos.net/images/Leadership_g403-Team_Leader_p19976.html
What are those emerging markets thinking? (IO Psychology)
Topic: Selection, Assessment
Publication: International Journal of Selection and Assessment (JUN 2012)
Article: Cross-cultural Examination of Applicant Reactions to Selection Methods: United States and Vietnam
Authors: Hoang, T.G., Truxillo, D.M., Erdogan, B., and Bauer, T.N.
Reviewer: Neil Morelli
Over the past several years I-O psychologists have become more interested in understanding applicant reactions to selection tools. Of course we still care about the reliability and validity of the selection tool, but we know that how an applicant reacts to the process could influence how fair or trusting the company is perceived to be, how well the newly hired employee adjusts to the job, or how vulnerable the selection tools are to legal challenge. And, as organizations become more global and enter emerging markets, it’s important to understand what the reactions of people in these new, unexamined candidate pools may be.
A good example of an emerging market is Vietnam—economic reports show that companies are moving operations to Vietnam in greater numbers and Vietnam’s economy is expected to become the 17th largest by 2025. To understand the potential reactions to selection methods in a country with very different cultural, legal, and socioeconomic conditions, the authors compared the reactions of 225 Vietnamese business students to 151 American business students. Examples of the selection methods included interviews, work samples, resumes, biodata, and graphology, while the specific reactions were process favorability and fairness (i.e.,
face validity, personal privacy, and legality).
Hoang et al. discovered that perceptions in either country were fairly similar to each other: interviews and work samples were viewed more positively than biodata, cognitive ability tests, and personality tests, which were viewed more positively than honesty tests and graphology. However, the finding that Americans viewed personal contacts as more positive than Vietnamese, while Vietnamese found written ability tests more favorable than Americans reveals that cultural differences do exist. Another important finding was that the perceived face validity and legality of the selection method were the strongest fairness predictors of process favorability—apparently no matter where you go the test should still look and feel appropriate to the job.
Overall, organizations should be mindful of the culture they’re moving to, but using more favorable methods while making sure they are as face valid as possible is a best practice for any organization looking to select candidates overseas.
Hoang, T.G., Truxillo, D.M., Erdogan, B., & Bauer, T.N.. (2012). Cross-cultural
examination of applicant reactions to selection methods: United States and Vietnam.
International Journal of Selection and Assessment, 20(2), 209-219.
human resource management, organizational industrial psychology, organizational management
Unproctored Testing: Increasing the Applicant Pool to Combat Concerns with Cheating (IO Psychology)
Topic: Selection, Evidence Based Management, Assessment
Publication: International Journal of Selection and Assessment (JUN 2012)
Article: Offsetting Performance Losses Due to Cheating in Unproctored Internet-Based Testing by Increasing the Applicant Pool
Authors: Richard N. Landers & Paul R. Sackett
Reviewed By: Thaddeus Rada
Unproctored Internet testing (UIT) has been a hot topic in IO psychology over the past several years. In a nutshell, UIT allows organizations to post some of their selection tests online, allowing applicants to access them from virtually anywhere, so they can complete them on their own time. Some of the research on UIT has confirmed its strengths, such as its accessibility and efficiency, but other research has highlighted some of its limitations; in particular, there remains widespread concern about cheating in UIT. Because UIT is unproctored (it’s right there in the name), applicants are not under any supervision when they take such tests, so it’s possible that cheating could occur in a wide variety of ways.
While many authors have debated the extent to which cheating in UIT is a problem (e.g., how prevalent it is, how to cope with it, etc.), Landers and Sackett argue that if UIT increases the size of the applicant pool, even if some of these individuals do cheat, the organization may still derive benefit from a UIT program. Their logic goes something like this: if an organization is only looking to hire a set number of people (e.g., 50), then increasing the size of the applicant pool allows the organization to increase the cut score that an individual needs to exceed in order to be hired, thus making it harder for cheaters to get job offers. To test their ideas, Landers and Sackett conducted a computer simulation, which confirmed their belief in the power of a large applicant pool to compensate for some of UIT’s limitations.
While it does not remove all the concerns that exist about UIT, Landers and Sackett’s study does demonstrate that some of UIT’s benefits outweigh its limitations. This study’s findings hinge, to a large extent, on the capacity of a UIT intervention to increase the size of the applicant pool. As such, the authors suggest that practitioners carefully assess whether or not they anticipate that this will be the case before recommending that an organization adopt UIT.
human resource management, organizational industrial psychology, organizational management
A Sequel to the Ring of Fire: How Internal and External Candidates React to Employment Testing (I/O Psychology)
Topic: Assessment, Personality Assessment, Selection
Publication: International Journal of Selection and Assessment (JUN 2012)
Article: Don’t you know me well enough yet? Comparing reactions of internal and external candidates to employment testing
Authors: G. W. Giumetti and E. F. Sinar
Reviewed By: Megan Leasher
Employment testing is gaining in popularity at all levels within organizations, leading internal candidates to complete assessment tests to be considered for promotion or lateral moves. When you test employees you already hired, you might expect some pushback!
When competing for the same job, do internal and external candidates react differently to employment tests? Gary Giumetti and Evan Sinar explored this question with over 2300 candidates across 12 organizations and found several key differences. As compared to external candidates, internal candidates reported lower perceptions regarding information they received about what to expect in the tests, but held more positive views on both the job- relatedness of the tests and their overall recommendation of the organization to others.
This study focused on candidates’ perceptions of the tests themselves immediately after completing the tests. But what if you asked them later on? As a practitioner, the article led me to a spiraling of even more questions; all of which could impact applicant reactions at various points in the hiring process. How were the scores on the tests used? Was there a hard cutoff that required a “passing” score (on one or more of the tests) to advance in the hiring process? If so, were exceptions ever made because of something else amazing in a candidate’s background? What were candidates told about the hiring process and how their test scores were used? Did any of the internal candidates take the same or similar tests when they vied for
their current job? What had internal candidates heard, if anything, about the ease or difficulty of the tests from their coworkers? Were interviewers made aware of their test scores?
When it comes to any hiring process, consistency in both process and communication is paramount, no matter who comprises the applicant pool. An organization can choose to be transparent, opaque, or somewhere in the middle in what they share about a hiring process to candidates. But they need to be consistent to ensure that all applicant pools are treated justly.
human resource management, organizational industrial psychology, organizational management
Scoring Biodata Measures (IO Psychology)
Topic: Selection, Assessment
Publication: Personnel Psychology (SUMMER 2012)
Article: Unlocking the key to biodata scoring: A comparison of empirical, rational, and hybrid approaches at different sample sizes
Authors: J. M. Cucina, P. M. Caputo, H. F. Thibodeaux, & C. N. Maclane
Reviewed by: Alexandra Rechlin
Cucina and his colleagues recently conducted a study to explore the best method of scoring biographical data (biodata) measures. Biodata can be scored using empirical keying methods in which the assessor weights item responses based on the objective relationship between the item and performance; rational keying, in which scorers come up with subjective estimates of that relationship based on theory; and a hybrid approach, in which assessors use both rational and empirical keying). In this study, the authors answered seven research questions. We’ve provided you with the questions and answers for your own biodata-scoring enjoyment.
1. Which scoring approach (i.e., rational, empirical, or hybrid) yields the highest criterion-related validity?
With sample sizes of 1600 cases or less, empirical keying with unit weights has the highest validities. With smaller sample sizes, hybrid keying with unit weights resulted in less variability.
2. Do the different empirical keying procedures (e.g., vertical percent, point biserial, mean criterion, etc.) have different criterion-related validities?
Not really – they’re quite similar.
3. Which biodata scoring procedures should practitioners use (considering factors such as validity, feasibility, and legal defensibility)?
The different methods are very similar, so you could use any of them. The simplest method is the point biserial raw weights method, which you can do in many statistics programs.
4. Do the different biodata scoring procedures yield similar (i.e., highly correlated) scores?
Yes, very.
5. Does sample size impact the validity of the different biodata scoring procedures?
Yes. For the empirical and hybrid keying approaches with unit weighting of items, the relationship is strongly positive up to about 500 cases, but then it levels off. When items are weighted with stepwise regression, the relationship is positive but diminishes after about 1600 cases. With rational keying using unit weights, sample size didn’t really affect validity.
6. What are the sample size requirements for empirical and hybrid keying?
It depends on the power and the specific method that you use (it always depends, but isn’t that just like psychology!). The researchers provide some general guidelines though.
7. Does hybrid keying decrease the sample size requirements?
Only when using stepwise regression and over 1600 cases.
So, when all of the methods work, it becomes important to tailor your scoring to the size and data that you have. Results may be similar across methods, but better methods give you better results. One size doesn’t fit all method types!
human resource management, organizational industrial psychology, organizational management
Team Intimacy and Organizational Interventions: Emphasizing Team Cohesion May be More Effective (IO Psychology)
Topic: Teams, Development
Publication: Human Resource Management Review (JUN 2012)
Article: Too Close for Comfort? Distinguishing Between Team Intimacy and Team Cohesion
Authors: Rosh, L., Offermann, L. R., & Van Diest, R.
Reviewed By: Thaddeus Rada
Within IO psychology, research on teams has become increasingly important in recent years. As organizations have begun to use teams for a wider variety of roles and purposes, it has become necessary for both researchers and practitioners to gain a better understanding of how teams work and how they can be designed to operate most effectively. Two constructs that have received research attention in the realm of teams include team intimacy and team cohesion. Although these constructs may appear to be very similar from the outside, Lisa Rosh and colleagues argue that there are important differences between these constructs, and that they are best conceptualized as distinct constructs.
Rosh and her colleagues explored the literature surrounding group intimacy and group cohesion, noting that, although there are areas of overlap between the constructs (e.g. interpersonal attraction), there are also key differences between them. Specifically, group intimacy necessarily requires some level of group cohesion, while a group may have high levels of cohesion without the added elements (e.g. interpersonal affection) of intimacy.
The authors suggest that, to date, many team-building initiatives in organizations have been designed to foster team intimacy, not cohesion. Noting that the link between intimacy and team performance has not been well-established, the authors suggest that practitioners shift their focus towards team-building interventions that focus, not on intimacy, but rather on the “work-focused” purpose of the group, such as the group’s commitment to their task and the task-based collective efficacy of the group. However, the authors do not discount the importance of intimacy entirely; indeed, they note that group intimacy is likely to become more common as teams take on more and more sophisticated projects in organizations; as such, they argue that additional research and examination of group intimacy is needed so that practitioners will be equipped to address this component of life in teams.
human resource management, organizational industrial psychology, organizational management
Predicting someone’s propensity to morally disengage (IO Psychology)
Topic: Assessment, Personality, Ethics, Counter-Productive Work Behavior, Workplace Deviance
Publication: Personnel Psychology (SPRING 2012)
Article: Why employees do bad things: Moral disengagement and unethical organizational behavior
Authors: Celia Moore, James R. Detert, Linda Klebe Treviño, Vicki L. Baker, & David M. Mayer
Reviewed by: Alexandra Rechlin
Organizations obviously want their employees to be ethical. While there are existing measures that are used to predict who will act immorally, the authors of this paper proposed a new construct that they called an individual’s propensity to morally disengage – an individual difference in how people think about ethical decisions and behavior that allows them to act unethically without feeling bad about it.
Celia Moore and her colleagues developed a measure of an individual’s propensity to morally disengage. In a series of studies, they then validated the measure for working adults by showing that the propensity to morally disengage was positively related to unethical behavior after accounting for a number of other related traits, orientations, and emotions. Predicted outcomes included self-, supervisor-, and coworker-reported unethical behavior, decisions to commit fraud, and self-serving decisions in the workplace.
You may be wondering how this paper is relevant to practitioners. This new measure of the propensity to morally disengage predicts unethical behavior, and it is short – it only includes eight items. While it has yet to be validated for employee selection, this measure certainly shows promise for its ability to predict unethical behavior. The authors also found that this measure had a low correlation with social desirability, so it seems to be fairly resistant to test-takers faking their responses to receive a good score. If your organization is using a lengthy integrity test in the selection process for the sole purpose of predicting those who would conduct unethical behavior, then this new measure may be something your organization might want to consider using instead.
human resource management, organizational industrial psychology, organizational management
