Interview Reliability: Statistics vs. Personal Experience

Publication: International Journal of Selection and Assessment
Article: Employment interview reliability: New meta-analytic estimates by structure and format
Reviewed by: Megan Leasher

This article focuses on the reliability of interviews. The more error introduced in interviews, the less reliable they are. The researchers targeted different sources of error, both from the interviewee and interviewers. Interviewees can introduce error into an interview when they answer similar questions from the same or multiple interviewers differently, while interviewers introduce error when they interpret, evaluate, and rate identical responses differently.

The researchers found that the more structured the interviews, the more reliable the interview results tended to be. In addition, panel interviews were more reliable than single-interviewer interviews.

I have to admit that I am a little torn regarding the desire to strive toward reliability in interviews. I know from a statistical perspective that interview reliability is important, since the more reliable interviews can be, the more capable they are of statistically predicting the future job performance of candidates (who become hires).

In industrial and organizational psychology, this conflict between ideal statistics and personal experience is the crossroads of “where research meets practice.” Sometimes what is helpful isn’t necessarily the ideal scenario or course of action from a statistical perspective. This crossroads can make for great discussion fodder, salty arguments, ideas for new research, or all of the above.

If interviewees answered the same or similar question in the exact, regurgitatingly same way, and all interviewers heard, interpreted, evaluated, and rated a candidate’s responses in the exact same way, you would have a perfectly reliable interview. From a statistical perspective, that is.

But would you be missing something?

Think of all of the times in an interview that you heard, saw, or interpreted something very differently from your colleague who asked the candidate a similar question. Then, when you debriefed after the interviews were complete, those differences gave you something really good to discuss, like nuances that the other didn’t pick up on, unique reactions to answers, and so on. Perhaps a third interviewer had another fresh perspective to share, turning it into a real debate. Debates lead to unexpected and more in-depth realizations about the candidate that one interviewer could have conjured up alone. Statistical reliability would not have wanted this scenario to take place. It would be unhappy that interviewers disagreed, because that would jeopardize and threaten the high reliability it would be striving for.

Which is more important? Reliability or the ability to learn or interpret something unique?

The Consequences of Fit Across Cultures

Previous research has demonstrated that fit – the compatibility between an employee and a work environment – tends to lead to better attitudes, better job performance, and lower turnover (Arthur, Bell, Villado, & Doverspike, 2006). However, this research has focused predominantly on populations in North America. Today, companies operate across geographical boundaries in a globalized world of business, and it does not seem prudent to apply results found in North America to countries in Europe and Asia. Therefore, it becomes necessary to understand if fit across cultures predicts work attitudes and job performance across the globe.

For this study, Oh, Guay, Kim, Harold, Lee, Heo, and Shin reviewed 96 studies that had previously been conducted in East Asia, Europe, and North America. First, they found that fit predicts work attitudes – such as organizational commitment, job satisfaction, and intent to quit – as well as job performance across cultures. In taking this result further, the authors then looked at different types of fit, how they may vary across cultures, and how this may influence job performance.

To that end, four types of fit were identified:

  • person-job fit, the compatibility between an employee and their job;
  • person–organization fit, the compatibility between an employee and the organization;
  • person–group fit, compatibility between an employee and their coworkers; and
  • person–supervisor fit, the compatibility between an employee and their direct supervisor.

These types of compatibility were then grouped into two types: impersonal and interpersonal. The compatibility between an employee and their job or organization were categorized as an impersonal type of fit, since they do not concern interpersonal and social aspects of work. The compatibility between an employee and their co-workers or their boss were categorized as an interpersonal type of fit, since they are directly concerned with how well an employee relates to other people in the workplace.

In comparing fit and performance across cultures, impersonal fit had stronger effects in North America and Europe, while interpersonal fit had stronger effects in East Asia. In other words, for Westerners, matching an employee to the right role and organization is most important, while human resource management in East Asian business environments should take special care to build positive teams in which social conflict is kept to a minimum.

Selection Tests and Job Performance

Ideally, when we test prospective employees, we gather valuable information that will help us determine if a candidate is suitable for a given job. But that’s not all. We also create an impression in the candidate’s mind about our company, its culture, and its values. Research has found that candidates’ reactions to selection testing do affect their attitudes. For example, candidates may react anxiously or perceive unjust treatment. These reactions can influence a candidate’s view of an organization, as well as determine whether they would recommend it to others. New research (McCarthy, Van Iddekinge, Lievens, Kung, Sinar, & Campion, 2013) explores the possibility that selections tests could also be influencing subsequent job performance.

The authors conducted four different experiments in a variety of settings. They found that reactions to selection tests did relate to job performance. However, they go on to explain that this connection is nothing to be concerned about. Many of the same personal characteristics that influence reactions to testing also influence job performance, so we’d naturally expect a relationship between the two. Similarly, reactions to testing may also have an effect on test scores, and test scores themselves are (hopefully) related to job performance. Additionally, the major finding of the study was that candidate reactions to testing did not diminish the usefulness of selection tests. That is to say, it makes little difference if some candidates feel discouraged by the testing, while others feel elated. Either way, the selection test will have the same ability to predict performance fairly.

The authors caution that these findings should not give an organization the go-ahead to completely disregard a candidate’s attitude. It is logical to expect that many benefits occur when candidates feel they are treated fairly. A company’s reputation is, in part, determined by word of mouth, and a sense of fair play may result in favorable attitudes toward the organization that are subsequently communicated to others (Hausknecht, Day, & Thomas, 2004).

This study is important, because it demonstrates that organizations need not make positive or negative feelings of candidates a primary objective when designing selection tests. Simply build a fair test that follows best practices. No matter how the candidates feel, we can be confident that a properly designed selection test is gathering the valuable information needed to hire the right employee for the job.

How Prospective Employees Judge Fit With An Organization

Publication: International Journal of Selection and Assessment
Article: How Interviewees Consider Content and Context Cues to Person–Organization Fit
Reviewed by: Megan Leasher

When you interview for a job, you make choices using the relatively small amount of information to which you have access. As a candidate, not yet on the job, your view of the organization and its work culture is limited. In a way, you are forced to judge a book by its cover, and maybe also by the sneak peak of pages the company gives you access to during the selection process. This research focused on how we make those judgments.

Person-organization (PO) fit is the match between a prospective employee’s personal characteristics and the organization’s cultural characteristics. When candidates feel that their personality and goals are a strong fit with an organization, they are more likely to continue pursuing the organization. When an employer senses strong fit between a candidate and the organization, that candidate is more likely to receive a job offer.

Participants in this study went through a series of scenarios to determine what judgments affect our sense of a good fit. The researchers found that both interviewer behavior and the interview process itself contributed strongly to candidates’ perceptions regarding fit, whereas the actual questions the interviewer asked did not.

In other words, it is not what interviewers say; it’s what they do. Interviewers are the ambassadors of their organization’s cultural values, whether they want to be or not. They are the book cover, and candidates will judge their potential fit within the organization based the image the interviewers give them. A candidate’s perception of being well matched can make the difference between accepting and declining a job offer. It would be well to make that presentation both positive and accurate.

There are tons of practical applications for these findings, but I want to focus on one the article does not address; the cost associated with a bad fit. Think about it. If an interviewer misrepresents the culture and values of the company, the candidate will likely misread their fit. This can lead to a candidate accepting an offer from a company for which they believe they are a match, only to find out later that the book didn’t match its cover. Turnover occurs. This is an expensive mistake for all parties; organizations have to begin the recruiting process again, selecting a new candidate, while productivity suffers from an open position. Candidates must start job searching again, leading to additional gaps in paychecks and benefits coverage. No one is happy.

Your book cover should always accurately represent the contents. Interviewers are brand ambassadors and should be selected with precision.

Emotional Labor: The True Cost of Service with a Smile

Publication: Journal of Applied Psychology (July, 2013)
Article: Affect Spin and the Emotion Regulation Process at Work
Reviewed by: Ben Sher

Talk about demanding work! In addition to their typical job duties, like waiting on tables, making sales, or assisting customers, customer service professionals must also perform emotional labor. When employees smile cheerfully at the end of a grueling shift, they are performing something called surface acting, which is a type of emotional labor. Research has shown that emotional labor can lead to psychological strain and fatigue. The current study (Beal, Trougakos, Weiss, & Dalal, 2013) has made advancements in this area of research by scrutinizing a new variable, called “affect spin”.

The authors define “affect spin” as the extent to which people experience a variety of different emotions throughout the day. For example, some people might fluctuate between two or three emotional states throughout the day, while others fluctuate between six or seven. Why does this matter? The authors conducted a study of restaurant servers in multiple restaurants, and found that levels of “affect spin” influence the degree to which the servers experienced psychological strain and other negative outcomes due to the emotional labor they performed.

When individuals experience many different emotions during the day (high “affect spin”), they also tend to react more strongly to emotionally charged events. Indeed, the study found that these servers experienced more psychological strain as a result of their surface acting. Additionally, people who experience many different emotions may find it harder to predict how they will feel at any given time. This unpredictability makes them exert more effort in forcing themselves to display the “right” emotions on the job. The study also found that people high on “affect spin” experienced more fatigue as a result of their emotional labor.
However, the authors also found that people with high levels of “affect spin” may also have the ability to experience higher levels of psychological strain without it leading to fatigue. This is because they may have more experience with feeling “stressed out”, and therefore handle it better. In other words, being high on “affect spin” has its advantages in addition to its disadvantages.

So yes, servers and other customer service professionals are at risk for psychological strain and fatigue due to the emotional labor required of them. However, thanks to this study, we better understand that not all individuals respond to those job demands in the same way, and why. Although this sheds light on who may be best cut out to do customer service work, as always (say this with a smile), more work is necessary.

Conscientiousness and Job Performance: Is Conscientiousness Always King?

Publication: International Journal of Selection and Assessment (2013)
Article: The validity of conscientiousness for predicting job performance: A meta-analytic test of two hypotheses
Reviewed by: Megan Leasher

Conscientiousness is a predictor of job performance in many jobs, job levels, and industries. But does being conscientious still predict job performance as strongly when characteristics and requirements of the job change? Is conscientiousness the Holy Grail of employee traits?

To learn more about this, the authors conducted a meta-analysis across 53 research studies where conscientiousness was a predictor of job performance. They then rated the jobs that were included in these studies on a number of factors including the level of worker autonomy, how much of the work followed a routine, how much thought and mental ability was required, and so on.

Overall, there were two big takeaways:

  1. Conscientiousness was a stronger predictor of performance for jobs that required more routine, structured work.
  2. Conscientiousness was a weaker predictor of performance in roles that required high levels of cognitive ability, possibly suggesting that intelligence in some way suppresses the influence of personality on job performance.

Taken together, conscientiousness may be more useful in roles with a lot of routine, which are more likely to be hourly and/or entry level roles. Alternatively, conscientiousness may not be as useful for higher-level roles that require more thought and mental ability. Thus, spending on assessments to determine conscientiousness may only be selectively useful. It’s a good lesson in not only what predicts job performance, but what is worth spending your budget on for successful hiring decisions.

What do you think about conscientiousness and job performance? We’d love to hear your thoughts in the comment section below.

Ask Me Online: Benefits of a Web-Based Reference Check Process

Publication: International Journal of Selection and Assessment (March 2013)
Article: Web-Based Multisource Reference Checking: An Investigation of Psychometric Integrity and Applied Benefits
Reviewed by: Thaddeus Rada

Reference checks, the process of asking past employers and colleagues about a job applicant’s qualifications and past performance, have long been a part of human resource management. However, despite the fact that the limitations of using reference checks in the hiring process have been well-recognized for many years, they continue to be a popular part of many organizations’ selection procedures. Recently, there has been increased interest in how reference checks might be made more useful.

To this end, Cynthia Hedricks and her colleagues tested a web-based reference checking survey. It allowed multiple references to all provide structured ratings about a particular applicant through a computer-based survey (as opposed the usual and dubiously valuable phone reference check). The researchers tested this online survey system in multiple organizations, using some basic questions about capabilities, such as professionalism and interpersonal skills. Each survey also had questions tailored to specific competencies and requirements of each organization. Overall, the authors found that the survey system was both efficient and effective. Information was gathered from references quickly, and the ratings that references provided were effective for predicting employees’ job performance and involuntary turnover. In addition, the standardization that the survey brings may help to improve the overall validity of reference checks, thereby improving the contribution that reference checks can make to the hiring process.

This web-based survey shows promise as a flexible, fair tool that could help many organizations streamline reference checks.

What do you think of web-based references? Are they effective? We’d love to hear from you in the comments section below.

When Reading Research Leads to a Brain Full of “What?!” (IO Psychology)

Publication: International Journal of Selection and Assessment (MAR 2013)
Article: Personality types and applicant reactions in real-life selection
Reviewed by: Megan Leasher

When you read scientific research, you should be left feeling as though you gained knowledge and/or have something new and shiny that can be applied to the real world. But once in a while you finish an article and there is nothing but unpoppable “What did I just read?!” bubbles floating in your brain.

This article focused on how applicants’ personality types might impact their reactions to assessment tests within a hiring process. Specifically, candidates for firefighter, dispatcher, and rescue management roles had to complete a series of personality and cognitive assessments as a part of the selection process. Immediately after, they were asked to complete a voluntary survey asking about their reactions to the tests. The researchers found that personality types had no impact on applicants’ perceptions that the assessments were related to the job and that the tests could predict future job performance. One personality type did perceive the tests as less fair than those with other personality types, but the difference may not have been large enough to have real meaning.

As I kept reading the article, I kept wondering how this information would be applied, or even how it would be useful. I kept wondering this because the authors never told me. The authors briefly mention previous research stating that applicant reactions can impact whether or not a candidate might accept a job offer and/or impact their future performance on the job. Yet they never relate their own findings to this previous research. I was left hanging.

The study also had a number of confounds, a few of which the authors acknowledged. Looking solely at rescue applicants isn’t representative of most jobs and applicants. Candidates had to first pass a physical test before they were allowed to begin the personality and cognitive assessments. The reactions survey only asked for their reactions to the personality and cognitive tests, but wouldn’t their perceptions of the physical test muck up their thoughts a bit?

Also, participants voluntarily completed the reactions survey, and not everyone completed it. Wouldn’t the thoughts of those who did NOT want to share their reactions be critical? Finally, their research found different reactions to the assessments based on gender and age, but they never investigated further, which I found disappointing.

Now I have to be fair and say that no research is perfect. All research has confounds. But when you feel as though you don’t get the “so what?” of the entire study and there are also lots of confounds, how are you supposed to react?

After reading this article I was left feeling a little icky inside. But it reminded me that reading research with a discerning amount of skepticism is not only healthy, it is mandatory. It reminded me of a wonderful quote by the philosopher George Santayana: “Skepticism, like chastity, should not be relinquished too readily.”

Should Your Spouse Interview for You? (IO Psychology)

Publication: Journal of Leadership, Accountability, and Ethics (APR 2012)
Article: Spousal interviews of applicants for employment: A legal and ethical dilemma
Reviewed by: Megan Leasher

How well can your spouse sing your praises? Well enough to help you get that job you’ve always wanted?

This article discussed the ethical and legal issues surrounding spousal interviews for employment. Ever heard of it? Some companies are choosing to include spousal interviews as a part of their hiring process, especially for sales roles. As sales jobs can include varying hours and unpredictable income, some organizations want to make sure that the spouse fully understands and is on board with what could come. I don’t personally know of any organizations doing this, but it honestly scares the crap out of me (that is one of those phrases I should probably try to stop using).

As a spouse-less individual, I am a bit overwhelmed by the idea of this. Are you telling me that in my hunt for a mate, I now have to add “awesome interviewer on my behalf” to my list of requirements? Isn’t dating hard enough already?!

The argued business rationale is that by making sure the spouse is on board, it will prevent salesperson turn and subsequent revenue loss. But what risk does this introduce? First of all, requesting a spousal interview reveals the candidate’s marital status before a conditional offer of employment. Marital status is not protected by federal law, but it is in many states. The simple act of revealing of whether or not a candidate has a spouse could lead to intentional or unintentional discrimination. In addition, what if that spouse happens to be a same-sex spouse? That could open a whole new can of legal worms (seriously, I need to update my phrases). And what if you don’t have a spouse to conduct an interview? Does that give you an advantage because there is no one the organization has to make sure is “on board”, or do you lose the opportunity to have someone speak on your behalf?

Regardless of the legalities, the author argues that it could cast a “veil of suspicion on the hiring process” (p. 123). And what could this veil do to the reputation of the organization? How do you even make a hiring decision…Think of the different scenarios to deal with; if a candidate is very qualified but the spouse isn’t on board, do you hire? What about the flip scenario? The organization could get screwed either way. The only way the organization wins is if they find an amazing candidate with a spouse who is up for the unpredictable nature of the job. Is it even worth it? Most likely not.

Think what job hunting and hiring processes would be like if they all included a spousal interview. Oh the madness…

Random Thoughts While Taking a Personality Test (IO Psychology)

Publication: International Journal of Selection and Assessment (DEC 2012)
Article: What is in Applicants’ Minds When They Fill out a Personality Test? Insights from a Qualitative Study
Reviewed by: Megan Leasher

Have you ever taken a personality test as part of a hiring process and found your mind wandering?  Where did it go?  Did you wonder how to best answer the questions to secure the job?  Or did you ponder why puppies are just so damn cute?  Or why the Flock of Seagulls hairstyle never took off?

In this study, researchers interviewed test takers to learn what they were thinking while taking a personality test.   I was excited about this, because in industrial psychology we tend to fixate on how well personality tests succeed in identifying rockstar candidates, but we should also think about how candidates react when completing them.  Here are a few of the findings:

  • Some test-takers had a difficult time answering the questions, as no context was provided.  For example, if they had to rate how likely they are to “start a conversation with a stranger” (with no other context), they felt as though their answer would differ if asked in a work setting versus their personal life.   I can see this one.
  • A few didn’t feel as though they knew themselves well enough to answer the questions.   This one really threw me; I don’t know about you, but I’m not sure I would want to hire someone who lacks enough self-awareness to answer questions on a basic personality test.
  • A number of individuals answered the questions in a way to make them look more favorable, to help secure the job in question.  Others tried to answer as honestly as possible, thinking the test would catch them and figure out their devious plan.   I see both reactions here; some assessment tests can, in fact, detect if someone is trying to cast themselves in a more favorable light; other tests do not.  This is one of the great mysteries when taking an assessment test…Will the stats police uncover my lies?
  • Many test-takers tried to figure out how their answers would be analyzed and interpreted.  We all have this natural curiosity, but voodoo-magician test publishers keep this secret close.  (Cueing “something D-O-O economics…”)
  • Most individuals noticed questions that seemed to be similar and/or repeating, but assumed this was a way to detect how consistently they answered the questions.  This is true; it helps test creators measure the consistency of your answers, see if you are paying attention whatsoever, or reveal if you are just having your cat select the answers for you (we all know how much cats love to sprawl out on keyboards).

So the moral of the story is: our reactions to personality tests are as different as our personalities themselves, which might mean that I am the only person out there that wants the Flock of Seagulls hairstyle to make a comeback.