Imagine you and I start a small business. The simplest example might be an ice cream truck. In the weeks after our grand opening, lines stretch around the block. It's time to hire more people. How should we interview?
Let's ask everyone an open-ended question. It'll give us a window into candidates' personalities:
Tell me about something you've done in your current (or most recent) job that is creative.
Video games are creative to me. When I interview candidates for our business, I'll judge them by the kind of games they've designed and developed. After all, gamers and game developers love ice cream; it's practically in the same category as Red Bull and Doritos.
Would you judge the interview differently? Maybe you'd be more impressed if the candidate's creativity related directly to food service. Or maybe they'd earn kudos for experience that resonated with your background — whether it be medicine, law, finance, etc.
By asking a purely open-ended question — one that isn't linked to a numerical scorecard, we're injecting randomness into the process. The randomness will blossom exponentially as new hires become interviewers. In a few months, our ice cream truck might include a game developer, a surgeon and a certified public account (which is, incidentally, the perfect set-up for a bad joke).
This is an exaggerated example, but it highlights real problems with open-ended questions.
Though interviewers tend to prefer open-ended prompts, a meta-analysis of hiring data shows they're a poor predictor of an employee's long-term success. In other words, when employers only ask unstructured questions, candidates who perform well in the interview don't reliably succeed after they're hired. (Of course, "success" is relative; the cited study measures it from an employer's point of view.)
The example question about creativity is good. "Good" means a skilled interviewer can ask it, gather relevant feedback and use it to hire a candidate who ultimately performs well. The question isn't predictive, though, because it gives interviewers too much wiggle room. It lets some jerk judge people based on video game knowledge.
Opinion-based interview feedback tends to be a vector for unconscious bias. If you emphasize video games — as in my ice cream example, you're unintentionally putting candidates at a disadvantage based on age, gender and nationality. Not everyone is a gamer. Almost every interviewer has a blind spot for conscious bias. It has little to do with your IQ or education level, and the bias is most prevalent among people who think they're immune to it.
Unfair or inconsistent hiring decisions can be self-perpetuating. It's kind of like tolerance stack-up in robotics. Check out the arm below. Its joints are networked devices with extremely precise timers. The wrist needs to know when the elbow moved, which has to know when the shoulder moved, and so on. The cumulative effect of a 50 microsecond delay can turn the circle drawing into a jagged oval.
Likewise, an unconscious bias may seem insignificant in the context of one interview. But a new hire who's selected through bias may perpetuate the bias in his or her hiring decisions. It's a stack-up of subtle misjudgments.
Approach with caution
Interviewers are overconfident when it comes to scoring open-ended feedback. We overestimate our abilities to judge responses fairly and consistently. Purely opinion-based questions — especially those focusing on personality or "culture fit" — can be a vector for unconscious bias. Hiring data shows these unstructured questions aren't predictive of an employee's performance.
That's not to say open-ended prompts are wrong or useless, but they should be approached cautiously and — in many cases — mapped to a concrete rubric. In an engineering interview, for instance, you might ask "How would you build X?", but compare it to a table of specific programming and technical design skills. Using a structured rubric to judge open-ended feedback boosts its predictiveness.
Here you're probably expecting a contrived joke that ties together the conclusion and the opening story. Well, I'm sorry to disappoint you. Analogies are like half-eaten ice cream cones: it's hard to dispose of them.
* The question about creativity is a real prompt recommended by one employer.
** To protect confidentiality, I've avoided discussing real interview scenarios. The hypothetical explores issues that are relevant to most companies.