The ultimate interview question for hiring software engineers

Note: The views in this post may not represent those of my employer, and it doesn't include any info specific to Amazon's hiring process.

"Water is flooding the office through the air conditioning vents. It's rising up to your neck," Mike said. He leaned forward and spread his palms flat on the conference table. "What do you do?! What do you do?! What do you do?!"

"Umm...break a window?" the interview candidate blurted out.

"Trick question," Mike shot back. "It turns out there's no water, but there's a zombie outbreak. Your manager is a zombie. What do you do?! What do you do?! What do you do?!"

"Knock over a filing cabinet," said the job seeker. "Try to slow her down."

This was Mike's first time conducting an interview. After college he worked for a San Francisco company that built apps for indie bands. The start-up had lava lamps in the lobby, a pool table in the break room, and an equally casual approach to interviewing. Someone asked Mike to do a "culture fit" interview with 5 minutes' notice; he drew on his past experience as a comedian.

Delicate trade-offs

We can probably agree that asking about zombies is a questionable approach, but I love the story because it illustrates (to an extreme) how each interviewer injects personality into the hiring process. Now my friend Mike is a senior dev. who has made dozens of successful hiring decisions. (He doesn't ask about zombies anymore, though he still spices up phone screens with his unique brand of humor.)

Here are a few of the more nuanced issues that Mike and I have discussed when it comes to picking appropriate interview questions:

  1. Knowledge vs. the ability to generalize: When you're hiring a front-end engineer, do you require someone who knows the ins and outs of JavaScript frameworks? Or will you accept a C++ expert on the basis of her abstract problem solving abilities?
  2. On-site performance vs. experience: When there's a strong emphasis on whiteboarding, you can hire candidates who can't afford a degree on the basis of their talent. That's awesome. At the same time, whiteboarding produces a lot of false negatives. Maybe the candidate didn't have coffee that morning or he felt anxious.
  3. Personality vs. technical skills: How well does the candidate communicate and collaborate with teams? How important are the soft skills relative to his or her engineering chops?

The law of large opinions

I wouldn't be surprised if 10 engineers had 10 different opinions about the list of trade-offs. The answers depend greatly on your personality and professional background.

That's why leveling is important: gathering feedback from peers to make sure a question is fair and relevant. But unless you work with a very small team, you'll never find a problem that everyone likes; it's always too specific or too broad or too easy or too difficult. I call this the law of large opinions:

If you solicit too much feedback about any opinion-based topic, the opinions will cancel each other out. From a decision making perspective, it will be as if you never did research.

A transparent bar

When interviewers use the metaphor of a "technical bar," they tend to talk about holding the bar at a particular height. You don't want candidates to breeze through an open hallway (easy bar; false positives) or to break their backs playing limbo (difficult bar; false negatives).

Once an organization figures out where the bar should exist, it tries to collectively cement it into the doorway. While this is a worthy goal, let's be realistic: personal opinions are inextricable from the hiring process; in any engineering org., folks will disagree on the bar's height and the competencies that define it.

Rather than thinking of "the technical bar" as a singular entity, I imagine the hiring process as an obstacle course where each engineer forges a bar from her own expertise. The bars don't have to exist at the same height. They don't even have to consist of the same material. It's okay for developers to assign different weights to each competency.

In this way of thinking, the bar's greatest attribute is its transparency. I don't try to persuade everyone that my method of interviewing is correct. Instead, I choose problems that are appropriate based on my experience, and I try to state my assumptions: listing the competencies that I want to test and documenting why I think my questions are a good way to measure them.

The "ultimate interview question" means something different to every engineer, and that's okay. We don't have to agree, but we should strive to make our personal bars as transparent and evidence based as possible.

If all else fails, you can always win an argument by knocking over a filing cabinet to slow down your opponent.