Finding the Best Software Developers

There are several components to a recruiting campaign, including advertising, working with headhunters, selling the company to candidates, checking references, and making offers. But the cornerstone of a good recruiting campaign is evaluating candidates' skills. With the ability to pinpoint the skill level of each candidate, you are no longer flying blind and you have a tremendous advantage over the average software development shop.

Know Thy Organization

There are many ways to succeed (and sadly, many more ways to fail), depending on how your organization works and what it expects of its employees. The same developer can perform brilliantly in one organization, and still fail miserably in another. So the best place to start looking for clues is in your current organization. (If you are starting a new organization, look at your last one. If you've never worked in a software development organization, you shouldn't be trying to hire software developers. Software development is a field where people routinely over-represent their abilities, and experience in the trenches is the only way to develop the skills to see through this.) Take stock of the developers you work with. Who consistently gets the best results? What evidence do you rely on to make that judgment? Once you can tell who is effective and who isn't in the context of your organization, you can make guesses as to why, and use these guesses as the basis for formulating interview questions.

Preparing the Interview

Before getting into the questions themselves, here are some tips that have worked for me when putting an interview together.

• Though some people have called my interview a "test," there is a big difference between asking technical questions in an interview and giving a candidate a written test. In an interview, you can pick up nuances of speech and body language, and can gain more insight into how a candidate works if you ask follow-up questions or prompt them if they get stuck. A test is an artificial and unnatural situation in the workplace, where the "cheating" of looking up an answer in a book or asking a neighbor for help is to be encouraged.
• Give your candidates a tough set of questions and root for them to succeed. Always remind yourself to be as fair and scrupulous to each candidate as you possibly can, and show respect for every candidate throughout every interview.
• Several interviewers should evaluate each candidate independently, each taking at least an hour. This is a standard interview technique for any industry, as it makes it more difficult for a candidate to pretend to be someone they are not. Get together after all the interviews and share your evaluations. If there is a discrepancy, explore it to see if it indicates any individual bias on the part of any interviewer.
• The organization's best software developers should always be among those doing the interviews, along with the hiring manager and more than one developer who will be working directly with the new hire. An organization's best software developers, once trained in effective interviewing techniques, are the most likely to accurately discern a candidate's software skill level. They will also tend to impress the candidate with their own skills. The hiring manager and coworkers are more likely to have their radar out for misfits or personality problems.
• The most revealing technical questions are those that get candidates to work directly with code or design concepts. Talking about past experience helps to show professional skills, but technical skills are best observed directly. As Tom Peters writes in The Circle of Innovation (Knopf, 1997), "We are what we do, not what we say we do."
• Ask at least four different kinds of questions in each interview, to examine a candidate's skills from as many angles as possible.
• Have all your interview questions set and well rehearsed so you can ask them naturally. If you are stiff or nervous, it puts the candidate on edge, and makes a bad impression.
• Keep your technical questions short and simple, and ask one question at a time. Good software developers will tailor each answer to each question the way they tailor their code to each new situation. They have no chance to demonstrate this skill if you ask confused, multipart questions, or if you elaborate on a question until you've led them to an obvious answer.

Developing Questions

You should continually try out new technical questions as you develop your interview. When you first try a question, you can't be sure if it will help you differentiate developers' skills. You can get a headstart by asking developers who currently work with you, and check if their answers correspond to their skills as you know them. If a question is too easy or too difficult, almost all your candidates will give similar answers, so the question tells you nothing about them. That said, some questions that are almost too easy or too difficult -- but not quite -- can help you differentiate skill levels at the extremes of the scale, so don't give up on a question before you've had a chance to hear a variety of candidates answer.

The more new questions you can qualify, and the more variety of questions you give each candidate, the more you avoid the tester's perennial dilemma. This dilemma is that good questions are hard to find, and they are only valuable if the person taking the test has not been coached with the right answer. For example, there is a question that many Silicon Valley firms ask their candidates, which is, "Why are manhole covers round?" There is a huge problem with this question (even if you cast aside doubts about its validity as a predictor of success at programming computers). The problem is that this question and its answer have been published in numerous web sites and in books about software interviews. So if a candidate gives you the correct answer, "So the cover doesn't fall into the manhole," you can't tell if the candidate has figured this out on the spur of the moment, or has simply heard the question before.

Scaring Out Development Skills

Though successful software developers combine a large number of disparate skills, these skills can be organized into three significant categories. These are engineering skills, creative skills, and professional skills.

Engineering skills include knowledge of programming languages, operating systems, and software packages.

• How skillfully can a developer manipulate language constructs?
• How quickly can they find and fix defects?
• How much logic can they hold in their mind without getting confused?

• Engineering skills show the depth of a developer's knowledge.

Creative skills include problem solving and pattern recognition and show the breadth of a developer's abilities.

• How many different kinds of solutions can a developer think of?
• How good are they at tailoring a solution precisely to a specific problem?
• How flexible are they to new situations?

Professional skills include regular work habits and getting along with coworkers.

• Can a developer reliably complete assignments?
• Can they resolve a technical disagreement without antagonizing others?
• Do they take suggestions well?
• Are they enthusiastic?
• Do they show any leadership qualities?

Professional skills show how a developer can apply their other skills to help make your company successful.

Every developer has a different mix of these skills, and different strengths and weaknesses. To build a winning development team, you must evaluate each new candidate's skills in light of how they will fit in, and where you need the most expertise. To do a good evaluation, you must develop a good set of interview questions that can be used to measure these markedly different kinds of skills, and you must fine-tune your interpretations of the answers as you gain experience. Your interview questions are the filter through which you perceive your candidate's skills.

Technical Questions

The most valuable technical questions I have found are questions that have a wrinkle or a twist to them. Developers at different skill levels will handle the twist differently, and so give you a chance to discern the differences. Here's an example of a twist. Take a standard, well-known piece of code and alter it significantly or recode it in a more complex way than necessary, and ask candidates to analyze it. Can they keep the alterations straight? Can they see through the recoding and recognize the function? Ask them to suggest improvements. What kinds of improvements do they suggest?

Here is another twist. Ask candidates to code a function, but leave out part of the requirements. Do they call you out on the missing part as they try to code it? Do they plow through by making an assumption about what the requirement was? Is the assumption valid? Or do they simply get stuck and flounder, wondering what to do?

To examine engineering skills, center your questions on language and coding. There are many lists of technical interview questions on the Web that you can use to help generate your own ideas. Whenever I need fresh ideas, I pick a search engine and enter "Java interview questions" or "computer interview," and see where surfing leads me. Again, avoid directly using anything you see published on the Web or in a book (or in this article, for that matter). Here are some examples:

• Ask candidates to bring in their own code examples and explain them. You can pick up a lot about a candidate's skill level and approach by looking directly at their code. Candidates' explanations of their code can tell you about their priorities, thought processes, and communication skills.
• Ask candidates to analyze code, especially code with a twist. Tell them to explain the code to you as if you didn't know the language, and to "talk out loud" so you can hear them work their way through the problem. From this exercise, you can tell how clearly a candidate thinks about code, and time how quickly they figure out a problem. (See Example 1 in the accompanying text box entitled "Sample Technical Questions.")
• Ask candidates to write small functions, again, with a twist. The speed at which a candidate successfully completes this exercise seems to correlate very closely with their success once hired. I try to find functions that can be coded using a number of different approaches, and see which approach a candidate takes. The Standard C Library contains many good examples for this, such as strncpy or strtok. The main thing to consider is the difficulty of coding the function versus the time you have in the interview. A standard depth-first binary tree traversal is too easy, but what happens if you add the twist of doing it backwards -- from the rightmost bottom node to the top? Is that too hard? It depends on how much time you give. In my experience, good candidates can code standard library-type functions in 10 minutes or less. Nontrivial functions may take a half-hour to an hour. The harder the function, the wider the time variation between candidates, and the more likely it is that good candidates will get stuck and seem less adept than they really are. You are putting candidates on the spot by asking them to code in a job interview, and most will do much better away from the unnatural pressure and scrutiny. If you want to ask this kind of question, you need to experiment and find what works for you.
• Ask candidates about specific language constructs, features, or "gotchas." Since the scope of each question is so small, it helps to ask a number of these and then average out the responses. Some candidates who are quite good may have simply never run across some obscure language feature or another, so don't give too much weight to each individual question. The Microsoft technical interview includes several of these kinds of questions (see Example 2 in "Sample Technical Questions").

Creative Skills Questions

To examine creative skills, center your questions on design and concepts. Because this is a more vague set of skills to evaluate, it helps to try different kinds of questions. Be careful not to ask any question where you expect a single "right" or "best" answer. Creativity can be interesting, thought provoking, and inspiring, but it is not often right or wrong. Here are some examples:

• Ask candidates to describe their proudest achievements. This sort of question, from the standard interviewer's repertoire gives you an idea of candidates' values as well as their accomplishments. A related question is asking what candidates like to do most at work.
• Ask candidates about the meanings and applications of computer-science concepts. Surprisingly, candidates' descriptions of basic computer-science concepts can reveal much detail about their skill level. Beginners will tend toward either blank stares or stilted textbook responses, intermediates will usually give basic, correct, but unimaginative answers, and advanced candidates will give deep, finely crafted responses like the sayings of a Zen master. Examples of concepts to ask about include: cohesion, coupling, polymorphism, inheritance, maintainability, reusability, unit testing, database normalization, and so on.
• Present candidates with a design problem, and ask for solutions. If you have time, ask for "as many solutions as you can think of." These kinds of questions are very revealing of candidates' idiosyncrasies. Do they approach the exercise eagerly, hesitantly, or reluctantly? Do they solve the problem as presented or try to reformulate it? Are their solutions appropriate to the problem? Can they accurately assess strengths and weaknesses of their own designs or is their assessment shallow? Do they consider one of their ideas to be the only true way to solve the problem? Do you agree? The venerable "round manhole cover" question mentioned previously is a variation of this kind of question, though it has a single right answer. More telling examples include asking candidates to design well-understood things such as a toaster, computer-file system, or linked list API, and asking candidates to solve problems such as weighing an aircraft carrier or picking apples.
• Ask candidates to optimize or otherwise improve code or designs. Be careful using the word "optimize" unless you want candidates to only consider execution speed. The question, "How would you improve this?" or, "How would you make this better?" leaves it up to candidates to state their views on how software can be improved. I have found these types of questions to be difficult for most candidates, and only the most advanced have made good showings. Examples include untwisting the twisted code you used for analysis questions (as in Example 2) and optimizing a complex design such as a Mandelbrot set generation function.

To examine professional skills, center your questions on work situations and relationships with managers and coworkers. There are hundreds of books available with examples you can use and tips on how to interpret the answers. A meaningful discussion of these questions is beyond the scope of this article, but here are some of my favorite books to recommend.

• Topgrading, by Bradford Smart (Prentice Hall, 1999), is an excellent text on recruiting and interviewing and on how interviewing integrates with managing employees once hired.
• Hire the Best...and Avoid the Rest, by Michael W. Mercer (AMACOM, 1993), is less comprehensive, but is also an excellent introduction to best recruiting practices.
• 96 Great Interview Questions to Ask Before You Hire, by Paul Falcone (AMACON, 1996), is another great source of ideas.
• Coping with Difficult People, by Robert Bramson (Dell, 1988), though not strictly about hiring, is a great reference of red-flag behaviors you should look out for in interviews.

Evaluating Answers

Though finding good technical interview questions is difficult, the most difficult skill is evaluating the answers. Unfortunately, because of the highly specialized nature of computer programming, general hiring reference books give little guidance. Here are some things that have worked for me:

• Don't base a hire/no hire decision on the answer to a single question. Always corroborate with other answers or other interviewers' impressions. Don't treat your questions and their answers as absolutes.
• Take into account whether the candidate seems tired or inordinately nervous and give them a break. A good technical interview is very unusual, so most candidates won't expect the kind of questions you'll be asking.
• Listen for clarity of thought and communication. You can evaluate this by paying careful attention to each answer to see if the candidate understood your question and made their answer understandable to you. Did they answer precisely the question you asked or did they get off track?
• Listen for honesty. Good software developers have trained themselves to interact successfully with computers, and lying to computers doesn't work. So, be on the look out for any fudging, hedging, weasel words, or the like. You should never ask a technical question that you can't answer yourself. You have to be able to pick up on fudging, even when it is subtle.
• Listen for humility. It is a bad sign when developers think their abilities are better than they really are. Developers with this problem are usually stuck at their current level of ability, and tend to get into conflicts with other developers. The best developers have an accurate view of their abilities, and do not feel the need to exaggerate how good they are. In an interview, look out for wrong or incomplete answers told with absolute conviction, put-downs of other developers, or a condescending attitude.
• Measure for speed. Good software developers are faster than their less successful counterparts -- and not by just a little. Industry studies show strong developers are more productive than their less successful colleagues by a factor of 10 (see Peopleware, by Tom DeMarco and Timothy Lister, Dorset House, 1987). Another surprising finding is that there is a strong correlation between speed and quality (see "Some Psychological Evidence on How People Debug Computer Programs," by John D. Gould, International Journal of Man Machine Studies, 1975). You can measure this in an interview by timing a candidate in coding or analysis questions. Since timing a candidate can be done easily (and discretely, lest you drive your poor candidate into a frenzy of nerves) it is a powerful technique for measuring skills.
• Listen for tenacity. Good software developers are not afraid to tackle difficult problems, and they will stick to development projects until they are completely done. This can come out in an interview in answers to open-ended design questions. One good sign is when candidates pause in thought for a long time at the end, wracking their brains to come up with one more answer. Even if they end in frustration, saying, "I know there's another solution, I just can't think of it now," give them credit for striving. If candidates give up on questions quickly, and seem relieved to be done with thinking, it may be an indicator of how little they will apply themselves to their job once hired.
• Listen for enthusiasm. Many successful developers share a delight for the game of solving technical problems. A developer who has this quality will not only tend to produce software more quickly, but will be more fun to work with. In an interview, look for a candidate to enjoy answering technical questions ("Ask me some more like that!"). Other good, but more subtle signs are: leaning forward in their chair to look at written material you show them, talking excitedly about solutions to problems you present, or relaxing with a satisfied smile after giving a successful answer. Many successful developers are introverted, or do not warm to strangers right away, so don't expect every good developer to bubble with childish delight in their interview. However, enthusiasm is a good quality for a developer to have, and can be a predictor of success, especially if accompanied by a good dose of clear thinking.
• Last but not least, listen to your gut. Though many interviewing texts will tell you this is a mistake, a structured technical interview will give you a huge amount of subtle information about each candidate. You will subconsciously assimilate this information and form an impression of each candidate by the end of the interview. Your gut isn't foolproof, but it's always worth listening to and evaluating, especially if the impression you get is strong.

Conclusion

So, there you have it. Finding the best software developers through interviewing is difficult, but it can be done. If you use any of the aforementioned techniques at all, you are already above average, and the more thought and experience you put in, the more excellent you will become. Good luck and happy hunting.

DDJ