Twitter Summary: The questions you pose to interviewing candidates reveals if your job is interesting or boring.
The questions you ask and your assessment of the candidates answers comprise the core of the interview. More importantly, excellent candidates will be paying attention to you and will be assessing you just as much as you are assessing them. If the questions you ask are the same the candidate has heard in other interviews, seen in interviewing manuals, or from another interviewer that day, they will have a clue that your interviews are not organized, your questions aren’t relevant, and that you may not be capable of capturing the depth and breadth of their capabilities.
Great interview questions have the following qualities:
- The problems you ask them to solve should either be ones that you are familiar with or are currently trying to solve yourself. This gives the candidate a clear indication of the type of work they will be asked to perform. This also gives the interviewer the ability to assess how the candidate would solve a problem that is pressing for the organization right now.
- If you are interviewing a software engineer for a kernel development position, ask them to code a scheduler and talk about memory allocation.
- If the candidate is interviewing for a social networking site, ask the candidate to generate code to generate a user to user similarity score.
- If the candidate is interviewing for a marketing position, ask them how they would evaluate marketing options for your current product.
- Don’t ask a “textbook” questions. — For software engineers, this means asking them to implement “atoi()”. For anyone it would mean questions like: “What is your greatest strength?” Your questions should be open enough that the candidate reveals their strengths in the process of describing another problem they solved.
Expandable in depth and breadth
- The questions you ask should be expandable in either breadth or depth. For software engineers, this would mean asking them to take their solution and and scaling to handle millions of users or machines, or alternately asking them to change their solution to handle multiple, media and content data types. For product or marketing candidates, you could ask them how they would modify the programs or projects to deal with if you wanted to handle a larger diversity of customer groups or a broader product line.
- Don’t ask questions with only one right/wrong answer. Asking questions that have only one correct answer is problematic as it doesn’t give the candidate the opportunity to discuss trade offs in choosing and decision making. In the end, you are hiring people to make judgment calls, and are not hiring a computer to give you a rote answer
- Present only the basics of the problem. The interview should be a dialogue, and interviewers should intentionally give less information then the candidate needs to solve the problem. This will get require them to ask you clarifying questions and give you insight as to how they approach their problem solving.
- Don’t spend all your time talking. If you spend most of the time talking you are not getting the information from the candidate you need to make your decision. You can spend more time talking with the candidate after they have answered your question, and especially if they are hired.
Great inteview questions make your job as an interviewer easier . A great set of interview questions will allow you to understand how the candidate would solve the problems you need to solve in your company, how the candidate deals with challenges as the problem expands, and how they communicate and ask for more information in answering the question. It also tells the candidate about what problems they can expect to be solving and if the role is a match for what they want to do.
Twitter Summary — If after interviewing a candidate, you are unable to make a Hire/No Hire decision, you have failed as an interviewer.
Interviewing candidates involves making a judgment call about the person you are interviewing. One thing that makes this difficult is that there are many resources (web based, books, college or career counselors) who can give a candidate the “right” answer for just about any question an interviewer can ask: “What is your greatest weakness?”, “What is your biggest strength?”, “Where do you see yourself in 5 years? 10 years?” Admittedly, offering this information is an attempt to help others who are interviewing answer the questions “well enough,” but often mask what you really want to know about the person. This puts the interviewer into a situation where they will not be able to make a clear decision on a candidate. When candidates answer all the questions “well enough” they run the risk of hiring someone that will only do “well enough” at their company.
My biggest difficulty in interviewing is when I find I personally like the candidate and think they would be great to know outside of work based on shared personal interests that come up in the course of our conversation. However, shared personal interests is not a determining factor as to whether a candidate can match all of the expectations of a role. The sole job as an interviewer is to form an opinion of a candidate and decide whether they are qualified for their role, and will excel in that role for the company.
The keys to success in interviewing a candidate are:
- Be present — Make sure that your phones and computers are off, and you have space where the door is closed. Distractions that pull you away from focusing on your candidate shortchanges the candidate and your opportunity to make a clear decision. As most interviews last only 45 minutes to an hour, your undivided attention is important in assessing the candidate.
- Assess for your “hidden agenda” — We all look for the same attributes for a candidate: Are they smart, punctual, enthusiastic, hard working? You cannot ask those questions directly, as the answer anyone would give to them would be “Yes.” Instead, you should ask questions like: “What was your favorite project?” and in their response you listen to see if the project was successful and required smarts, or enthusiasm, hard work or punctuality to solve.
- Communicate using multiple methods — Require the candidate to communicate using graphs, words, speech and even assess non-verbal communication, regardless of role. As a future employee, the candidate will need to work with many people in the company and needs to be able to communicate to anyone using whatever technique is most important at the time. By assessing the spectrum of the candidates ability, you will hire people who can express themselves and represent well within the company. The non-verbal communication is important to assess the confidence of their responses. If they are not confident in their responses with you during an interview, they will lack confidence when communicating with their colleagues
- Induce Stress — During the interview you should create some conflict that the candidate needs to justify so that you can see how they manage stress. Stress is a fact of life at companies and is important to assess during the interview. The stress doesn’t need to be excessive, but by requiring the candidate to deal with stressors on their feet you can get a good idea of how adaptable they are to the environment. The most appropriate ways I have done this during interviews have been to either (a) repeatedly change the requirements of the system (steadily increasing the scope of the project as they answer each part successfully), or (b) express to the candidate that I think there can be a better solution using technology X, and to justify their use of technology Y. Get the candidate to discuss the tradeoffs and assess the impact of the stress.
- Choose your questions well — The questions you select for a candidate need to be the ones that allow you to make a “Hire” vs “No Hire” decision within the allotted time frame. If you are not asking questions that get you to the point where you can make a hire decision, you are asking the wrong questions. Don’t waste the candidates time or yours by asking questions that don’t help you assess a candidate.
- Look for the superlative — The thing that distinguishes the “Good Enough” candidates from the “Must Hire” candidate is typically a superlative where the candidate was the “Best” at something. Typically this comes up when the candidate mentions they were the “go-to” person for certain areas or solely responsible for certain pieces of technology. Assess why they did so well in those areas and make a determination if that would be important for your company.
- Take Notes — If you interview many people a day, you should write-up your notes immediately afterward so that you don’t confuse candidates and can discuss the candidate during the interview debrief. Notes also help to re-enforce a second pass assessment of the candidate to see if they matched the requirements for your “hidden agenda” or had a “superlative” associated with their prior work experience.
Interviewing takes lots of focused listening, but once you have identified a candidate to interview they deserve the attention you give to them. This attention is how the candidate assesses your non-verbal communication and gives the candidate the confidence that a “hire” or “no-hire” response from you company was the correct decision for you and them.
The next in the series will cover “What” you should ask during an interview.
Twitter Summary: You have more leverage to influence the direction of a company by the people you hire then what you do.
In my career so far, I have interviewed well over 1000 people. I developed a reputation for being good at it which meant I was interviewing software engineers, product managers, engineering managers, VP, CTOs, and even prospective new attorneys which is far out of my core expertise. My job was to assess their ability to be good at their position and to help figure out if they would be successful within the broader context of the company.
I frequently get groans from engineers about how much of their time interviews consume. In fairness, a badly scheduled interview will fragment a developer’s day to be unusable and developer’s want to spend the time doing what they were hired to do. Unfortunately, a well run interviewing effort typically takes multiple hours of time if you have:
- Phone Screen — Assess if there is a match to the position you are looking to fill.
- Pre-planning — Scheduling and assigning interview roles and competencies to assess.
- Interview — Spending the time to figure out if this is someone you would like on your team.
- Debrief — Write up of interview and discussing whether the candidate is a match.
In spite of the time sink, as an employee you should be on as many interview loops as you can handle and make time to schedule. The benefits of interviewing are tremendous to both your quality of life and the future success of your company. The two reasons to do this are:
- You get to decide who works with you! — Never are you more influential in your own work life then when you get to decide which colleagues you will work with daily. That interview process is crucial to figuring out: Is this candidate smart, innovative and adaptive to your environment? Can this candidate help me solve my problem? What about if the problem occurs at 2AM? Will this colleague help me make a right decision and support me, or will the colleague argue with me when I am wrong and explain it in a way I can understand and agree?
- You are helping direct the compass of the company. — The people you hire are the mission statement of the company. By hiring people that not only have a talent for their role, but also have abilities in other domains you get to broaden the reach of what your company can do. If you hire only system kernel engineers, you shouldn’t be surprised if the next innovation is a recommendation to rebuild a new O/S. If you hire framework developers, you will inevitably be requested to approve a new framework that will help develop your product. If you hire a diverse set of talent that can do their jobs, and also innovate in new directions, you will be able to innovate in many new directions. The abilities of the people you hire, will help dictate what will be considered the “core competency” of your company, and which features will be outsourced to different companies.
Yes, interviewing can be time consuming. However, you can assist in planning your schedule so that interviews are less intrusive, because the benefits are immense for your day-to-day work environment and the future success of your company.
Future posts will cover, How to interview, What your interview should cover, When you should interview.
Twitter Summary: When your see a problem that requires you to spend time, money or space. Spend money or space, because time is the only one you can’t reuse.
A classic tradeoff in computer science is the time vs space tradeoff in executing an algorithm. As mentioned in an earlier blog post (An Introduction to Search Engines). If you spend extra disk space storing index data, you can improve the speed of a search by orders of magnitude. The decision here is clear cut. It does take some extra over-head to manage the indices, and update them as the data changes, but the benefits to the customer are immense. The other practical advantage is that as the cost of disk space has plummeted, all the overhead is the management of the data and very little of what was actually stored and computed.
In the startup world, the tradeoff presents itself as “Build” vs “Buy”. Should we build a website from scratch or buy a content management system to display our information? The decision here is ocasionally not as clear-cut and depends on what you want to be the core expertise of your company. If you have a company who’s core expertise is selling cars, investing in creating a new web framework would not be to the advantage of getting sales quickly and is likely to add unnecessary expense. If your expertise is a new web technology that allows you to experience a car prior to driving (and closes more sales) that is where you should consider building your own technology.
When bootstrapping a company, the best decisions I have seen is to fit together as many pre-built components as you can, even if they are your presumed “core expertise”. Only when they begin to show that they are not flexible to your core expertise, should you consider replacing them with some newer, better, faster technology built by your own team. The time saved getting your product to market can help you evaluate whether you have a market for what you want, and you can later decide where your time is best spent in buliding for your customer.
In talking with other technologists and entreprenuers, the wisdom has generally been “fail fast”, so that you can limit investments down dead ends. You can summarize this as spend the money or take the extra space to try something rather then waiting to build it yourself. The time spent will be minimized and your can always pursue the next better idea if the current idea doesn’t work.
Time vs Money/Space is not a correct tradeoff, as they are not interchangeable. With Money and Space, you can always re-use it, loan it or find somebody who can get it for you cheaper. When it comes to Time, you are not getting it back.
I frequently joke that working at a startup is a great analogy for dating.
It generally starts hot and heavy and you don’t know if this is going to be a great relationship or the ex from hell. Having experienced both extremes with startups, I recommend that you take each of them with a light attitude, a determination to try and make sure to enjoy the ride. If you find that one isn’t working out there is no shame in being honest and ending it quickly and moving on to the next startup. Once you have seen enough startups, you begin to find the patterns of the ones that appeal to your sensibilities and which will keep you engaged to the best of your abilities.
Recently the analogy showed a little more depth when I was discussing with friends about most Web 2.0 websites tried to use the same techniques to get customers that Pickup Artists (wikipedia) use to get dates.
- Quickly identify who in a crowd you would like to meet.
- Send lots of email to people who could use your website.
- Introduce yourself and quickly demonstrate that you’re something special/have unusual value
- Create a special welcome landing page for first time visitors telling them about yourself. Make sure website design communicates correct image.
- Get them to state an opinion about anything to see if they are interested in talking. If not, move on.
- Collect any information from the customer via clickstream, keyword analysis, or customer provided poll information
- Tell them something about themselves that they don’t know and imply you can tell them more if they hang around.
- Create personalization based off collected interactions and tell them how much better it would be if they keep using the website.
- Take them to somewhere private and create trust
- Communicate private information using SSL protocol
- Get their email address.
- Set their expectations you will contact them later to meet again.
- Create permission based marketing email to entice them to interact with your website some more.
- Ask them to invite their friends to meet you.
- Provide incentives about how much better the website is by encouraging all their friends to join.
I don’t know what is scarier, that the analogues are so close, or that nobody has done a more systematic analysis as to which of these techniques are successful.
Don’t hate the player. Hate the game.
Businesses are created and run for many reasons beyond “to profit” or “to make money.” My favorite write up on this was done by Richard Feld’s “What is the purpose of business?” who argues that businesses are created primarily to provide goods and services to a population. Profits are important as they are useful to keep the company continuing, but not the purpose of a business. The quest for profit outside of the purpose of an business risks the loss of direction for the company and the loss of the business itself.
Profit for profit sake contributed to the downfall of several large financial institutions throughout 2008. Banks and investment organizations provide loans and high quality investment advice to sustain themselves. Many of these organizations took larger risks in lending what people now realize were lower quality investments for the sake of an increased rate of return. These organizations bankrupted their purpose in offering poor quality investments, which ended up bankrupting themselves and their investors along the way. Surprisingly, banks also increased service and usage fees, for which there was no tangible benefits, and further alienates customers. The increasing quest for profit from banks from the same population they are supposed to be benefiting will most likely end up driving customers away to other alternatives.
A no less expensive but better customer experience and clearer purpose is that of a casino. In contrast to banks and investment firms, casinos are explicit in how they are making their money in exchange for entertainment. Their service is entertainment via dining, drinking, gambling, and unique experiences, in exchange for a percentage of their customer’s cash. The casinos usually request the money either up-front in the case of gambling or provide a price list so that the customer can make the informed decision of what they can or cannot do. In this model, the customer is aware of the direct costs and benefits to themselves. From a customer service and experience viewpoint, the casinos, although often lambasted for taking money from customers, is a more honest profit model and is in line with their purpose, entertainment.
Creating a product or service that fulfills a community need is why businesses should exist and working with a business that matches your goals benefits both you and your colleagues. A team with a shared purpose is why we continue working in groups despite the hassles of scheduling, personalities and annoyances that come along the way. If the company can be profitable in its purpose it should be able to sustain itself and provide continued value to the community. As an employee and investor, you should select the companies you want to support explicitly. Whether that company is a bank or a casino it is usually best if they stick to their purpose and continue to create value for their community.
One of the best things you can do for you career at any company and across all your companies is a keep a personal copy of every email where someone says “Good Job”, “Great work”, “Your absolutely right”, and “This team delivered the impossible”.
The purpose of keeping a separate folder for these types of email is two-fold:
- Every year when you have your annual review its great to add direct attributions that come from your colleagues, peers, and manager complimenting your work throughout the review period. It especially helps if you can counterweight your inevitable mistakes with praises from the CEO on your work on behalf of the company. ,
- Every once in a while you will have a difficult day for reasons you can control and many that you cannot. When those bad days happen, having a folder full of affirmations of all the spectacular things your have done before really helps. 
If you find you aren’t getting enough compliments and praise then you should either change your approach or change to a role where you can be successful.
 “Ruben, you rock!” — Jeff Bezos.
 “I can’t congratulate you enough or say enough about how I feel about this team. I worked with many good teams, but nothing like that, not even close. I love you all! Now let’s get back to work…. ” — Udi Manber
 “You gave me some really great advice last year. I’ve been saving the praise I’ve received for my annual review. While it’ll be great for that, it’s also just great to have for the occasional work-related rainy day. Thanks, Ruben ” -kim.
Giving technical presentations to curious people is a joy. You work harder to get the facts correct and make it compelling. This is especially true if the audience is unfamiliar with the domain. This presentation was created for high school students who are learning programming
The hardest piece is explaining to them that getting them the “Best” result for a two word query is difficult as it depends on factors that most search engines can’t possibly know. The decision the search engine must make immediately is if my query for “bike repair” means that I want a local bike repair shop or a manual on how to repair my bike. Without the context of what I mean by “bike repair”, the search engine will have to make a best guess based on what other people think are good results. Fortunately, this is working ok now, but with the massive growth of data and types of data this problem will only get more difficult.
Digitizing the “ambient information” of my daily life would likely lead to better search results if the search engine could get the data ahead of my search query. Most of my search engine queries are actually rooted in a conversation I had immediately prior to sitting down at a keyboard to type in my query. If the microphone on my cell-phone, computer or Kindle could transmit that conversation to the search engine, the system may be able to guess what I am looking for before I type anything. A working system would likely return great results, and we would be one step closer to the creation of the science fiction artifact of the “Young Lady’s Illustrated Primer” as imagined in The Diamond Age