Compensation: Transparent/Open systems

Twitter Summary:  Open compensation systems seem ultimately fair, so why do closed compensation systems still dominate?

Compensating engineers and managers in most organizations is a haphazard affair. Most organizations have “pay ranges” where they are provided a window of salary that each employee should be for stock, cash, bonuses, and even the amount of office space allocated for each employee level. If compensation ever felt “unfair”, it was more often caused by the secrecy surrounding the compensation process then any actual structural unfairness in the system. Most software engineers were in a few thousand dollars of each other, and by the time we are talking about 6 figure salaries,  “fair” was really just worrying about pride or status. However, it is hard to make people feel they have gotten a fair offer if the whole system is clothed in secrecy and they don’t feel that their compensation reflects the projects they have completed.

One solution is to have a transparent compensation process that can be inspected by the team and they can help contribute to its ultimate success. It is possible to run the largest of organizations with an open/transparent compensation system as the military and the U.S. government have published all their pay levels and expected benefits. Why don’t more companies open up their compensation for their employees to help contribute?  In my search for open and team based compensation systems my favorite example has been Joel Spolsky’s Fog Creek Software Compensation System. I was lead to his plan by the compelling article “Why I never let employees negotiate a raise”.  The summary is that if there is an issue that requires a raise for one employee, the company should consider if every employee at that pay level should get a raise. Frequently, the issue of a raise is simply the engineer is having some issues with some other aspect of their job. By having an open compensation system, the engineer can understand what it would take to increase their compensation individually, or by making the company more successful such that the profit sharing mechanism benefits them and all their colleagues.

Closed compenstation systems have difficulty getting it right for team based development methodologies such as Agile development. Agile software development practices require a different way to think about how to align team goals with the incentive to launch goals. Mary Poppendieck’s article on Managing People and Projects(PDF) is a great example of what a manager can do if they are trying to help create an appropriate reward structure for a team within a closed compensation system organization. The compelling part, is that if you follow her guidelines it will eventually lead you to a transparent/open compensation system as advocated by Joel Spolsky.

There are ultimately some holes in these compensation systems as they don’t take into account organizations that have roles that have been traditionally incentive based. An example of this is a sales force, that is provided a bonus if they sell a certain amount of the company’s product, however as a whole these seem like a great way to think about creating a company’s compensation plan.

Information Aggregators: A marketer’s dream

Twitter Summary:  Information aggregators have the best chance at getting the right advertisement to the right person at the right time.

Prior to advertising on the web, advertisers were primarily paying for repeated impressions over broadcast technologies, such as radio, billboard and TV advertising. The amount they paid was based on the number of people that saw the shows, heard the radio, or passed by the billboard while the advertisement was displayed.  The metrics that the marketers use to target their customers are viewer age, sex, and zip code in the attempts to influence the viewer when they made their next purchase.

The web changed the nature of advertising by being much more precise then any of the previous methods. My first “favorite” search engine was Altavista. It returned results quickly and it was relatively complete for its time. It made money by selling banner advertising to whomever would pay for the space above the search results.  Advertisers were still paying primarily for impressions following the same strategy as for broadcast technologies, but this time they could get complete information about how many times their advertisements were actually seen by customers.

Google also began with marketing deals to display sponsored results at the top of a search results page based on their customer’s keyword searches.  Their innovation was to eventually only charge if the customer clicked on the advertisement to go to the resulting page, instead of just viewing the advertisement. This new scheme, branded as Google AdWords,  improved advertising results by scoring advertisements and only presenting the ones that were good enough to earn their customer’s clicks. Cost per click (CPC) advertising ensured the quality of the ads were high, and they created a multi-billion dollar business that replaced the impression based advertisements that appeared in the Google search results.

The challenge for marketers in this environment is getting the right advertisement to the right person at the right time.  With customer’s jumping from website to website, and searching for disjoint topics, its difficult to know who the customer is, which advertisements they have seen already, and whether they are spending the money effectively to reach everyone they can.

My prediction is “information aggregation” utilities such as Google Wave, Gist, or the customer’s web browser with integrated email client will be the next best source for complete marketing information. These aggregators could solicit customers for biographical data both explicitly (customer surveys) and implicitly  (observing the customer as they search and click on web links), and selectively share the biographical data with advertisers. This complete biographical view of  the customer will allow advertisers to more effectievely target fewer and better ads to everyone.

The challenge will be convincing the customers that the advertising they get is to their benefit as well as the advertiser’s benefit. There are concerns that this may be considered too invasive to the privacy of the customer as they navigate the web. Fortunately, the trends on the web indicate that younger audiences are more willing to share information via Youtube, Twitter, and Facebook. This would imply they are more likely to share their information with marketers provided they are included in the value proposition and actually get a benefit from the advertisements they see.  The success of Hulu bears this out since they are explicit about the tradeoff of getting on-demand video viewing while being required to sit through commercial breaks.  As people get used to the idea of providing more information to marketers, they will have the ability to get more focused marketing information that could help them make informed decisions in the future.

Interviewing — Early Hires (Co-Founders / First Employees)

Twitter Summary: You are going to spend a lot of time with your early hires. Make sure you enjoy their company.

A special case in interviewing is finding and assessing co-founders and early employees.  The first hires are critical as they will help dictate the direction and eventual outcome in success of the company.  The interview process for early employees is different in that the people you are typically interviewing are referrals or perhaps “friends of friends.” Generally these are people who have been successful in some other organization and your job is to determine if they can successfully help you bootstrap your small organization. Spending extra time with them during the interview phase is especially important as you will be spending many more hours with them getting the company off the ground.

The interview questions in this case are not very different. However, the skill set you are hiring for is broader and personality attributes are much more important.  Interviewing early employees should take more time as you need to assess if they have all the skills you require.  Following up with the referral candidates is also critical. Even if the candidate is not offered or decides not to accept a role they may turn into an invaluable resource for other future hires.

Key traits to look for in early hires are:

  • Can you spend a lot of time with them?
    • Rather than a 1-3 hour interview, spend a full day with each other talking about the business or social matters.
    • Consider taking a trip with a potential candidate to a client site to see if you would enjoy working with them.
  • Would you want to be in a foxhole with this early member?
    • Does the candidate have a wide range of skills or more importantly the enthusiasm to do things beyond their skillset in order to get what needs to be done completed.
    • Consider asking the candidate how they deal with stress, both their own and others? Startups are by definition stressful environments and having a candidate who is self-aware to provide a good answer with examples is really important.
  • How do they deal with ambiguity?
    • All early stage companies lack phone systems, HR resources or people to help with basic maintenance tasks. In the absence of information, a policy or even an ability to do something, what is their approach to make sure the task is completed.
    • If the garbage is overflowing do they (a) Take it upon themselves to do it, (b) complain about the garbage overflowing, or (c) create a schedule to rotate who takes out the garbage until an office manager or someone is hired to take care of it as a permanent responsibility.

This post was inspired by Mark Goldenson’s comment in an earler blog posting.

Search Engine Relevance: User Feedback Loop

Twitter Summary: User relevant search results can only be achieved by incorporating a user-centered feedback loop.

The largest improvement in the relevancy of modern search engines has been the incorporation of the user feedback loop. Historically, search results were evaluated only on the basis of the contents of the documents:  Words in the title were thought to be more valuable then words mentioned in the body of the document, author names were more important than if the name appeared in the subject, and words mentioned earlier in the document were more important then words mentioned later in the document. The net effect was that the search results were limited to weight and score of each document. A user had no input to the scoring function that determined if the document was a good match to their search result.

The most successful search engines learned early on that by using feedback loops to leverage human feedback you could add additional information into an index and get a better search result. Google implemented a “user feedback loop” through the incorporation of PageRank. PageRank took the network of outbound links from websites and weighted pages that they referred to higher in search results. The search engine used the basic weighting of words in title, subject, and header as important, but also included an additional factor that leveraged the human behavior of linking pages to  to websites that they thought were valuable. The result was a search engine that has become the defacto  standard for finding information on the web.

Amazon.com was able to incorporate an even stronger feedback loop by tying together three human activities: the words the customers used to search for an item,  to the items they clicked on, and the items they eventually bought. Their algorithms are exceedingly strong because Amazon’s scoring function involves money. Nothing says something is valuable to a person more than their willingness to pay money for it.  If you present results to a customer that doesn’t produce a sale, that is a sign the algorithm could potentially be improved.

Yelp includes a different user feedback loop in the way it refines a search for distance, specifically, including an option for walking distances.  If picking out a “Thai” restaurant in Seattle, the search results are biased towards good places that are within walking distance to where Yelp currently understands the user to be.  The presumption is that a restaurant that is close by is an important factor in where a user decides to eat. The effectiveness of the algorithm is evident by the fact that the company continues to use it as a default in search results.

In all these cases, a machine could clearly make a decision as to what is a better search result based on just the name of a restaurant, or title of a book.  However, a useful search result for a machine doesn’t require user relevant feedback as it would be meaningless for a machine to walk to lunch or decide whether it wants to buy a book.

If you are building your own search engine, think about what user feedback loops you should incorporate to return a better result.  Great user feedback loops should consider both positive results (a sale being made, a link being clicked on) and also negative results (what does a customer do after an empty search result, or if they are presented a result where they don’t click on any of the links).  The end product will be more relevant search results for you customers, and faster navigation through the website to the customer’s desired pages.

You are what you index

Twitter Summary:  Search engines only return results for items indexed. The more garbage they add, the more garbage their customers see.

Garbage In, garbage out

Search engines can only return results based on the content they index. The items indexed need to (a) match the user’s expectation, and (b) need to be relevant high quality results. This seems really obvious but the implications for building high quality search results are tremendous in that web sites are quickly limited to the type of information they can return. In addition the nature of the items indexed create requirements for information quality that most search engines can’t completely control.

Prime examples of this are:

Search Engine Items Indexed Relevancy
Amazon.com Things People can buy Sales Rank
Google Web Pages Page Rank
Twitter search Tweets Chronological Order

These search engines will break the customer’s expectations if they begin returning results that the customer did not intend to see. If Amazon.com were to start returning web pages or Twitter “tweets” as part of its search results, it would entirely miss the expectations of its customers as to what the search results should provide for them. If Google began returning a page full of “tweet” results, and product information, it would certainly capture more information to index, but it would unlikely return results that would be satisfying to the customer.

The relevancy of the results also becomes a limiting factor for these search engines as they need to spend effort controlling for quality when their basic algorithms return inadequate results. In Amazon’s case, quality of results is impacted by manufacturers creating product names that have typos, or music band names that were intentionally misspelled. It would be easier to just let the misspellings float through the system, but by returning a poor result or not showing the customer a potential match they can reduce a potential sale. In Google’s case, they have to deal with malicious websites that create inflated page ranks for pages through the use of “link farms”,  or websites that are mirror images of other websites that only vary by the URL at the top of the page. Rather then display the same information repeatedly on the same result page, Google spends effort removing duplicate web pages and eliminating rank inappropriately created by a link farm. In Twitter’s case, returning results in chronological order is simple, but a search for “milk chocolate” and “chocolate milk” are actually two distinct searches, and search.twitter.com can’t improve the results without breaking their default time ordering. Twitter also suffers from their user’s typos (or simple pluralization mistakes) that could be remedied, but because of how they return and display search results makes it difficult to fix.

At various companies I have worked, there have been multiple attempts to return search results that returned a variety of types of data. The user interface challenge was enormous and difficult, since that type of results requires the user to know what type of result they should expect to see prior to clicking on the link. In the end, it was simpler to design, create and explain to the customers that depending on which search box they used, they would have the type of result they were expecting. By managing customer expectations prior to the search, it resulted in making the page easier to design and for the customer to use.

Improving data quality is challenging regardless of domain and frequently requires human judgment as the data is typically made by humans for humans to consume. If the search engines were made for just machines to use, we wouldn’t need relevancy as a machine could process all the results.  People using a search engine require assistance in helping figure out which search result matches their query.

In the end, the old rule of “Garbage In, Garbage Out” or in this case, “You are what you index” is tremendously meaningful in figuring out what it takes to return a great search result.

Better Meetings — Openspace Technology

Twitter Summary:  Improve the quality and the outcomes of all your meetings through the use of Openspace Technology and its principles.

I was introduced to Openspace Technology (OST) at Amazon.com and have used it at subsequent organizations with great success.  OST is not “technology” as in gadgets that surround us. It is a process to create meetings that takes minimal work from organizers, and pushes the responsibility for success to the participants who are encouraged to take ownership and solve the issues. The best summaries of OST are the wikipedia entry and the user’s guide hosted at OpenspaceWorld.org.

Rather than highlight how OST works, which is already detailed in the above links, I want to highlight the impact on the participants.  It can best be described in there phases:

  1. Introducing OST —  When first introducing Openspace Technology to a group of people, I typically see  incredulity that this could be a successful way to run a meeting.  The expectation from other presenters that it can’t be successful without having a detailed  agenda and preparing powerpoint slides to lead the group. The expectation from attendees is that it is too fluid to capture what was really important.  Fortunately, those impression change once everyone realizes that this is an opportunity to contribute their critical issues. By stating aloud what they think is important, they can hear other people come up with  issues and help determine the relative priorities.
  2. Meeting Sessions — The self-organizing nature of the meeting becomes apparent as everyone offers session ideas.  As  ideas are contributed, they spawn more ideas that are added to the pool of potential discussions. As more important issues are made known, some ideas are tabled for future discussions.  The impact on the group is they realize that with the limited number of people in the room, they either need to take ownership of an idea, or contribute to an initiative they believe will make the group successful.  The participants become energized as the process makes it clear that some issues won’t be resolved unless someone (a) states there is an issue, (b) takes ownership of fixing that issue, and (c) finds others who think the issue is important and can help resolve it.
  3. After-effects — The after-effects of running an OST meeting can be great and frustrating. The impacts of running a meeting is that it energizes the participates, distributes ownership, and creates buzz around the important issues. Really important issues get lots of support and can enlist help from a broad base. Frustrations that emerge stem from either (a) having a long list of remaining tasks that may not get addressed, and (b) the team feeling the company is not pursuing the priorities as generated from the meeting.  Frustration can be managed by capturing and prioritizing the future work and ideas generated at the meeting.

The principles of OST work in smaller non Openspace meetings. I often would run my weekly engineering meetings much the same way.  I would send out a minimal or blank agenda prior to the meeting,  soliciting the team for topics that they wanted to share in the group setting.  If a team member knew of a topic that should be discussed, there was a regular forum where they had the responsibility to surface issues.  If we had an empty or light agenda the meeting would be skipped or shortened. If the agenda was full we would need to decide what was important for that week and re-schedule other issues for another time.

Having a shortage of time or resources is a fact of life in most organizations. Using OST prinicples to manage the list of things to do helps focus on getting the most important things done in the time alotted.

Performance Reviews — Family and Friends

Twitter Summary: Some of your most important performance reviews don’t happen at work.

Performance reviews come from everyone in your life, its just that work is the only place where they have formalized it and based your compensation on its result.  I frequently joke that every year my family should give me a “How am I doing?” customer service card.

Many years ago I was approached by a colleague who was concerned that his performance review at work for the year would not be up to his previous year’s reviews.  He was intending to take the maximum amount of time he could take for paternity leave, but also wanted to keep his standing as one of the top engineers in the company.  His concern was how would he be able to maintain his high ranking while still keeping his commitments at home.  I was about to offer a suggestion to help, but realized that the best advice was to make sure when he went on paternity leave, he actually took the time off.

Your work performance review is important enough to rate the time and energy to create a formal process to collect information, assess performance relative to others, and make recommendations for future improvement.  Unfortunately, a 360 review encompasses only your performance at work, it doesn’t cover your performance as a father, friend or spouse.   I imagine our perspective about the importance of our yearly work reviews would be much different if the 360 review included a) your kids review your abilities as a parent, b) your friends rating of how much their lives are made better with you in their life and c) your significant other reivews your contribution to their life.

Performance Reviews — Keep it focused

Twitter Summary:  Short-term focused performance reviews can help you deliver on tangible results.

I have a personal loathing for the yearly multi-page performance review where: A) All an employee’s past year’s work is reviewed, B) All their work for the upcoming year is planned and mapped to company, organization, and team objectives,  C) All their personal development tasks for the year are listed, and D) 360 review feedback is summarized, E) Everything is distilled to a single letter/number/score, and F) Becomes the basis of the employee’s compensation, bonuses and future path within the company.  The process is difficult and drawn out for the managers who have to assemble the information for their team. It can take weeks to assemble the information and craft a review and schedule a meeting for each  employee. The process is also difficult and drawn out for the employee who has to wait until everything is assembled to find out how the company thought they performed this year. This is further complicated by the opacity in what is going on with the reviews as it is taking so much time to do. The management gurus at Harvard Business School highlight that performance reviews are imperfect and are looking for better ways to improve the process suggesting that the best way is “continuous performance reviews”.

In startups and agile organizations this approach is especially problematic as the goals for the team and company can change over the span of a few weeks. Rewriting documents for each employee so that they can have the benefit of an accurate assessment for the employee’s performance review adds even more time to organizations that are striving to be nimble.

The best process I have seen to date has been to a) de-couple all the reviews bundled into the yearly review and spread them out through the year and b) set the employees expectations as to the goals of each review element and how it impacts their career growth.  The types of performance reviews that should be split throughout the year are:

  • Company and team performance reviews can happen quarterly or bi-anually so that management can set expectations if the company is doing great, or if it is struggling to keep payroll and what everyone can expect in terms of compensation and future tasks.
  • Employee performance reviews that happen quarterly can cover what happened last quarter and what needs to happen in the next quarter for the employee and the company to be successful
  • Personal development reviews can happen 2 to 4 times per year covering what the employee would like to learn to do their job better and what the team would like to help deliver for the customer.
  • Compensation reviews can happen 1 to 2 times per year, on a separate cycle where all the previous performance and personal reviews for the past year are assembled and the employee’s contributions and the impact of the team’ and company’s work can help reflect what types of compensation adjustment are necessary.

By splitting out all the reviews into smaller more frequent reviews you gain the benefits of:

  • Less work for managers and employees who can focus on deliverables for the next quarter.
  • Smaller focused reviews are easier for managers and employees to write-up and explain.
  • Manager and employees have more frequent checkins as to employee’s performance in the context of the team and the company.
  • Personal development tasks are given priority as it is not hindered by people thinking about their performance or compensation bundled in the same document.
  • Decoupling the compensaton conversation from multiple performance evaluations helps the manager set the employee’s expectaions as to how much of an adjustment can be made.

The role of performance reviews is important in helping make sure that the team is focused on the correct things for the company to deliver on its goals.  By setting the employees, managers and company’s expectations for each performance review element covers, you increase the transparency in how the company will succeed.  This advice does take some extra scheduling steps for managers, but spreading the tasks throughout the year ensures that there is constant communication of managers with employees.  In addition anything that takes a company away from a 20 page per employee review is likely to be less onerous, and help move towards a clearer and better run performance process and happier employees.

Artifacts of Success

Twitter Summary: Recognizing individual contributions with artifacts of success are a part of any team culture. Make sure to do it right.

Physical artifacts of success have been around as long as people have been cooperating in accomplishing goals.  Hunters collect trophies of their first kills. Microsoft programmers get “ShipIt” awards. These physical mementos are typically very group sensitive and can be very elaborate especially when you compare the United States Military awards and decorations to the simpler but more permanent Maori tattoos. Despite the differences, their purpose is so that groups can identify their own and have an idea of how each has contributed to the benefit of their group.

Most companies I have worked for has had an analogue to this from t-shirts and laminated sheets of Amazon Patent Blockspaper for product launches to Lucite plaques for creating patents.   In the best of cases, there are pretty clear requirements for awards. To get a patent block, you must have a patent application.  In the worst case, it can feel like the mementos are just being handed out to anyone who is standing nearby or to curry favor.  This can cause resentment among the team members and create disincentives among the very people who you want to show your appreciation.

With the potential issues surrounding physical artifacts of success, it is tempting just not to create any at all.  However, every group develops its own symbols to self-identify. If you can come up with a clear simple rule set for how these artifacts are given out, you can maintain the integrity of the mementos and give people recognition for how their efforts have helped the team.

Qualities for creating a meaningful physical “Artifact of Success”:

  1. Goal based — Give “artifacts of success” only for contributions that help with company’s goals. Product, project launches that create substantive value and took focused organization effort for the company should be recognized. Don’t recognize things that are nice, but not distinguishing for the company.
  2. Limited — The mementos will only be created for the few that achieved the goal or contributed for the benefit of the team. If necessary, create a simple rule to identify who gets things and who doesn’t. Don’t devalue the identification by handing them out arbitrarily.
  3. Inexpensive — By using inexpensive tokens (paper, baseballs, golf balls, origami birds, colored pins), there is no mistaking that the purpose of this is recognition not compensation.
  4. Portable — As much as I love a good t-shirt, I can’t carry all of them with me, so having a jar of launch golf balls, or tokens that I can post on a wall or hold on a desk will suffice.
  5. Sincere — I am always a fan of the memento listing: The name of the event, the date of the event, and being cross-signed by all the team-members at the launch party, or by the CEO/organizational lead for the company.
  6. Composable, stackable, groupable — If the physical artifacts of success can be composed, similar to military badges, Maori tatoos or Amazon’s patent blocks, you create a compelling visual effect that enhances the value when you have sets of successes.

By spending a little time on what qualifies as something that deserves “recognition” and what would be team appropriate, you should be able to create your own visual shorthand to recognize team members contributions to the company.

Interviewing — Hire or No Hire

Twitter Summary: A great hire will often generate lots of energy during a hiring meeting and some of it negative.

Once you have established whether a candidate passed your interview,  the final step is to have a hiring meeting where all the interviewers provide their assessment of the candidate.  The hiring meeting is important to attend in-person for multiple reasons: a) You are compelled to make a decision as you have a scheduled time and duration to decide on a candidate, b) You can confirm or provide counter-evidence for skills or behaviors people saw during their interviews, and c)  You can calibrate with your team whether you are all asking the right questions with the right degree of difficulty.  Email doesn’t work in this context as in-person dialogue allows the team to quickly clarify questions among all interviewers.

How do you decide if you are going to hire someone? The best technique I have seen to date has been requiring unanimous consensus by all the interviewers that the candidate should be hired. The tremendous flaw of this style of decision making is that it will reject candidates who could have done well at the company, but because of a sub-par interview, they are eliminated from consideration. The incredible advantage of consensus driven hiring is that everyone who is hired has had at least six people say “Yes” to hiring the candidate. If all of your colleagues have gone through the same process to join the company you should have more confidence that they are doing the right thing. If  a co-worker begins having difficulties, there should be a sense of ownership by the team that they picked the candidate for a reason and they can help them work through those issues.

It is important to note, that consensus means that everyone agrees the candidate should be hired. It does not mean that everyone likes the candidate. Some of the best employees I have ever hired had at least one or two other interviewers indicate they were not inclined to hire. There have been cases where a college intern was determined to be a great hire by two of their interviewers, and a strong no-hire, by their other two interviewers. This intern was fortunately hired and ended up being an exemplary hire for the company as the advanced and took on more projects quickly. It was fortunate that in these cases the candidate was able to express a deep technical capability and being a “superstar” in some fashion that an interviewer was willing to champion them  in the hiring meeting to take a chance on a hire. The dissenting interviewers shouldn’t ever be bullied into hiring a poor candidate, but they should be open to listening to the interviews of others and decide if it is worth the risk of sharing a payroll with the new hire.

Some of the most questionable hires have been hiring meetings where the candidate did well enough that everyone was inclined to hire, but no interviewer was able to figure out where the candidate was great.  The problem with hiring these candidates is that they frequently have a similar career within the company of doing good but not great. Finding a passion or the superlative associated with each candidate means that if they decide to join the company you will increase the overall depth of your company’s employees and the probability that something great will emerge.

Customer Joy. Team Success.