Better Game Playing using Parallel Algorithms

Twitter Summary: Innovations in playing the game “Go” using Monte Carlo Tree Search and parallel algorithms.

I have a new post up at blog@CACM, “Better Game Playing using Parallel AlgorithmsThe post was inspired by a friend of mine who visited from New York who was working on IBM’s Blue Fuego. Blue Fuego is their  “Go” playing system similar to Deep Blue the chess playing software and machine that defeated Kasparov in 1997.  The interesting part was that this work was an incremental innovation in game evaluation that emerged in 2006  that lead to a major leap in the performance of the “Go” playing software and would lead to the defeat of a human professional by 2009.

I for one welcome our computational overlords.

Hope vs Fear: Privacy challenges in online health communities.

Twitter Summary: The difficulty in creating privacy controls in online health communities represents the struggle between hope and fear.

I have a new post up at blog@CACM, “Hope vs Fear: Privacy challenges in online health communities.”  The post was inspired by all the controversy surrounding Facebook privacy updates which reminded me of  my previous startup which also struggled with defining how much privacy our customers wanted in the website.  My advice is to not post anything you don’t think will be seen by your mom, and at the same time to absolutely post stuff that your mom will see how you are your own person.

The Joy of Building

Twitter Summary: The joy of building and making something successful in any size of organization.

I recently signed on to work for Google as an Engineering Director in its Seattle offices. This move was a little surprising to everyone including myself.  I had been working on consulting with various startups around Seattle, and had an eye towards finding a new thing to build. When Google approached me this time, I was compelled by the opportunity to work with their teams improving its core technologies. There was also the opportunity to explore what else you could do if you had an “infinite” amount of computing resources. What I was happiest to hear in talking with members of the leadership team, is that they all wanted to improve how agile the company was while improving their current system and creating new products.

Generally, I write about lessons I have learned at previous companies for use in a future startup. The lessons for that startup can be summarized as, “Get focused. Find a great team, customers, vendors and investors. Build something great and have your customers buy it.” Funny enough the same lessons are applicable at larger companies.  In a startup, you have resource constraints in figuring out how much people, money and time it will take to build your product. In a large organization, you have the exact same constraints.  The key difference is that in a large organization the scale of the constraints gives you more flexibility in what you can deliver. Organizations like Google have made it their business to build tools like BigTable and MapReduce to provide their employees better tools to improve search. Amazon has been investing in web services and working to figure out how to have it work better for both external and internal groups. The constraints in these types of larger organizations are more often building the right team and setting expectations of when projects can be delivered.

Innovating within a large company has the advantage of being able to leverage the whole organization’s people, time and money if you have a runaway success.  Apple recently reported that 40% of their revenue comes from the iPhone where only a few years ago it was only making desktops and laptops. Amazon’s Kindle was it’s top seller in electronics this past year and has been successful enough that it is drawing engineers from other groups within the company who want to work on the “next cool thing.”  If you have an innovation within a startup you are much more susceptible to having it be starved by a lack of funding to promote it, a lack of time to have it gain traction, or not enough team members to make it excellent.

I am looking forward to working at Google as I will be able to ask the interview question:  “What would you build if you had an infinite amount of CPU/RAM/Disk?” The big reward will be if the candidate is hired and joins the company, I will be able to give them access to an infrastructure that will help them build what they imagine and see if it helps the customer.

Successful Software Platforms are the Byproducts of Successful Businesses

Twitter Summary: Startups need their current business to be successful and self-sustaining before they place serious effort in building a general platform.

I have a new post up at blog@CACM, “Successful Software Platforms are the Byproducts of Successful Businesses“ which was originally brought on by reading startup plans that all highlighted that they would have a general software platform in the middle of a virtuous cycle. In looking at these business plans, I realized that any substantive effort making the platform production-ready, would likely divert efforts from making the company profitable or able to deliver on its original mission. Exactly the type of expenses startups typically don’t have the resources to spend.

In doing a little research for the article, I was pointed to Chris Devore’s article about the value of platforms outside of software development. Examples include franchises like McDonald’s/Subway.  and Avon.

You should name your dogs, but number your cattle.

Twitter Summary: Naming conventions for scaling systems.

Naming machines used to be one of the fun parts when buying new machines to develop software. At one company we named the machines after characters of the Brady Bunch (Greg, Marcia, Jan, Bobby). At the University of Washington, a research group named their servers after fish (Sturgeon, Smelt, Scrod). At, the first servers were named after flesh eating fish (barracuda, piranha, tiburon). The flaws of this approach readily became apparent because we ran out of names when more machines were purchased. The solution is either to add the characters of another favorite TV show or create a scalable naming convention.

By adopting some standardized naming convention, it becomes easier to keep track of the systems. Naming standardization practices typically included adding a prefix of the machine role so that they could be easily grouped in printouts and display (search-, web-, desktop-, db-). Other ways to make machines easier to identify is to put their owner’s name in their name like Apple and Windows do by default for their personal computers. The problem with these fleets is that frequently they became specialty machines or are built in a fashion that they can’t be easily renamed and reused should you need to re-provision the systems for new roles or users.

If you are managing fleets of servers as in any one of Amazon, Google’s or Microsoft’s datacenters, the only option you have is to just naming a random string for each server name to avoid naming collision with a prefix only for the physical location of the server. The role of each of the servers will be allocated based on the role required for each machine based on the demand of the day. By naming the server randomly, and mapping its functionality based on a separate software system you can get the flexibility required to provision and re-provision machines as opposed to tying the machine to some functionality based on the name.

It is all a far cry from having a small number of systems to write software and deploy a web application. However, if you are growing a company for success you should probably have some standards to name the systems you work with daily, and having naming conventions for servers that grow without bounds. In other words, you should name your dogs, but number your cattle.

How Much Software Testing is Enough?

I have a new post up at blog@CACM, “How Much Software Testing is Enough“, about the value of Test Driven Development and the benefits of testing even when you don’t feel you have the time to do it well.  Included with this post is a vignette of one the most memorable bugs in my career as a software developer since it happened on a memorable date and the poetic justice from the book that triggered the problem.

Hiring is Timing

Twitter Summary: Hiring is Timing. The timing of who you need, when you need them, and when they are available.

I have had the joy of helping three friends get job offers in the past few months at two Seattle area startups ( and Offering to help people is nothing new, its just that the last few months have had an unusually high hit rate for matching people with a unique skillset and introducing them to companies that needed those skills now. The whole process reminded me of an adage I was taught at my last startup that “Hiring is Timing.”

Deciding what you need next at a startup is generally simple. If something is not getting done and is core to the business, you should either prioritize the job higher for the people working at the company, or hire somebody to do it. If you don’t have the money for it you either need to raise the money, or generate enough to support the role. Resource constraints (Lack of money) really help the prioritization process. In a software company, I have always heard the rule-of-thumb is that you shouldn’t really need to worry about hiring a full time HR person until you hit employee 25 or in-house legal help unless you are creating revising lots of contracts and have the revenue to support them.

Which person should we hire for the role? We all want to hire “the best” but frequently the best are already employed. This also can be a cop-out for why things haven’t gotten done. “If we only we had this senior software engineer at company Y, she would have totally finished the job by now.” Not everybody is available when you need them, or have an interest in working in your brand new startup. The better solution is to hire people who are available now who have the desire to grow, learn and expand their skillset to build the things you need them to do. I was reminded that is is better for the organization to “Hire well, and develop people” then it is to hire “The Best” []. The best are frequently unavailable, and frankly became the best in the environment they are in and will likely continue their success by staying in that environment. By developing people, any new startup has the opportunity to create a whole new team of “The Best” and creating an environment where they can flourish.
P.S. If you are looking for a software job in the Seattle area, I have contacts with a few startups that are hiring so send me an email and maybe there will be a match.

Why do developers put up with “Crunch time”?

Twitter Summary: Given the well known increased risk of burnout during “Crunch Time,” why do developers put up with it?

I have a new post up at blog@CACM, “Why do Software developers put up with ‘Crunch Time’?“ about managing software developers during the push to launch.

Included with this one is a story about my experience on Amazon Auctions. The blog post was inspired by reading about an open letter posted by the wives of Rockstar games and the months they have been dealing with absentee spouses while they try to release the game, and an article from Harvard Business Review about what motivated workers the most. The serendipity of reading the two the same day was a great reminder of the importance of recognizing the employees are the company’s most important asset.

Lessons in Leadership

This winter was the first time in 40 years that I didn’t spend the holidays and New Year’s with friends and family. Instead, I spent time in Peru as a tourist and one week helping to build a community center an hour north of Lima. The Seattle Times has a great article about the project. If you have some money to donate I would encourage you to donate to the Lake Union Crew Outreach Foundation.

This blog article is about a lesson in leadership that I learned helping build the community center. In watching the lead engineer work and manage the effort, the thing that made the largest impression in making the project successful was her constant involvement in every component of the construction. The project was run as an 8 week relay with new people coming on and off the team every week. The factors that made the project feel successful were directly attributable to the following:

1) Leadership by Example -- The project lead was involved with every element of the construction when she wasn’t helping coordinating the volunteer’s effort. Whether it was lifting every bucket of water into the concrete mixer to make sure the concrete pour was going correctly, or coaching the pouring of the concrete foundation she was always present and visible during the critical portions of the project.  I was doubly impressed that there was no job too small. At the end of the day, she would walk the site and carry lumber, or move rebar to the correct location so that it would be available for the next day’s work.

2) Tolerance for mistakes and re-work — One day we stacked a row of concrete blocks in a row 80 feet long and 5 feet high, next to the wall where they would be placed. Unfortunately, the blocks were set in an unstable pattern, some of the blocks were resting on their sides (making them more susceptible to cracking), and they were 4 inches too close to the wall meaning the scaffolding couldn’t move along the wall to place the blocks.  The only solution was to re-stack the blocks. She started the re-stacking and trained the team of 5 on how to fix the problem. Two hours later the re-stacking was done, none of us would make the same block stacking mistake again. As unfortunate as it was that the mistake happened, the only solution was to re-do it and show us how to fix it.

3) Trust and verify — After everyone had left the site, I watched her wander through the interior of the empty structure with a measuring tape verifying the interior plans would actually work for subdividing the space within the structure. I jokingly asked her if she was seeing the forest through the trees, and she replied “Yes.”  Her experience helped her visualize the internal space, and come up with some last minute revisions that would make the site more suitable for its intended usage, and not what was just laid out on the sheet of paper.  A second example would be that she would delegate portions of the project to experienced and trusted team members, but later walk through and confirm or have adjustments made based on what would work best for the work site.

4) Progress — The beauty of working on construction project is that every day you can see the structure change and get closer to completion. One day there would be no roof, and the next there would be a roof installed. One day there would be a mesh of rebar on the floor, and the next day you would have poured a complete concrete floor. Every day the masonry wall would climb steadily higher, and the window and door frames would get placed.  A recent Harvard Business School study of the notes of 12,000 workers the biggest motivator for keeping workers engaged was progress. Progress was more important than interpersonal support, instrumental support, collaboration or “important work.” Simply making progress kept the team engaged.

There are many valid critiques about the project engineer’s leadership methods. Her communication style chafes individuals who want more collaborative exchanges. That said, no one could deny that the progress of the project was proceeding as smooth as it could using a volunteer work force, sub-standard equipment, and dusty/sandy conditions. In looking at the lessons of the project, it is not hard to see how these analogues work just as well for leadership when writing software and building companies.

Addendum: “It Takes a Team”

The project engineer and lead had the responsibility for delivering a large facility in 8 weeks of work using volunteer labor. She had the support of the Lake Union Crew Outreach Foundation who helped do the fundraising, teams who contacted and scheduled the volunteers for weeks, architects who designed the building, runners who helped coordinate construction resources in Peru, and a core team of members who had built similar structures with her in Lesotho. Her responsibility was making sure that the physical construction of the building and the supporting terraces were done on schedule.  The volunteers were people who self selected for the project. They varied in age and ability from 7 to 60+ years, and with no experience working in construction to having worked with the lead on prior similar construction projects.  The volunteers included up to 24 English speakers and 20-30 Peruvian volunteers who spoke smatterings of English. The volunteer experience included more than just construction, as it also involved shared meals, conversation and singing, but those stories are for a different venue.

Better GPS Software through user feedback

I have a new post up at blog@CACM, “Better GPS Software through user feedback“, about some of my experiences and frustration in using the software for my myriad of GPS devices.

I love my GPS devices and use them because they provide a great utility in either tracking my running and biking or informing me of the location of the closest coffee house. However, in using them its easy to see how they could be massively improved if they leveraged all the other people who use the devices similarly and anonymously shared the aggregate information. Amazon and Google improved their search engine relevance by getting lots of traffic and aggregating user behavior in web space. I believe that software like Garmin’s Connect and Google Maps could benefit through the use of aggregating people’s behavior in physical space.

Customer Joy. Team Success.