Search Engine Relevance: User Feedback Loop

Twitter Summary: User relevant search results can only be achieved by incorporating a user-centered feedback loop.

The largest improvement in the relevancy of modern search engines has been the incorporation of the user feedback loop. Historically, search results were evaluated only on the basis of the contents of the documents:  Words in the title were thought to be more valuable then words mentioned in the body of the document, author names were more important than if the name appeared in the subject, and words mentioned earlier in the document were more important then words mentioned later in the document. The net effect was that the search results were limited to weight and score of each document. A user had no input to the scoring function that determined if the document was a good match to their search result.

The most successful search engines learned early on that by using feedback loops to leverage human feedback you could add additional information into an index and get a better search result. Google implemented a “user feedback loop” through the incorporation of PageRank. PageRank took the network of outbound links from websites and weighted pages that they referred to higher in search results. The search engine used the basic weighting of words in title, subject, and header as important, but also included an additional factor that leveraged the human behavior of linking pages to  to websites that they thought were valuable. The result was a search engine that has become the defacto  standard for finding information on the web. was able to incorporate an even stronger feedback loop by tying together three human activities: the words the customers used to search for an item,  to the items they clicked on, and the items they eventually bought. Their algorithms are exceedingly strong because Amazon’s scoring function involves money. Nothing says something is valuable to a person more than their willingness to pay money for it.  If you present results to a customer that doesn’t produce a sale, that is a sign the algorithm could potentially be improved.

Yelp includes a different user feedback loop in the way it refines a search for distance, specifically, including an option for walking distances.  If picking out a “Thai” restaurant in Seattle, the search results are biased towards good places that are within walking distance to where Yelp currently understands the user to be.  The presumption is that a restaurant that is close by is an important factor in where a user decides to eat. The effectiveness of the algorithm is evident by the fact that the company continues to use it as a default in search results.

In all these cases, a machine could clearly make a decision as to what is a better search result based on just the name of a restaurant, or title of a book.  However, a useful search result for a machine doesn’t require user relevant feedback as it would be meaningless for a machine to walk to lunch or decide whether it wants to buy a book.

If you are building your own search engine, think about what user feedback loops you should incorporate to return a better result.  Great user feedback loops should consider both positive results (a sale being made, a link being clicked on) and also negative results (what does a customer do after an empty search result, or if they are presented a result where they don’t click on any of the links).  The end product will be more relevant search results for you customers, and faster navigation through the website to the customer’s desired pages.

One thought on “Search Engine Relevance: User Feedback Loop”

  1. One other type of user feedback is asking the user to alter or add to the result. Wikipedia makes this most prominent, but most sites have this to some degree (yelp, 43 things, wolfram alpha…).

    Even google has their “SearchWiki” thing, but it supposedly doesn’t affect other users.

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>