One of the unique interplays between psychology and language is the way in which we communicate using examples. If we were talking about a football game, for example, we would tend to focus on the highlights: the made catch, the missed opportunity. But in reality, the game is a series of plays, each slightly different but driving to the same conclusion.
Science is much the same: while we sometimes like to talk about the highlights, we arrive at the truth only through repetition and expansion. So five months ago, when we released the results of a study that showed that people preferred Bing to Google for web search results nearly 2 to 1, it made quite a few headlines. But as scientists, we were already thinking about new tests and new information.
The experiment we did first was fairly simple: participants entered a query of their choosing, got side-by-side results from both Bing and Google but with all branding removed, and picked which one was better for them (or declared it a tie). Each person repeated it ten times and their votes were totaled to determine an overall preference.
So today, we wanted to share our latest addition to this line of study. Previously, we let participants choose their own queries to make it feel as natural as possible. But that raised the question “what if people are just searching for really weird things, instead of searching like they normally do?” To address that, we’ve done a fresh study with a new twist.
First, a few factual details. So that we could avoid any bias from Microsoft researchers, the study was conducted by an independent research company (Answers Research based in San Diego, CA) using a representative online sample of nearly 1,000 people, ages 18 and older, from across the US. To make sure they represented typical searchers, the participants were chosen from a random survey panel, were required to have used a major search engine in the past month, and had no idea that Bing and Google were specifically being tested, nor were they told Microsoft had commissioned the study. And we used just the web results pane only, so no ads, no Bing Snapshot and Social Search, no Google Knowledge Graph.
Otherwise, the experiment was much the same as our earlier study, but with one key difference: instead of coming up with their own search queries, participants chose from a list of five possible queries (drawn randomly from a longer list of queries, which we’ll get to in a second). If none of the five appealed to them, they could refresh the set anytime. After selecting a query, it was the same as before: side-by-side results with no branding, pick a winner or declare a tie, repeat ten times, sum for overall preference.
So where did this long list of queries come from? This was the tricky part. We wanted queries that matched what people typically searched for, so we finally settled on using terms from Google’s Zeitgeist 2012, because while we could have used our own Top Searches of 2012, we figured the right thing to do was to go with our competitor’s terms. After all, you’d think Google would be better at their own top queries, right?
Wrong. In a blind test, people preferred Bing to Google for the web’s top searches. And that is just based on pure web results, so no ads, no Bing Snapshot and Social Search, no Google Knowledge Graph. Even taking away some of our most innovative features and with the handicap of using Google’s top search queries, Bing still comes out on top, with 52% of people preferring Bing’s results over Google’s, 36% preferring Google’s, and 12% choosing Bing and Google equally (for those that favor discarding ties, that’s 60% Bing, 40% Google when people had a clear preference). For the especially geeky, all those numbers are +/- 3% at a 95% confidence level.
Because Bing Your Brain is meant to offer a scientific perspective on search, you’ll notice this post is loaded with a lot of details about how and why we’re investigating just how good our search quality is. But you don’t have to just rely on this new study; you can do your own personal experiment at BingItOn.com anytime and see which search engine gives you better results.
– Matt Wallaert, Bing Team