Bing vs. Google

Ok, so I found it amusing that they don't include Blekko, I mean we did make this into a fun game with our three card monte game with our /monte tag in search results :-) And I suppose it doesn't really serve their interests to 'lose' to a little guy either.

That said, its an interesting experiment. And at blekko.com we've been running for over a year [1].

What we've found is that there is a huge brand bias, which is to say that if you use some search engine as your primary search engine, you tend to think that is the best one regardless of whether its Blekko, Bing, or Google (or even DDG although DDG is more a search utility rather than a search engine as it doesn't have its own web index).

But 'quality' is also a very subjective thing as well. So if you search for highly SEO'd categories you will find the Blekko and Bing do better than Google mostly because there is a curated input (in Blekko's case it was from day one with it's from slashtags and in Bing's they started doing outsourced curation (putatively after seeing how effective it is in Blekko :-) in some topics with their 'editors' program [2]) At some point Google will realize what Bing and Blekko have which are that the 'indexing the web' problem became the 'filtering the web' problem when the signal to noise ratio started decreasing in about 2005, and that the only viable weapon at the moment for human on human spamming (this is where real humans are working for 5 cents a page to write web pages that draw hits) is human judgement.

If the Bing guys are reading (and I know you are) why not open up your challenge to the new kid on the block, we don't just do Blekko vs Google in our monte results :-)

[1] http://techcrunch.com/2011/06/21/be-the-mark-in-blekkos-3-en...

[2] http://www.bing.com/community/site_blogs/b/search/archive/20...

As a Google shareholder, I really want to know whether Bing search results are better than Google. And I really, really want to know what the trend is over time.

However I don't think Microsoft's study is any good. They say, "In the test, participants were shown the main web search results pane of both Bing and Google for 10 search queries of their choice."

So it seems that the study participant had to think of 10 search queries, one after the other. But this is not how search works. In the real world users need to perform one query every so often, and their choice of search terms is based on an immediate need rather than what comes to mind.

A good Bing vs Google study would ask 1000 people to use a split-test search engine for their browser search box, then monitor the results over time. I've asked HN about this before and got nowhere: http://news.ycombinator.com/item?id=3966121

So is there business opportunity here? I imagine that there must be thousands of Google shareholders willing to pay £100 per year for the results of an unbiased, ongoing study into who gives the best search results. In fact, why doesn't Google set up the study and provide such information to shareholders?

Google, 4 to 1.

Now, all of my searches were technical; I plan to try again later with more general, "normal people" searches. Bing may do better there.

Also, good results from even google searches have become harder to come by in the last year or so. I have to work harder at my search terms. I don't know if that's Sturgeon's Law at work, or if there's room for a search engine to improve enough to become better than google.

Perhaps targeted search engines are the answer (i.e. engines's who have both their search and crawler algorithms tuned for a specific target audience) are the answer?

Maybe I was imagining it, but four out of the five I did I felt I knew which was Google and which was Bing - not from doing searches that I would recognise, but just from the extra non-search information / lay-out.

I did my best to still pick without bias to my current search engine (Google) but ended up picking them 5/5 and Bing 0 times.

I have actually found myself using bing lately only because it doesn't try to confuse me into landing on google+.

Seriously, guys; this is a problem. I'm searching for a coffee shop or restaurant or something, and I want to link my friends to the map.

I --DO NOT-- want to go to google+. In fact, I don't really want to google+ ever. For anything. Ever.

Seriously I feel like I'm navigating a maze of accidental google+ links every time I use google anymore. It's really frustrating. Bing at least seems to be just...a search engine.

In fact, google hiccuped all over my 8 years old gmail account, deleting my inbox (yay! Thanks for that guys, oh, and it's just lost, too bad for me!) two days ago.

The upside to this? It somehow also unsubbed me from google+!

Google 4, Draw 1, Bing 0.

Why? Because I tested it with fairly specific queries. A while ago I tried to switch to Bing because after testing it with a few general searches, it seemed just as good. However, when I switched and started searching for compiler errors, etc - I soon switched back because the results were just nowhere near as good.

Here's what I searched for:

    bash: !": event not found error
    gtk calendar tutorial
    python import gtk error
    seed funding in uk

Something to remember by the way is that Google has spent a lot of time and effort on their "bubble", that is, search results personalisation (http://dontbubble.us) and it is a big part of what makes their results so great. That influence won't be present in this test.

Ended up picking google 4-1. I'm somewhat of a bing fan in the sense that I find them good enough for most searches. So I was a bit surprised that the results ended up skewing in google's favor ... guess that kind of backfired for them :-P

Bing seems to be a bit better at generic, broad searches (games, file extension $something, etc) and Google seems to be better at exact, specific queries (stop error 7B, RPGs announced at PAX 2012).

Bing ended up winning, which shocked the hell out of me, but after trying a couple more times, I noticed that pattern.

I did a comparison by actually searching the same phrases on google.com/ncr with Incognito. The actual Google results are much better than the Google results on this page.

Search for "Hunger Games" on Google returns the IMDB page as the first result, which is exactly what I want. But neither panes on the Bing's test has the IMDB page as the first result.

5-0 for Google. I tried mostly programming related queries as I find the differences most noticeable there. Especially when searching for specific APIs or objects, Bing tends to give you just the project homepage and some unrelated pages, while Google gives you the actual deep link.

I remember a while back when Bing was shown pretty conclusively to be taking Google results.

So I wonder if this test is really "Google vs. Bing + Google".

EDIT: Source - http://news.ycombinator.com/item?id=2165469

The very fact that this site exist says something. To me there's a weird feeling of desperation about it.

BTW it is best to use Incognito mode.

In my case Bing won 1 out 5 of the rounds. I used searches I had done recently that hadn't given satisfactory results on Google.

In any event Microsoft now at least has searches it can work with and improve. I can't imagine any other way they could get Google users to provide search data.

The study results are not very accurate. They made it with only 1000 US random users, and ~60% chose Bing more often, ~30% chose Google more often (on 10 queries).

See the following fictive example:

1000 people, 600 have the following score: 4 Bing, 3 Google, 3 Draw; 400 have the following score: 10 Google. So, in my example, 60% of the people chose Bing more often and 40% chose Google more often. But, if you look at queries numbers, Google was chosen for 3600+10400=5800 queries, and Bing for 4*600=2400 queries, that's a 2:1 ratio, but in favor of Google.

>Bing services aren't available right now

After 5 searches the screen greyed out and was unresponsive.

Guess Google wins by disqualification?

Much easier to use blekko's 3-engine monte, since it strips formatting that might give away the source engine. Just add /monte at the end of a query:

http://blekko.com/ws/hunger+games+/monte

This is not a real blind test though. (Pardon me for being a web developer, but) I can tell which is which by just looking at the green colors used in the links. The green in Google results is a little bit more saturated than the one in Bing.

Google results also seems to show share counts from Google+ as in: 73,352 people +1'd this

1. survey is organized by Microsoft, 2. I think the survey is skewed because typically when you go there you don't have anything specific in mind to search so you type one-word query (they even propose them!) and one-word queries are hard for google

I had two different runs that both ended 3-2 for Google. Even if Bing came out ahead, they seem so similar that I couldn't be bothered to retrain myself to bing stuff, instead of googling it, especially when I have gmail accounts.

Nice try Microsoft. But if you really need to create a contest to prove your search engine is better you're probably desperate.

The 'pepsi challenge' of search engines?

Tip: try to search for plot lines in movies such as: 'kid that is able to see people already dead'

I was Google 5-0 but with technical queries whose results i did not know for either bing or google but knew what pages should be on top. I think it is actually deceiving. While choosing I was pretty sure two of them were bing. Because one's default address for movie timings was Redmond and in other the video results were all non-youtube. Something does not make sense. Either google is ignoring youtube results and moved to Redmond or they (MSFT) changed it so that people choose other result. (not many people would get the Redmond reference though)

5/5 Google. Two technical queries, three more average ones.

But I could tell which side had which search results in 2 or 3 of the tests, so I can't claim complete blindness in the test. I tried to be impartial, though.

Bing you fool but not we :) it is definite we use google search engine everyday and what search term we will use in this contest is more likely the term we already use to search frequently let say "Design" and almost know the top ten search results of google. So when bing produces couple of different search results people will give definitely give first 2 or 3 clicks to bing just for the sake of change but for long term user most important and refined results belong to Google we know :D

I wound up with 3/1/1 in favor of Bing. I sort of felt like I was choosing the Bing options, but Google still hasn't given me a real reason to switch.

MS has definitely caught up quite a bit, though.

Google 5-0

Google's strength is not in searching for head or torso terms like "facebook" or "2013 calendar". It is in searching out tail terms; those obscure terms which might include a typo or two, and the phrasing may not be correct.

Unfortunately (and I don't have much time to explain this) one simply _can not_ judge a search engine's comparative performance using random users. And the Bing guys are falling into that trap.

EDIT: To expand more on what I wanted to write.

// DISCLAIMER: This is my personal view, and has nothing to do with my employer.

There are 3 large (fuzzy) classes of queries on search engines: head, torso and tail. The head queries are small in number but large in volume: queries like "Facebook", "Google", "Yahoo", etc. (yes, people do 'search' for these on search engines. Just a handful of such queries may make up 10% of the entire search engine traffic, which would be 100M queries per day (or so) on Google. It is depressing to see the numbers). In general, any search engine worth its salt will get the head queries right.

The torso is basically deciles 2-6 or so (there's no hard and fast definition). Here you'd have most musicians, actors, popular restaurants, etc. Bing and Google both would do a mostly adequate job here, for the first result. But: if you want to explore the concept a little, Bing goes astray and Google does not. For example, search for "idle hand tattoo" on http://bingiton.com/ . The first result is the same, to my local tattoo parlor. But the second result on Bing is from Mesa, AZ; while Google sticks with the right result and offers up other, related sites (Yelp, etc.). Google knows that "Hand" in the title is not the same as "Hands", and gives me not only the store, but also their Yelp, FB and Twitter pages; in other words: Google _understands_ your query better.

And finally, the tail. Here we have obscure error codes, weird technical terms, etc.; most of which are seen maybe once or twice a day. These constitute the bottom ~30% of the query stream. This is where Google really shines. It is very rare indeed to look for an obscure term and find that Bing does a better job than Google (I have yet to find an example).

So, as a user, I have a choice of 2 search engines; and one of them I know will do a better job on obscure terms. Guess which one will I pick?

When Bing claims that "most people prefer Bing in our user studies" (or something similar), their conclusion is flawed because the study itself is flawed. You can't sit a person down in front of a computer and ask him to evaluate your performance on tail queries! Where are those tail queries coming from? If they are provided by Bing ("here's an obscure term, search for it and tell us what you think"), then the user has no way to evaluate how relevant the results are, because, by definition, it's an obscure term! On the other hand, if you ask the user to come up with an obscure term, it will most likely be a term they are familiar with and they'll already know the answer, so they won't really hunt for information. So in my "idle hand tattoo" example, they'll see the top result and claim satisfaction; when, in real life, I would like to see the Yelp reviews, maps, etc. for the place.

If they (Bing) really want to compare how well they're doing, here's a suggestion: setup a search engine (like bingiton, but with only 1 pane of results, and neutral branding), and make it the default for a large pool of users (with their permission). Randomly, switch the backend to Bing or Google. Then, monitor the heck out of what the users do (all with their permission): how often do they click on the first result, how often do they re-formulate queries, the time to the first click, how often do they quickly come back to the results page from a bad click, etc. etc. etc.

This comment is quite late. Hopefully somebody at Microsoft will see this.

I used Bing for several months. I actually found it indistinguishable from Google except for linux howto questions (Google seems to understand forum websites better).

The dealbreaker for me was lack of SSL support. Switched back to Google.

Nobody who uses Google regularly will switch to Bing: not enough of an advantage. But people who start out with Bing may never switch. That's the long-term game Microsoft has to play.

I ended up with a draw the two times I tried. I made the mistake of trying a Google search I had just done previously and ended up seeing the purple links, which tipped me off to which was which. Bing was the only one that gave me local and mapped results for "Chicago Loop pizza", but Google won out just as often.

Bing is as good as Google on average. The problem is, when you're up against an entrenched competitor, good enough just isn't good enough.

3/2 in favor of Google. I was surprised by how similar the results were, and by what I apparently considered important about search results.

I have this theory Microsoft keeps funding bing as a tax writeoff.

I've yet to see a website where the search referrals are even within 30 points of Google.

"we can make him better we have the technology"

How about we build a community test? We could make a little page showing the results for various search engines (e.g. google, bing, blekko, duckduckgo) and and put them to a similar test. We could open and show the data collected in real time so the community gains from the experience.

Immediately there is a problem: how to get the results.

Naively we could use iframes to show the pages for each results. But it is not possible.

We could use their APIs to get the results and show them. However these are limited to N calls on their free package. That would make the survey limited.

Another way is to screenshot the page (fetched using a simple browser call) and present it. This, however, should also be paid for more than N calls..

Anyway, after resolving this, we should be go to go.

We could gather data in an unbiased way and present it to the world.

This is simple to make, I think. Upvotes if you think it is a good idea (for this to be a join effort probably).

I selected one draw and then Google every other time. It was close for most searches but Google was slightly better. Google seems to be better at more detailed searches whereas Bing is better at searching for broader things (like they suggest 'wedding dress', 'chicken nuggets' etc.).

4 for Google and a draw. Google is still way ahead using my mother language (which is not English).

3-1 in favour of google for me.

I suspect a lot of this may be down to how challenging the queries are that people put in. Bing's historically been great at generic searches, but I made a point of looking for more difficult stuff, where google has (for me) always been better.

I found that I was not able to take the test fairly, because it was blatantly obvious from the style of the search results which one was Bing and which one was Google. The Bing results use a slightly duller green for the URL displayed underneath each result's title, among other stylistic differences.

So, the comparison is completely useless, as they did not anonymize the style of the results, and thus either conscious or unconscious bias can creep in. I expect that for users that are not aware that the results look different, there will be a significant preference for the results that they see every day (e.g. the engine they normally use).

I got Google 5 - Bing 0. I did 3 math related searches, 1 math graduate student by name and 1 search about a python math library. Looks like I'll only switch to Bing if I leave math, which is not very likely. ;)

5-0 in favour of google. I was quite surprised too since I used fairly generic queries that I thought Bing would do as well as google. - whats on in london - ready player one - bash history across sessions

Anyone else seeing different results on this vs. what Blekko's monte (e.g. http://blekko.com/ws/your+phrase+/monte) produces?

Google 4, Draw 1. Did not pick Bing results once. Methinks their "blind trial" only applies to queries Microsoft specifically gave the testers to make Bing results shine.

Using five technical searches I might actually make on a day by day basis, I chose Google 4 times and a draw once, though the results were close almost every time (once I chose Google because I searched for an organization name and wanted their homepage first rather than relevant news, and twice because the relevant result was from MDN rather than MSDN or W3C since I found that MDN tends to be the best site for actual technical reference, so YMMV).