Note that, based on progress so far, they're not actually building a new search engine as such - they're spending the money on buying the results in from Bing and on marketing[0].
back
2 comments
I really don't think this is the answer at all.
Any true Google competition will need a world class crawler and ML like Google.
I'm not convinced.
A world class crawler definitely.
ML? Maybe, I guess. Google took over search before ML was even a thing(or at least, a popular thing) because they were fast and had good results. And that's all most folks probably care about. Sometimes good results are truly those most relevant to the terms entered...not injected with ads, or targeted, or SEO'd to death.
In fact, I'd go as far as to say Google search results are worse today than years ago for a lot of queries.
Yeah, you do need ML. But a tweakable one. I’ve built a crawler, scraped the first 10 pages of all public domains I could find, then made those billions of pages freely queryable with ElasticSearch. The result: moderately good. It was great for discovering hidden gems. But you needed a lot of longtail keywords and exclusions.
There is a massive amount of textual garbage out there!
I gave it more thought in passing and am a bit curious what real, heavy ML might look like. And if there's value in such things.
Example: I often find myself remembering bits of things, and querying Google, and the results are way off or completely irrelevent. Things a human would know in a matter of seconds, generally.
I was for example trying to remember that newish Netflix movie that is very similar to the Michael Douglas movie "The Game." It feels like such a question should be easy for a sufficiently smart system. Something like Watson, I guess?, at the search engine level would be immensely mindblowing to me.
The answer btw for anyone curious was "Rebirth." Nearly every review of said movie makes the comparison.
This is where semantic data would be really handy (admittedly, it's not a great example because the two movies are not linked at the moment):
Google took over search when Alta Vista was a thing. Their search has improved gradually over the years to the point where it "understands" the web and the questions people are asking way better than any simple crawler/index could. It's not perfect but if Google turned off the ML, I expect I'd miss it.
You forgot it will need world class $$$. To pay a very big captcha farm because your crawler will be blocked everywhere. And a very big proxy farm, for the same reasons, because everywhere google/bing crawler is hardcoded, including your frienldy cloudflare, and you cannot have a partial of the internet.
It highlights Bing’s strength. There are many memes in popular culture that if you want political results, you use Google; if you want shockingly down-to-earth results, you ask Bing. I’m still missing Altavista to get the raw results, but you get the idea: Bing cultivates the image of being accurate, and wins in all the secondary markets.
>I’m still missing Altavista to get the raw results
This is called Yandex nowadays.
Yandex: whatever the search query the image results are half naked women.