How to build a search engine to disrupt Google
We are at a uniquely and historically weak moment for Google.
The quality of their results has even non-SEOs complaining, against a backdrop of being found illegally maintaining a monopoly and losing market share.
It’s more possible than ever for another search engine* to replace Google.
But there’s a reason that no one has been able to supplant Google thus far (monopolistic advantages aside): no one has really tried to make a search engine meaningfully different from Google.
So, here’s my pitch for how to build a search engine (or AI search) to disrupt Google: a way to provide better results and motivate users to switch away from their familiar Google habit.
In my plan, disrupting Google requires three steps – and a monumental amount of money.
Step 1: Make Google
The first step is to duplicate what Google has done: build a modern search engine that:
- Crawls the web.
- Categorizes and ranks content.
- Uses user behavior data to validate and train algorithms.
Step 2: Add a human layer
Next, we hire a team of subject matter experts and have them evaluate sites to determine how well they serve various topics.
We turn this into a ranking factor that can be weighted differently by query than other ranking factors.
This is not just surface coverage. We need at least 2-3 experts reviewing each site/topic combo to look for consensus (and catch malfeasance).
These are not random low-paid quality raters either; they know their subjects and know what a good and great article looks like.
They can identify:
- The master baker who specializes in croquembouche.
- The hiker who has the best tips on the Appalachian Trail’s 100-mile wilderness.
- The book reviewer who is an expert on Cozy Core titles.
I think it’s fair to ask a search engine to outperform its competitors the same way they say websites need to: by offering real human expertise and insight rather than just scaled AI content.
Automation and machine learning are essential, too.
We need to build systems to find sites for this manual review team. What sites might be relevant for which topics?
Design it to surface sites that wouldn’t normally rank due to lack of brand or authority signals in addition to reviewing the sites that dominate.
This can turn our search engine into a discovery platform that helps users uncover hidden gems across the web while filtering out the recycled content that dominates Google today.
When you search for a birdhouse plan, you get the guy who’s been building birdhouses for 40 years and has 200 plans online, rather than Home Depot and This Old House ranking for every DIY search.
But Home Depot can rank, too, if it’s a great result for birdhouse plans.
No limits on what a site can rank for – just a quest for the best results, regardless of where they come from.
All we care about is the quality of the content.
Sites with endless ads, pop-ups, and layout shifts are always rated a bad experience.
So are recipe sites that take six screens and 500 words before they get to the start of the recipe.
Good results must have a good user experience, and humans can spot in an instant the garbage Google’s most advanced machine learning algorithms are blind to.
The web is too large to evaluate every site and query out there manually – but our imaginary search engine doesn’t have to.
It’s looking at sites on the topic level, and for anything we haven’t covered, we still have a Google-like search engine beneath.
Step 3: Marketing
Here’s the real secret: Being better than Google isn’t enough.
People are so used to using Google that even with Google’s results worse than ever, it will be hard to get a majority of their users to switch over.
To really dominate search we need to get the entire web on our side by using the YouTube model.
Pay 30% of our search ad revenue to the sites ranking organically.
We pay sites based on the traffic they get and the ad revenue for those queries.
Like YouTube, we set a minimum threshold and make the entire thing self-serve, and the highest-trafficked sites get reps they can talk to.
This single move has so much capacity to make the entire web a better place. Just thinking about it drives me bonkers.
Sites would have a way of monetizing that doesn’t require shitty disruptive ads or potentially corrupting affiliate relationships.
Sites would be motivated to make better content to rank better.
Unlike Google, the human layer of our hypothetical search engine means that sites can’t rely on just tricking the algo ranking factors.
Search is such a disgustingly profitable enterprise that we can afford to give away 30% off the top.
Bing, maybe?
This requires so many resources that I feel Bing might be the only company capable of making it a reality.
Microsoft has already taken the hardest and most expensive step of replicating Google.
The biggest problem with Bing is that it never really tried to do anything substantially different to truly disrupt Google.
Wouldn’t it be great if instead of pretending to be Google to keep people from leaving, they could proudly be Bing and attract new searchers?
Can you imagine how compelling the commercial for the new Bing would be?
I could imagine it so well that I wrote a fake commercial and hired Cool Cat Audio for the voiceover.
Tell me this wouldn’t get you to try out a new Bing:
There’s a darker side to the marketing angle: we break the open web by incentivizing exclusivity deals.
This could be explicit like offering a 40% cut instead of a 30% cut to sites that block other engines in their robots.txt.
Or it could be implicit like the way Google paid Reddit $60 million for its data, and now Reddit blocks all other search engines from crawling its site.
But I don’t think we must go down that dark rabbit hole to win.
Using knowledgeable human evaluators combined with offering a cut to the sites that make our business should be enough to take a majority share of the search market.
Best of all, anyone else who wants to compete is forced to share revenue, too, just to keep up. Tell me: Wouldn’t this improve both a search engine and the entire web?
*Call me optimistic, but I don’t think conversational AI will replace search engines for anything other than short-answer or navigational queries. At best, they’ll become a different back end for the search experience – basically a different form of ranking algorithm.