I rigorously tested the most popular AI search engines to discover their strengths and weaknesses.
I ran this test for two reasons:
- As a human being, I want to use the best search engine around. I spend my life on the internet. I want to find information in the fastest, most effortless way possible.
- As a marketer, I want to know where the tide is headed. Will AI search engines replace “traditional” search engines? Are AI chatbots really good enough to dethrone Google? Should I start investing more effort in LLM optimization?
There are basically a thousand products claiming to be AI search engines, but I focused on the biggest and most popular.
I tested out the AI features on offer from two existing search giants—Google’s AI Overviews and Bing AI—and two trendy AI upstarts—ChatGPT Search and Perplexity.
The best AI search engines are—in my opinion—ChatGPT Search for the best chatbot experience, and Google’s AI Overviews for the best all-around search engine.
Read on for pros and cons, screenshots, and short videos of the search experience.
I ran these searches through every AI search engine and recorded my experience. I focused particularly on:
- Relevance: were the results appropriate for my query?
- Accuracy: did the search engines return valid, up-to-date information?
- Usefulness: did I meet the intent of my search? Did I leave happy?
- Speed: how quickly did I get my answer?
- Ease-of-use: how much effort did it require?
This is a completely unscientific but very interesting analysis of the best AI search engines. Let’s get started.
Pros
- Excellent at summarising lots of sources.
- Great for news-related topics.
- The “Related topics” feature is useful.
Cons
- Plenty of incorrect answers and hallucinations.
- Bad at image, commercial, and local searches.
- Didn’t really feel like a search engine.
Perplexity feels more like a Wiki or a knowledge base than it does a search engine. Many of its responses are terse bulleted lists, and it excelled in summarising technical queries (like DA09Dawnbreaker) or news-related topics (like elder scrolls vi news).
I also noticed lots of errors and hallucinations in its output. One example: for Should I become a vampire or a werewolf in skyrim?, Perplexity suggested that werewolves were immune to silver weapons—they actually take bonus damage from them. (C’mon Perplexity, honestly.)
Information often seemed outdated too. Perplexity said that Skyrim SE was on sale for $7.99 on Steam (it wasn’t), and the sale extended until May 30… 2024. It feels like Perplexity’s index isn’t anywhere near as fresh as other AI search engines.
Perplexity also regularly failed to “get” the intent behind my searches, and defaulted to word salad paragraphs of text. When I searched for “Steam skyrim”, I got a long, rambling response with no actual link to Steam. For “Skyrim creations”, it listed no actual creations, and just shared a Wiki-style overview of the Creation concept.
Perplexity also struggled with image searches, commercial queries were very unsatisfying, and local searches were decidedly not local, with responses that recommended websites and shops that were nowhere near me.
Pros
- AI Overviews were generally helpful (when they showed up).
- Does a great job at surfacing different media formats, like images and videos.
- Excels at commercial and transactional searches.
Cons
- Very cluttered search results, often with duplication between AI Overviews and Fatured Snippets.
- AI Overviews feel like a dead-end—you can’t continue to interact with them.
- Felt a little weird having Reddit in the majority of SERPs.
It was fun evaluating Google Search through fresh eyes. Unsurprisingly, Google was very good for commercial and local queries: interactive product carousels made it easy to find relevant products in no time, and map packs were fantastic for finding local shops.
I was also reminded of just how powerful Google’s integration with YouTube is: any queries that were best resolved with video content almost always included YouTube videos—often timestamped—in the SERP.
AI and machine learning underpin all of Google Search, but as far as specific AI features, AI Overviews only triggered for about 15% of my queries. These were all informational queries, and in most cases, they were very specific long-tail queries.
The AI Overviews were actually pretty good, relevant and accurate—but I definitely missed the ability to keep interacting with the AI.
AI Overviews generally felt like amped-up Feature snippets, providing direct, certain answers to long-tail queries in SERPs (and in fact, many of the AI Overviews competed directly with duplicate Featured snippets directly below them in the SERP).
It was also a little weird having Reddit present in so many SERPs. Some queries really are best-suited by user generated content—like those with no objective “best” answer—but Reddit muscled its way in everywhere.
Pros
- Excellent for very specific long-tail queries.
- The ability to ask follow-on questions is extremely good.
- “Memory” feature is useful.
Cons
- Outright failed on several occasions.
- Pretty bad for image searches and commercial searches.
- Verbose, likes to write long answers even for simple questions.
ChatGPT Search is fantastic for very specific long-tail queries. Answers to queries like Should I become a vampire or a werewolf in skyrim? were like reading a short blog post that was written just to summarise a bunch of sources specifically for your query—because, well, it was.
Couple this with the ability to ask follow-up questions, and ChatGPT’s ability to remember information from previous conversations (its “memory”), and asking long, complicated questions is a ton of fun.
But this strength is also a weakness. ChatGPT excels with text-heavy queries, and kinda sucks for many others. Short-tail queries felt way more limited than Google. Navigational queries (like skyrim creation kit download) were unsatisfying: ChatGPT would explain precisely how to download the Creation Kit (a modding engine for Skyrim), but it couldn’t actually let you do it.
It was also pretty weak with commercial queries, and very bad at image queries, with a default response limited to just four images (nowhere near enough to sate my appetite for hi-res skyrim landscapes). Although one surprise: ChatGPT Search incorporated playable YouTube videos directly into its responses.
ChatGPT Search outright failed for a few queries, requiring multiple refreshes to get results for very basic queries. All-in-all, it was strongest for queries that benefited from long, detailed text responses, and worse on almost every level for all the other queries.
Pros
- Great at navigational and commercial searches.
- Interesting search features for short-tail queries.
Cons
- AI features were quite often broken and unhelpful.
- Slightly disappointing local and news searches.
Bing has two outwardly obvious “AI” features. One is a text snippet, very similar to Google’s Featured Snippets and AI Overviews. These worked okay, but not as well as Google’s AI Overviews.
Often, the AI answers didn’t quite understand my search (the response for Should I become a vampire or a werewolf in skyrim? didn’t actually include an answer—just steps to become either a vampire or werewolf).
The other was a dynamically generated “table of contents” module, and despite looking nice, it generally fell somewhere between not helpful and totally broken, with the different modules appearing in jumbled orders or missing steps.
Like Google Search, Bing performed perfectly at navigational queries and showed tons of products and inspiration for commercial searches. It returned good results for newer, trending topics (as did all the search engines, surprisingly).
Most of all, I loved the visual sidebar that appeared for the short-tail searches, offering tons of facts and trivia for skyrim. When the search intent is so so vague, it seems incredibly helpful to treat the SERP as a mood board of inspiration and possible wormholes to pursue.
Surprisingly, I was disappointed with local searches, and the news search (Elder scrolls vi news) was pretty awful.
Final thoughts
“Traditional” search engines have decades of experience in making searchers—and all the different types of searches they make—happy. This experience is obvious across commercial, local, and image searches where Google and Bing dominated the newer tools.
But “pure” AI search engines also unlock new types of search behavior. The query Skyrim bosmer stealth archer build backstory was sensational with ChatGPT, allowing me to actively brainstorm and refine a story in a way I never could with Google Search. Perplexity excelled at summarising news and complex information.
Are the strengths of AI search engines enough to make up for their many weaknesses? Will searchers abandon a company that controls 88.5% of the current search engine market to experiment with relatively unproven new tools?
More likely (in my estimation) is that we’ll see the existing search engines build, integrate, and partner with AI companies to bring the best features into the existing search experience (like Bing’s partnership with OpenAI.)
In the meantime, the best AI search engines are—in my opinion—ChatGPT Search for the best chatbot experience, and Google’s AI Overviews for the best all-around search engine.
