Search Chatbots? Pah, This Startup's Trying On Yahoo's Old Outfit Of Web Directories

Interview Web search, long dominated by Google, is in play again, at least among incumbents and entrepreneurs if not frustrated web searchers.

It's not just that Google Search has by some accounts got worse, though Google maintains its results are still better than other search engines. It's that both Google and Microsoft are betting search can be improved with the addition of generative AI, despite the risk of hallucination and misinformation. Game on.

As recent research out of the University of North Carolina at Chapel Hill observes, "For years, the fields of information science, information retrieval, and human-computer interaction have studied how people make use of search systems and have investigated ways to improve search interfaces. The integration of generative AI chat components is a major development that may profoundly change the way users interact with search systems."

Microsoft's merger of chatbot and search in Bing has yet to meaningfully improve its search market share, as we've noted. Nonetheless, search firms are experimenting with AI to see whether confident - but not necessarily correct - answers from large language models can loosen Google's stranglehold. There's also the possibility that antitrust intervention will change the market dynamics within a year or two.

Google isn't sitting still either. The Chocolate Factory just renamed its Bard AI service to Gemini and made the chatbot more accessible on mobile devices, as well as more expensive via an optional premium subscription offering. It's not replacing Google Search but making prompt-based querying for answers an alternative to keyword searches for documents.

UK search startup begs to differ

Thomas Chopping, a UK-based economist who founded science-focused search startup El Toco, discussed this topic with The Register. He believes that as long as there are websites there will be a place for search engines, due to the economics of web publishing and the different needs met by conversational queries and web search.

"During the decades since it launched, we've been watching Google steadily trying to make search more predictive, by adding things like autocomplete and eventually instant answers," Chopping told The Register. "This has the byproduct of increasing the amount of time users spend on their site, at the expense of visiting the underlying sources of the data."

"Most of this time, their attempts were laughably feeble," he said, "because for a company drowning in money, they seemed unable to marry common queries with the correct answer, and were resorting to personal data to try and guesstimate what people wanted to see."

Chopping started working on El Toco in 2016 and launched the service in 2023.

"We were winding down development of our website when ChatGPT came online, which has the crucial difference of being vastly much better than Google's search box at understanding actual human queries," he said.

"So the Google vision of a magic box, which finds whatever you type into it, seemed one step closer to becoming reality. Obviously, this was quite scary, having spent so many years on a competing idea. Because of this, especially the fear bit, we have spent much of the last year thinking very deeply about its applications for search."

Suddenly there will be no incentive to produce content for a website if it doesn't bring in users

His first observation is that conversational tools and web search serve different purposes.

"One aims to answer discrete questions. The other aims to help users find websites," he said.

"Let's postulate for a second that, in time, the conversational version becomes so overwhelmingly good that nobody looks at websites any more. That is a real paradigm shift for how the web works because, as you have pointed out, suddenly there will be no incentive to produce content for a website if it doesn't bring in users.

"The websites could stay online, and go behind paywalls, but this obviates the usefulness of chat-based search, because then it won't be fed with any information. The chat-based search providers could then pay the websites for their data, which is a fight that Google has already inadvertently kicked off, particularly in Australia.

"But that business model would have to be funded somehow, and it's not clear that people en masse would want to pay for instant answers to questions. So it's a business model that will be forever imperiled by website-based search engines that let you get the answers for free, by visiting their original source."

• Everyone wants better web search – is Perplexity's AI the answer?
• If you use AI to teach you how to code, remember you still need to think for yourself
• What is Model Collapse and how to avoid it
• Microsoft's vision for the future of work is you trusting Redmond to get AI right

His second observation is that Google and Microsoft, in their rush to deploy chat assistance, seem to have forgotten about how people use the web.

"People do a lot of things online which are not just asking questions, one of which is browsing," he said. "If I'm bored in a train station, or just curious about a topic, I often want to just browse around. It's impossible to browse with conversational-style search tools, which are entirely focused on answering questions.

"Right now, this is playing into the hands of Meta and TikTok, because it takes so much effort to find good quality websites via search engines that people stopped bothering.

"I visit The Register because I've been reading it since university, not because I did a Google search for 'good IT-related news websites.' Social networks are then competing with each other for this attention, and have almost stopped being social networks in favor of being content factories. This is particularly evident on Facebook itself, which is flooded with non-content, aiming at keeping you there in favor of using competing platforms showing more of the same mindless content."

Going back to basics

El Toco, Chopping said, is a search engine with filters. Crawled pages get labeled with various metadata like genre and page purpose to make it easier to filter out less relevant results.

"The aim is to make web search efficient and precise," Chopping explained. "But also, it can all be done without knowing any of the context of the search.

"We don't need the user's personal data to work out which results to show, because the user can express this on their own. We don't need AI to turn the search into a conversation, because this can be done with a few clicks of the user interface.

We don't need AI to turn the search into a conversation

"Charging users for web search is a model which clearly doesn't work, thanks to Neeva for demonstrating that, so we allow adverts but if the users care they can go into a menu and simply switch them off."

Chopping expressed skepticism about people's willingness to pay for AI search help when free search results can still be had.

"The economist in me would suspect they're trying to see what they can get away with as far as the pricing goes, because it's easier to cut the prices than it is to raise them," he said.

Nonetheless, he believes AI is important in the search business, even if it's overhyped.

"It's definitely very important," he said. "But it's more a question of marketing, in my opinion, than anything else. People in finance are now fitting a line of best fit to charts and saying, 'Oh, we use machine learning to generate this stock prediction.' I'm like, 'No, you didn't. It's a line of best fit.' Just a few years ago, that's what they would have called it – they'd have said, 'Oh, we use statistical models.' Now everything's AI."

Chopping insists that the scale of the internet means you have to have some level of automation. El Toco uses machine learning on the backend to classify the pages it crawls and label them so they can be filtered more efficiently.

"I think that the front-end presentation of AI is a bit of a question mark at the moment, and we've decided to go in a very different direction from Google and Bing, and time will tell which one is correct," he said.

A new era?

Chopping believes a new era of search competition is at hand.

"I almost don't want to say this, but the barriers to entry for search are a lot lower than people think," he said. "Brave has been making a lot of noise about their search engine. And one of the things that made me smile in the article that sparked this conversation was their search guy saying how expensive it is to run a big search architecture.

"Right now we're sort of at the friends and family stage. We've got ours up and running. We're quite a way behind Brave in terms of the volume of traffic, but it's actually a lot more scalable than people think now that the cloud has come along. So I have a feeling that you'll be seeing more competition to Google attacking its heartland of basic general search queries."

I almost don't want to say this, but the barriers to entry for search are a lot lower than people think

"That doesn't mean those companies will necessarily make money," he said, "because Google has got the first mover advantage and it's been very difficult to take that away from them. But one of the other things that I do notice about all of the other search platforms is they don't really seem to have very many original ideas.

"Now along comes ChatGPT and they're all like a sort of herd of cows saying, 'Ah, search, there's gonna be a conversation. Let's all go off after that.'"

Chopping is skeptical that searchers want conversations. "I'm actually an economist, not from tech," he said. "So most of my friends work in finance. And when I asked them about all these tools, they're like, 'we just really don't see how it's going to help me in my day to day like trying to find things on the internet.'"

The opportunity in search, as Chopping sees it, is in vertical markets that Google with its crawl-everything approach doesn't serve very well.

To address the increasing difficulty of determining that content is genuine and not AI-generated nonsense, Chopping said El Toco is going for quality over quantity. Asked whether the engine is essentially reviving the model pioneered by Yahoo, a directory of curated sites, Chopping smiled.

"You're literally the first person I've spoken to who has mentioned that without my prompting it first and the answer is 'yes, it is,'" he said. "That's exactly what we're doing. … all of those web directories disappeared, at least sort of, around 10 years ago. What we've basically done is resurrect them.

"And there will come a point where we have to be a bit more loose about what search results go in there. But to your point about whether the information is genuine or not, the only real way of doing it right now, in my opinion, is to actually pay attention – with people – to what you're what you're indexing.

"It's become extremely difficult to find web content away from social media if you don't already know the website you want to visit," said Chopping. "There is growing awareness of this problem, people are increasingly asking whether there is somebody, somewhere, doing something about it. That's what we're trying to do." ®