Google logo is seen in this illustration.
Credit: Reuters
By Parmy Olson
After being caught on the back foot by OpenAI’s launch of ChatGPT, Alphabet Inc.’s Google has performed a miracle: It has caught up. Its artificial-intelligence models are now as good as its rival’s. But did Google achieve that acceleration by playing fair? Perhaps not. And the regulatory answer to that question could hurt if the company is forced to make a change that hampers its AI prospects, even if that’s healthier for the market.
European regulators have just launched an investigation into Google’s rollout of its AI Overviews and AI Mode features — which augment its dominant search engine — to see whether it imposed unfair terms on content creators, giving its model an edge over others.
While OpenAI, Anthropic PBC, Amazon.com Inc. and other AI competitors spend hundreds of millions of dollars on licensing deals with publishers to get hold of their content, Google has privileged access to the entire internet through the web crawler it relies on to power Google Search. This is an automated program that browses the web to index its pages, and the crawler known as Googlebot organizes everything it finds into the company’s vast searchable index.
So even though other firms pay to use high-quality data to teach their AI, Google gets it for free. It uses the same program that indexes the world’s information to also help train the models that sit behind its Gemini chatbot and AI Overviews. Compounding the problem, website owners have suffered a drop in traffic because people are getting what they need from the AI summary and not clicking on links in Google search results. An old joke is that the best place to hide a dead body is the second page of Google results — but who even looks at page one anymore when AI Overviews sits atop them all?
Publishers are hamstrung. Block Google’s web crawler and they risk not being seen in its plain-vanilla search results. Let the company use their content to train its AI systems and they’re losing an opportunity to get paid. And thanks to an organizational quirk, Google uses content for AI training even when a website does opt out, the company revealed in court earlier this year.
“Google is saying we have a God-given right to all content in the world, even if we don’t pay for it,” Cloudflare CEO Matthew Prince said at the Bloomberg Tech Summit earlier this year.
Prince has been steering regulators in Europe toward an elegant and simple solution: force Google to only use Googlebot for search, and create a separate web crawler that only scrapes content used for AI Overviews. That way, publishers can properly opt out or ask for compensation. That would hardly smother progress in AI. After all, if Nvidia Inc. can charge for chips and engineers for their time and brainpower, why can’t website publishers charge for their output?
Google’s engineers would find it technically easy to build such a bot with a different identifier that publishers could block separately. But Alphabet will resist anything that forces it to negotiate and pay for AI training content like everyone else. The company is enjoying the monopolistic benefit that comes from being so popular it’s a verb, accounting for 90% of all web searches.
And let’s not forget that Google bought its way to market dominance, paying browsers to be their default search engine and Apple Inc. $20 billion in 2022 alone to be on the iPhone, according to court documents.
The AI boom should be driving a competitive marketplace with hundreds of viable companies, just as the dotcom boom did originally. Instead, it’s on track to keep profits within existing giants like Google. Double-dipping with Googlebot is just the latest example of how it has capitalized on its dominance to further entrench itself.
Google argues that the EU probe “risks stifling innovation in a market that is more competitive than ever.” But the opposite is true. The underlying advantage that may well have helped Google catch up so quickly needs to go.