How Google missed the start of the AI ​​revolution. The story of “transformers”, the technology that put the T in ChatGPT

Eight years ago, a few Google engineers managed, after months of testing, a giant leap in the history of artificial intelligence. They created the “Transformer” algorithm, a breakthrough that would underpin the entire AI revolution. To their astonishment, Google management did not rush to integrate it into its own products, and the company that took advantage was its much smaller rival OpenAI. A new book published by ZYX Books tells the story of this technology that could change the world much faster.

Transformers technology, a new AI architecture developed by Google researchers in 2017, was the basis of the entire revolution in artificial intelligence, a revolution that led to the creation of applications such as ChatGPT, Gemini, Claude AI or Perplexity AI.

In the book “Supremacy – Artificial Intelligence, ChatGPT and the race that will change the world”, journalist Parmy Olson explains at length the race for the development of AI in the last decade and the competition between OpenAI and Google DeepMind, but especially between two visionary heads from the two companies: Sam Altman and Demis Hassabis.

“The T in ChatGPT stands for 'transformer,' and it's central to the new wave of generative artificial intelligence, which can produce lifelike text, images, and video, as well as DNA sequences and many other types of data,” writes journalist Parmy Olson.

The invention of the “transformer” in 2017 had an impact on AI comparable to the release of smartphones to the general public, the 334-page tome says.

When chatting with chatbots was annoying and rudimentary

“Transformers have greatly expanded the scope of possibilities for AI engineers. They can handle much larger volumes of data and process human language much faster. Before Transformers, interacting with a chatbot was rudimentary, because old systems were based on rule sets and decision trees. If you asked a bot a question that was not intended by its programming (which happened often), it would crash or make a major error. That's how they were originally designed. Apple's Siri, Amazon's Alexa, and Google's Assistant treated each question as an isolated request, and because of that, they couldn't remember previous questions in a conversation like a human does,” explains Parmy Olson, a tech journalist at the Wall Street Journal and Forbes.

Basically, explains the volume from ZYX Books, the new AI architecture has freed chatbots from these limitations. They started picking up on nuances and slang, referencing things mentioned a few sentences earlier, answering almost any random question and giving personalized answers.

This technological leap has also sparked a heated debate: have computers begun to “understand” language in the same way that humans do, or do they still process it only through mathematical predictions?

“It just takes attention”

The creators of this “Transformer” technology from Google worked for several months in 2017 to refine it, experimented and tested various architectures. Eight researchers ended up working on the project, and two of them are credited in the book for their contributions: Noam Shazeer and Jakob Uszkoreit.

The people at Google, it is explained in the cited book, analyzed the concept of “attention” in AI, by which a computer can identify the most relevant information in a data set, and their idea was to apply this technique to translate words faster and more accurately.

The researchers were impressed by how the Transformers were able to intuitively infer information from unstructured text, and six months after the first cafeteria discussions, the researchers documented their findings and wrote a paper that became public in the academic space.

“The headline Attention Is All You Need” hit the front page of scientific papers, perfectly summarizing their findings. Using a “transformer” algorithm, the AI ​​system can simultaneously pay attention to huge amounts of data and use it much more efficiently,” writes the volume signed by Parmy Olson.

Although this architecture was born in the “courtyard” of Google, Open AI was the company that capitalized on the discovery and, more than that, turned it into a weapon that launched, for the first time in many years, a real threat to the giant corporation, says the book about the evolution of AI.

OpenAI published the GPT‑1 model in 2018 which is based on the “Transformer” architecture. Since the architecture had been made public in the academic literature, OpenAI could use it, extend it, and develop new models based on it.

Why it's amazing that Transformers architecture saw the light of day at Google

Although Google had tremendous resources and talent, organizational sprawl and fear of disrupting its advertising business prevented employees from driving innovation, Parmy Olson explains.

According to former employees, the Google Brain division, which was home to the company's best deep learning researchers, struggled due to management's unclear goals and strategies.

The culture of laziness was influenced in part by the presence of a large number of talented scientists, including Geoffrey Hinton. The bar was set very high, and Google was already using state-of-the-art AI techniques, such as recurrent neural networks, which processed billions of words daily, the cited volume explains.

But the technology developed in 2017 was far beyond what was commonly used in the company. Some of the researchers tried – without success – to convince the Google company of the huge potential of “transformers”.

A revolutionary chatbot that has not been released to the general public

In 2019, a few people at Google created a chatbot called Meena, trained on billions of social media conversations. He was so advanced that he could make up jokes or discuss philosophy. Enthusiastic, the researchers wanted to present it to management, but Google blocked the project, fearing it might produce inappropriate messages.

Parmy Olson notes that the people who developed Meena were super excited about the chatbot and wanted to do a demo, hoping that the company's bosses would also be satisfied and use the technology in modernizing the famous Google Assistant. A scientific paper entitled “Towards a Human-like Open-Domain Chatbot” was also published.

Google executives weren't happy, but blocked their efforts, fearing the new chatbot might make inappropriate remarks that could damage the company's reputation or, worse, its $100 billion advertising business. According to an article published by The Wall Street Journal, Google has prevented all attempts to publicly launch the Meena chatbot or integrate it into the company's products, the author also writes.

The slowness with which Google adopts new technologies can be interpreted in several ways, writes Parmy Olson, who explains that the Transformer allows computers to generate not only text, but also answers to various questions. If they started using this technology more often, users would turn to the Google search engine less often.

*ZYX Books is part of the same media group as Hotnews.