Skip to content
these robotic eyes

Google’s RT-2 AI model brings us one step closer to WALL-E

"First-of-its-kind" robot AI model can recognize trash and perform complex actions.

Benj Edwards – | 52
A Google robot controlled by RT-2.
A Google robot controlled by RT-2. Credit: Google
A Google robot controlled by RT-2. Credit: Google

On Friday, Google DeepMind announced Robotic Transformer 2 (RT-2), a "first-of-its-kind" vision-language-action (VLA) model that uses data scraped from the Internet to enable better robotic control through plain language commands. The ultimate goal is to create general-purpose robots that can navigate human environments, similar to fictional robots like WALL-E or C-3PO.

When a human wants to learn a task, we often read and observe. In a similar way, RT-2 utilizes a large language model (the tech behind ChatGPT) that has been trained on text and images found online. RT-2 uses this information to recognize patterns and perform actions even if the robot hasn't been specifically trained to do those tasks—a concept called generalization.

For example, Google says that RT-2 can allow a robot to recognize and throw away trash without having been specifically trained to do so. It uses its understanding of what trash is and how it is usually disposed to guide its actions. RT-2 even sees discarded food packaging or banana peels as trash, despite the potential ambiguity.

Examples of generalized robotic skills RT-2 can perform that were not in the robotics data. Instead, it learned about them from Internet data.
Examples of generalized robotic skills RT-2 can perform that were not in the robotics data. Instead, it learned about them from scrapes of the web.
Credit: Google
Examples of generalized robotic skills RT-2 can perform that were not in the robotics data. Instead, it learned about them from scrapes of the web. Credit: Google

In another example, The New York Times recounts a Google engineer giving the command, "Pick up the extinct animal," and the RT-2 robot locates and picks out a dinosaur from a selection of three figurines on a table.

This capability is notable because robots have typically been trained from a vast number of manually acquired data points, making that process difficult due to the high time and cost of covering every possible scenario. Put simply, the real world is a dynamic mess, with changing situations and configurations of objects. A practical robot helper needs to be able to adapt on the fly in ways that are impossible to explicitly program, and that's where RT-2 comes in.

Ars Video

 

More than meets the eye

With RT-2, Google DeepMind has adopted a strategy that plays on the strengths of transformer AI models, known for their capacity to generalize information. RT-2 draws on earlier AI work at Google, including the Pathways Language and Image model (PaLI-X) and the Pathways Language model Embodied (PaLM-E). Additionally, RT-2 was also co-trained on data from its predecessor model (RT-1), which was collected over a period of 17 months in an "office kitchen environment" by 13 robots.

The RT-2 architecture involves fine-tuning a pre-trained VLM model on robotics and web data. The resulting model processes robot camera images and predicts actions that the robot should execute.

Google fine tuned a VLM model on robotics and web data. The resulting model takes in robot camera images and predicts actions for a robot to perform.
Google fine-tuned a VLM model on robotics and web data. The resulting model takes in robot camera images and predicts actions for a robot to perform.
Credit: Google
Google fine-tuned a VLM model on robotics and web data. The resulting model takes in robot camera images and predicts actions for a robot to perform. Credit: Google

Since RT-2 uses a language model to process information, Google chose to represent actions as tokens, which are traditionally fragments of a word. "To control a robot, it must be trained to output actions," Google writes. "We address this challenge by representing actions as tokens in the model’s output—similar to language tokens—and describe actions as strings that can be processed by standard natural language tokenizers."

In developing RT-2, researchers used the same method of breaking down robot actions into smaller parts as they did with the first version of the robot, RT-1. They found out that by turning these actions into a series of symbols or codes (a "string" representation), they could teach the robot new skills using the same learning models they use for processing web data.

The model also utilizes chain-of-thought reasoning, enabling it to perform multi-stage reasoning like choosing an alternative tool (a rock as an improvised hammer) or picking the best drink for a tired person (an energy drink).

According to Google, chain-of-thought reasoning enables a robot control model that perform complex actions when instructed.
According to Google, chain-of-thought reasoning enables a robot control model to perform complex actions when instructed.
Credit: Google
According to Google, chain-of-thought reasoning enables a robot control model to perform complex actions when instructed. Credit: Google

Google says that in over 6,000 trials, RT-2 was found to perform as well as its predecessor, RT-1, on tasks that it was trained for, referred to as "seen" tasks. However, when tested with new, "unseen" scenarios, RT-2 almost doubled its performance to 62 percent compared to RT-1's 32 percent.

Although RT-2 shows a great ability to adapt what it has learned to new situations, Google recognizes that it's not perfect. In the "Limitations" section of the RT-2 technical paper, the researchers admit that while including web data in the training material "boosts generalization over semantic and visual concepts," it does not magically give the robot new abilities to perform physical motions that it hasn't already learned from its predecessor's robot-training data. In other words, it can't perform actions it hasn't physically practiced before, but it gets better at using the actions it already knows in new ways.

While the ultimate goal for Google DeepMind is to create general-purpose robots, the company knows that there is still plenty of research work ahead before it gets there. But technology like RT-2 seems like a strong step in that direction.

Listing image: Google

Photo of Benj Edwards
Benj Edwards Senior AI Reporter
Benj Edwards is Ars Technica's Senior AI Reporter and founder of the site's dedicated AI beat in 2022. He's also a tech historian with almost two decades of experience. In his free time, he writes and records music, collects vintage computers, and enjoys nature. He lives in Raleigh, NC.
52 Comments
Staff Picks
Prev story
Next story