Google DeepMind unveiled Robotic Transformer 2 (RT-2) on Friday. This groundbreaking vision-language-action (VLA) model leverages internet scraped data to enhance robotic control through plain language commands. The objective is to develop versatile robots capable of navigating human environments, similar to fictional characters like WALL-E or C-3PO. RT-2 employs a large language model, trained on text and images sourced from the web (similar to ChatGPT), to recognize patterns and execute actions, even for tasks it hasn't been specifically trained on. This concept, known as generalization, enables RT-2 to perform tasks like recognizing and discarding trash, based on its understanding of what trash is and how it's typically disposed. RT-2's ability to generalize extends to identifying extinct animals, as exemplified by an engineer instructing the robot to "Pick up the extinct animal, " and the robot successfully locating a dinosaur figurine among three options on a table. Unlike traditional robot training methods that rely on an extensive dataset, RT-2's approach capitalizes on transformer AI models' capacity for generalization, making it adaptable to dynamic real-world scenarios. RT-2 builds upon earlier Google AI work, including PaLI-X and PaLM-E, and was co-trained with data from its predecessor, RT-1, collected over 17 months in an office kitchen environment using 13 robots.

The RT-2 architecture involves fine-tuning a pre-trained VLM model on robotics and web data to process robot camera images and predict actions. Actions are represented as tokens, similar to language tokens, enabling the robot to output actions. RT-2 employs chain-of-thought reasoning, enabling multi-stage reasoning and decision-making. In performance assessments, RT-2 demonstrated similar proficiency to RT-1 on tasks it was trained for ("seen" tasks) and showcased an impressive 62 percent success rate on new ("unseen") scenarios, a significant improvement over RT-1's 32 percent. While RT-2 exhibits remarkable adaptability, Google acknowledges its limitations, highlighting that web data inclusion enhances generalization over semantic and visual concepts but doesn't confer new abilities for physical motions that the robot hasn't already learned. As Google DeepMind strives for the creation of general-purpose robots, further research and development efforts are required. Nonetheless, RT-2 represents an important milestone in that direction.