A single-armed robot was positioned in front of a table. On the table sat three plastic figurines: a lion, a whale, and a dinosaur. An engineer provided the robot with an instruction: "Retrieve the extinct animal. " After a moment of whirring, the robot extended its arm, opened its claw, and descended. It successfully grabbed the dinosaur. Up until recently, witnessing this demonstration during a podcast interview at Google's robotics division in Mountain View, California would have been unthinkable. Robots were incapable of reliably manipulating unfamiliar objects, let alone making the leap from "extinct animal" to "plastic dinosaur. " However, a quiet revolution is occurring in robotics, leveraging recent advancements in large language models—the same kind of artificial intelligence system that powers ChatGPT, Bard, and other chatbots. Google has begun implementing cutting-edge language models into its robots, equipping them with artificial brains. This secretive project has significantly enhanced the robots' intelligence and bestowed them with new problem-solving abilities. I had a glimpse of this progress during a private demonstration of Google's latest robotics model, known as RT-2. The unveiling of this model on Friday represents the first step toward a transformative leap in robot construction and programming, as described by Google executives. "This change has compelled us to reevaluate our entire research program, " remarked Vincent Vanhoucke, head of robotics at Google DeepMind. "Many of our previous endeavors have become completely obsolete. " While robots still fall short of human-level dexterity and struggle with certain basic tasks, utilizing AI language models to enhance their reasoning and improvisation skills represents a promising breakthrough, according to Ken Goldberg, a robotics professor at the University of California, Berkeley. Goldberg stated, "The ability to link semantics with robots is highly impressive and an exciting development for robotics. " To comprehend the significance of this advancement, it helps to understand how robots have traditionally been built. Engineers at Google and other companies have typically programmed robots for specific tasks by providing them with a predefined list of instructions. For example, to flip a burger, the robot would be instructed to lower the spatula by 6. 5 inches, slide it forward until it encounters resistance, raise it by 4. 2 inches, rotate it 180 degrees, and so on. The robots would then practice the task repeatedly, with engineers making adjustments to the instructions until success was achieved. While this approach worked for limited purposes, the process of training robots in this manner was slow and labor-intensive. It necessitated collecting significant amounts of real-world test data. Moreover, if one wanted to teach a robot a new task—for instance, flipping a pancake instead of a burger—the robot would typically need to be reprogrammed from scratch. These limitations have hindered the progress of hardware robots relative to their software-based counterparts. OpenAI, the creator of ChatGPT, disbanded its robotics team in 2021 due to slow progress and a lack of high-quality training data. In 2017, Alphabet, Google's parent company, sold acquired robotics firm Boston Dynamics to the Japanese tech conglomerate SoftBank. (Boston Dynamics is now owned by Hyundai and primarily focuses on producing viral videos showcasing humanoid robots with astonishing agility. ) In recent years, researchers at Google conceived an idea: instead of programming robots for specific tasks individually, why not enable them to learn new skills using AI language models?"Around two years ago, we began experimenting with these language models and realized the vast knowledge they possessed, " said Karol Hausman, a Google research scientist. "We started connecting them to robots. " Google's initial attempt to merge language models with physical robots was a research project called PaLM-SayCan, which was unveiled last year. Although it garnered attention, its usefulness was limited. The robots lacked the ability to interpret images—an essential skill for navigating the world. While they could generate step-by-step instructions for various tasks, they were unable to convert those instructions into actions.

In contrast, Google's new robotics model, RT-2, possesses this capability. It is dubbed a "vision-language-action" model, meaning it can not only observe and analyze its surroundings but also guide a robot's movement. This is achieved by translating the robot's movements into a series of numerical tokens—a process known as tokenizing—and incorporating those tokens into the same training data as the language model. Eventually, just as ChatGPT or Bard learns to predict the next words in a poem or history essay, RT-2 can learn to anticipate how a robot's arm should move to pick up a ball or throw an empty soda can into a recycling bin. "In essence, this model can learn to speak robot, " explained Mr. Hausman. In a one-hour demonstration that took place in a Google office kitchen filled with objects from a dollar store, my podcast co-host and I witnessed RT-2 accomplish several impressive tasks. For instance, it flawlessly executed complex instructions like "move the Volkswagen to the German flag, " accomplishing this by locating and grasping a model VW Bus and placing it on a small German flag several feet away. RT-2 also demonstrated its ability to comprehend instructions in languages other than English and establish abstract connections between related concepts. Notably, when I instructed RT-2 to pick up a soccer ball, I told it to "pick up Lionel Messi. " On the first attempt, RT-2 understood and executed the task correctly. While the robot was not flawless—for instance, it wrongly identified the flavor of a LaCroix can placed on the table (guessing orange instead of lemon)—Google has equipped RT-2 with numerous safety features. Aside from a prominent red button on the back of each robot, which halts its movement when pressed, the system utilizes sensors to prevent collisions with people or objects. The AI software integrated into RT-2 incorporates its own safeguards to prevent the robot from engaging in harmful actions. For instance, Google's robots can be taught not to pick up containers with water, as water can damage their hardware if spilled. Although some may be concerned about the prospect of rogue AI, with Hollywood fueling these fears through movies like the "Terminator" franchise and "M3gan, " Google researchers are celebrating the notion of robots with reasoning, planning, and improvisation capabilities. After years of stagnation, hardware robots have made a comeback, thanks to their chatbot brains. While Google currently has no immediate plans to sell RT-2 robots or deploy them on a wide scale, its researchers believe that these language-equipped machines will eventually have applications beyond mere entertainment. They envision robots with built-in language models being used in warehouses, medical settings, or even as household assistants—performing tasks like folding laundry, unloading dishwashers, and tidying up. Vincent Vanhoucke remarked, "This truly paves the way for using robots in environments where people are present: office spaces, homes, and anywhere physical tasks abound. " Naturally, manipulating objects in the messy and unpredictable physical world presents greater challenges compared to controlled laboratory environments. Additionally, AI language models often make mistakes or generate nonsensical answers—a phenomena known as hallucination or confabulation—thus introducing new risks when they serve as the brains of robots. However, Berkeley robotics professor Ken Goldberg believes that these risks are still remote. He stated, "We are not discussing unfettered autonomy for these robots. In the controlled environments of these labs, they are merely manipulating objects on a table. " Google assures that RT-2 is equipped with ample safety features. Alongside the red emergency button, sensors are employed to prevent collisions, and the AI software possesses its own safeguards to prevent harmful actions. For those concerned about AI going rogue, Google's precautions aim to address these worries. Whether one embraces or fears the integration of reasoning and improvisation into robots, it is an idea that elicits celebration among Google's researchers. After a prolonged period of dormancy, hardware robots have made a triumphant return, owe. . .