WIRED recently conducted an experiment with a new type of voice assistant that has the ability to browse the internet and perform tasks online. This development indicates that virtual assistants like Siri, Alexa, and Google Assistant could become significantly more powerful in the near future. While these well-known virtual helpers are currently outshone by advanced AI-powered chatbots such as ChatGPT and Google Bard, integrating the recent advancements in generative AI into these legacy assistant bots will undoubtedly make them much more interesting. For instance, tasks like purchasing items online on the open web are far more complex and challenging than the typical functions performed by Siri, Alexa, or the Google Assistant, such as setting reminders or retrieving sports results. Buying something online involves comprehending the request, accessing the web to locate the appropriate website, and effectively interacting with the relevant page or forms. In an experiment, an assistant successfully navigated to WIRED's subscription page and even located the form there. However, the assistant stumbled at the final hurdle because it lacked a credit card. Additionally, experiments with the AI assistant, VimGPT, revealed its proficiency in searching for humorous cat videos or finding affordable flights. VimGPT is an open-source program developed by Ishan Shah, a solo developer, though it's important to note that it's not a product currently in development. Nevertheless, companies like Apple, Google, and others are likely conducting similar experiments to enhance Siri and other assistants. VimGPT is constructed on GPT-4V, a multimodal version of OpenAI's renowned language model. By comprehending a request, VimGPT can more reliably determine what to click or type compared to text-only software, which struggles to make sense of the web due to convoluted HTML. Ishan Shah, the creator of VimGPT, predicts that using a computer will look significantly different in a year's time. He envisions that most applications will require less clicking and more conversational interaction, with agents playing an integral role in web browsing. Shah is not alone in this belief, as computer scientists like Ruslan Salakhutdinov from Carnegie Mellon University, who previously served as Apple's director of AI research, see a substantial AI upgrade in store for Siri and other assistants. Salakhutdinov asserts that the next step in evolution will be agents capable of accomplishing practical tasks.

While connecting Siri to advanced AI akin to ChatGPT would be beneficial, Salakhutdinov argues that it would be far more impactful if Siri could proactively solve problems when instructed to do so. To facilitate the development and improvement of AI helpers capable of getting things done, Salakhutdinov and his students created simulated environments, such as a dummy e-commerce website, a Reddit-like message board simulation, and a platform for classified ads. These environments, collectively known as VisualWebArena, serve as testing grounds for AI agents and offer promising glimpses of future practical applications. For instance, an AI agent can analyze a photo of someone wearing a sweater and search through e-commerce listings to identify similar garments below a particular price, subsequently adding the cheapest option to a user's shopping cart. Another example involves an agent effectively navigating a Reddit-like site's settings to hide posts from a specific user upon receiving such a request. However, these advancements are not without their limitations. In their experiments, the team at CMU discovered that their AI agents were successful in accomplishing complex objectives only 16 percent of the time, while humans achieved the same objectives 88 percent of the time. Failures often stem from mundane issues like difficulty navigating websites or becoming trapped in an infinite browsing loop. Occasionally, failures may manifest as misbehavior, such as accidentally adding numerous items to a user's cart or mistakenly befriending an irritating user on a social platform. These incidents highlight the importance of not yet entrusting vital information to AI assistants like vimGPT. The CMU environments are valuable for researchers as they allow AI agents to freely operate within them without causing real-world harm. Collecting data on such occurrences aids in understanding the agents' performance levels and the specific reasons for their failures. Salakhutdinov emphasizes that setting agents loose in environments like VisualWebArena imbues them with the ability to actively learn from both successes and failures. This approach mirrors how simulations train machine learning algorithms to excel at game-playing tasks, culminating in achievements like Alphabet's AlphaGo defeating champions. Salakhutdinov may not possess insider knowledge of Apple's current projects, but he expects the company, along with other tech giants like Microsoft and Google, to be actively advancing in this field.