Large language models (LLMs) are capable of impressive feats, such as writing poetry and generating functional computer programs, primarily by predicting subsequent words in text. While these capabilities suggest that LLMs might learn general truths about the world, a new study challenges that assumption. Researchers found that a popular type of generative AI could provide highly accurate driving directions in New York City without actually having an accurate internal map of the area. When they altered the environment by closing streets and adding detours, the AI's performance significantly declined. The study indicated that the model's internal representation of New York City was flawed, featuring many nonexistent streets and illogical connections. This raises concerns that AI models can appear to perform well in specific contexts but may fail under slightly altered conditions. Senior author Ashesh Rambachan emphasized the importance of understanding whether LLMs grasp coherent models of the world if researchers aim to apply these tools in scientific fields. The research team included multiple collaborators and will present their findings at the Conference on Neural Information Processing Systems. The researchers concentrated on transformer models, the backbone of LLMs like GPT-4, which are trained on extensive language data to predict tokens.

The team developed two new metrics—sequence distinction and sequence compression—to evaluate whether a transformer had formed a sound world model. They tested these metrics on two deterministic problems: navigating New York City streets and playing Othello. Contrary to expectations, transformers trained on random sequences performed better in developing coherent world models, likely due to exposure to a broader range of possible moves during training. Despite generating accurate outputs, only one model showed coherence in Othello, and none effectively navigated New York under the new conditions. When the researchers simulated detours, the AI's accuracy dropped dramatically from nearly 100 percent to 67 percent with just a minor change. The generated maps depicted an unrealistic and overly complex version of New York City, underscoring that transformers can excel at tasks without understanding the underlying rules. Rambachan urges caution against assuming these models understand the world simply because they achieve impressive results. Future research will explore diverse problems with gradually understood rules and apply the new evaluation metrics to real-world scientific challenges. This study is supported by various grants, including those from Harvard and the National Science Foundation.