A study on advanced versions of three popular AI chatbots reveals that they tend to generate incorrect answers more frequently than they admit when they don't know something. The research, led by José Hernández-Orallo from the Valencian Research Institute for Artificial Intelligence, analyzed the errors of large language models (LLMs), noting that while accuracy improves with model size and refinement, the rate of incorrect responses has also risen. Instead of opting to decline difficult questions, these models often provide answers, leading to an increase in misleading responses. Hernández-Orallo observes that chatbots are becoming more adept at mimicking knowledge without genuine understanding, a phenomenon described as "ultracrepidarianism. " This can lead to users overestimating chatbot abilities, which poses risks. The team examined models like OpenAI's GPT, Meta's LLaMA, and the open-source BLOOM, assessing their accuracy across various question types.

They found that even with improved models, over 60% of their responses were incorrect or unqualified. Moreover, human volunteers often miscategorized incorrect answers as correct, demonstrating a lack of ability to supervise the models effectively. To enhance user understanding, Hernández-Orallo suggests that developers should improve performance on simple questions and train chatbots to refrain from answering difficult ones. This would help users identify where AI is reliable and where it isn't. Although some chatbots can acknowledge their lack of knowledge, the push for models to tackle difficult questions remains prominent, especially for those marketed as general-purpose.