The rapid advancements in artificial intelligence (AI) have unveiled new possibilities, particularly in reasoning models based on inference-time compute, leading to discussions about the potential for artificial general intelligence (AGI). OpenAI CEO Sam Altman and Anthropic CEO Dario Amodei both highlight the prospect of reaching human-level AI within the next few years. However, current AI systems lack a critical capability: the ability to learn continuously after training. Today’s AI operates in two phases—training and inference. After the training phase, the model's knowledge becomes static, and it cannot adapt or learn from new data during inference. To update its knowledge, the model typically requires extensive retraining, which is resource-intensive and impractical.

This limitation prevents AI from achieving continuous improvement like human intelligence, which learns and adapts in real-time. Several workarounds exist, such as model fine-tuning, retrieval-augmented generation (RAG), and in-context learning, but none achieve true continuous learning. These approaches are limited by scalability issues, the necessity of coinciding data distribution, and the ephemeral nature of in-context learning. The promise of continual learning lies in its ability to create personalized AI applications that adapt over time, enhancing user engagement and creating competitive advantages or "moats" for AI companies. In contrast to the static nature of current models, continual learning would enable AI systems to evolve based on ongoing interactions, making them more tailored to individual users. Despite the transformative potential, continual learning faces challenges, particularly catastrophic forgetting, where new learning degrades previous knowledge. Solutions in existing methods—such as replay-based techniques, regularization, and architectural approaches—have their limitations, particularly regarding scalability and adaptability to new environments. Recent innovations in continual learning have emerged, particularly from startups like Writer and Sakana. Writer’s self-evolving models adapt in real-time by storing insights in a short-term memory pool within the model layers, while Sakana's Transformer² enables dynamic adjustments of model weights based on the task context, highlighting a shift toward more adaptable AI systems. In conclusion, as research progresses in continuous learning, AI's capability to learn and adapt throughout its operational lifetime holds immense potential. This evolution will reshape perceptions of AI and redefine possibilities in the field, creating new avenues for adaptability and competitive advantage.