The rapid generation of high-quality images is essential for creating realistic simulated environments, which help train self-driving cars to navigate unpredictable hazards safely. However, current generative AI techniques, particularly diffusion models, are often too slow and computationally demanding. While autoregressive models, like those powering LLMs such as ChatGPT, operate much faster, they typically produce lower-quality images filled with errors. Researchers from MIT and NVIDIA have introduced HART (Hybrid Autoregressive Transformer), a new image generation method that combines the strengths of both approaches. HART utilizes an autoregressive model to outline the main features of an image quickly and then employs a smaller diffusion model to refine these details. This innovative tool generates images that rival or surpass the quality of state-of-the-art diffusion models but operates approximately nine times faster and with less computational resource usage, allowing for operation on ordinary laptops and smartphones. Applications for HART include assisting researchers in training robots for complex tasks and helping designers create captivating scenes for video games.

“Just like refining a rough painting with detailed brush strokes enhances its quality, HART combines broad image generation with meticulous detail work, ” says Haotian Tang, one of the lead authors of the research. Diffusion models, which require multiple steps to denoise images, can produce highly detailed visuals but are slow and resource-intensive. In contrast, autoregressive models generate images more swiftly by creating patches sequentially but suffer from information loss that leads to lower quality. HART counters these limitations by first predicting discrete image tokens with the autoregressive model, followed by using the diffusion model to add back any missing details, allowing for fast and high-quality images with only eight steps. During development, researchers faced integration challenges but improved HART's quality by applying the diffusion model solely for predicting residual tokens. Their final design employs a 700-million-parameter autoregressive model alongside a 37-million-parameter diffusion model, achieving image quality comparable to larger diffusion models (up to 2 billion parameters) while consuming 31% less computational power. Looking ahead, the team plans to build on the HART architecture to develop vision-language models and explore applications in video generation and audio prediction, potentially revolutionizing interactions with generative models. This research was supported by various organizations, including the MIT-IBM Watson AI Lab and NVIDIA, which provided GPU resources for training the model.