Google Research scientists have recently published a paper introducing GameNGen, an AI-powered game engine that generates original Doom gameplay using a neural network. GameNGen utilizes Stable Diffusion, a cutting-edge technology developed by scientists Dani Valevski, Yaniv Leviathan, Moab Arar, and Shlomi Fruchter. This AI-based engine leverages previous frames and current player input to produce new frames that exhibit remarkable visual fidelity and coherence. One of the remarkable accomplishments of GameNGen is its ability to generate a complete game engine with consistent logic. Players can experience Doom gameplay with features like turning, strafing, weapon firing, and accurate damage from enemies and environmental hazards. As players explore the game, levels are dynamically built in real-time, with the game even keeping track of ammunition for their pistols. Remarkably, the generated gameplay is so authentic that it is often difficult to distinguish it from actual Doom gameplay in short clips. The game runs at a smooth 20 frames per second (FPS). To train GameNGen accurately in modeling its own Doom levels, the team at Google trained an AI agent to play Doom at various difficulties and simulate different player skill levels.

The agent was rewarded for actions like collecting power-ups and completing levels, while damage taken or death resulted in punishment. This training process provided hundreds of hours of visual training data for the GameNGen model to reference and recreate. A key innovation in this study lies in how the scientists maintained coherence between frames over extended periods using Stable Diffusion. Stable Diffusion is a widely used generative AI model that generates images based on image or text prompts. However, it often suffers from a lack of continuity between consecutive frames and a decline in visual quality over time, as demonstrated in Corridor's Anime Rock Paper Scissors short film. To address these challenges, Google Research improved Stable Diffusion by training new frames using a longer sequence of user inputs and preceding frames, while also introducing Gaussian noise to corrupt context frames. Additionally, a separate but interconnected neural network is utilized to fix and constantly self-correct the context frames, resulting in images with high levels of stability and visual coherency for extended periods. Although GameNGen's current examples are not flawless, there are occasional instances of visual artifacts such as blobs, blurs, and flickering elements. Dead enemies may transform into blurry masses, and the HUD's Doomguy character continuously raises and lowers his eyebrows like The Rock on Monday Night Raw. Furthermore, the generated levels may exhibit inconsistencies, as shown in the embedded YouTube video, where Doomguy suddenly stops taking damage at 4% and the layout of the level changes completely after a 360-degree turn inside a poison pit.