Google has introduced a new AI model called Lumiere that utilizes a diffusion model called STUNet for video generation. This model helps determine the positioning of objects within a video (space) as well as their movement and changes over time (time). Unlike traditional methods that piece together smaller frames, Lumiere creates videos in a single process. It starts by constructing a base frame from an initial prompt and then uses the STUNet framework to predict the movement of objects within that frame. This process generates a series of frames that seamlessly flow into each other, resulting in the perception of smooth motion. In comparison to Stable Video Diffusion, Lumiere produces 80 frames instead of 25. Google has provided samples and prompts on the Lumiere website, allowing for comparison with other tools like Runway. While some clips may exhibit slight artificiality, particularly when examining skin texture or atmospheric scenes, the overall results are impressive. Notably, the movement of a turtle in water appears highly realistic.

A video editor who viewed the Lumiere intro video acknowledged that it was not entirely real, but believed it to be CGI until informed otherwise. In terms of video generation, other models rely on pre-existing key frames to stitch together the movement, similar to a flip book, whereas STUNet enables Lumiere to focus on the movement itself and generate content accordingly. Although Google has not been a major player in the text-to-video category, it has gradually released more advanced AI models and embraced a multimodal approach. The Gemini large language model, for instance, will eventually incorporate image generation into Bard. While Lumiere is not currently available for testing, it demonstrates Google's ability to develop an AI video platform that is on par, if not slightly superior, to existing video generators such as Runway and Pika. It's worth emphasizing that this advancement represents Google's progress in AI video over the past two years. In addition to text-to-video generation, Lumiere will support image-to-video generation, stylized generation for videos in specific styles, cinemagraphs that animate only a portion of a video, and inpainting to modify colors or patterns in selected areas. However, the Lumiere paper acknowledges the potential for misuse in creating fake or harmful content and stresses the importance of developing tools to detect biases and malicious use cases for responsible and fair usage. The paper does not provide specific details on how this goal can be achieved.