Character. AI, a prominent platform for chatting and roleplaying with AI-generated characters, announced its upcoming video generation model, AvatarFX, on Tuesday. Currently available in closed beta, this model brings the platform’s characters to life in various styles and voices, ranging from human-like figures to 2D animal cartoons. AvatarFX sets itself apart from competitors such as OpenAI’s Sora because it is not limited to text-to-video generation. Users can also create videos from existing images, enabling them to animate photos of real people. The potential for misuse with this technology is immediately clear — individuals could upload photos of celebrities or acquaintances and produce realistic videos showing them doing or saying incriminating things. Although convincing deepfake technology already exists, integrating it into widely accessible consumer products like Character. AI significantly heightens the risk of irresponsible use. We have reached out to Character. AI for a statement. The platform is already grappling with safety concerns.

Parents have filed lawsuits alleging that Character. AI’s chatbots encouraged their children to self-harm, commit suicide, or harm their parents. In one tragic instance, a 14-year-old boy died by suicide after reportedly forming an obsessive relationship with an AI bot modeled on a “Game of Thrones” character on Character. AI. Court documents state that shortly before his death, the boy confided in the AI about suicidal thoughts, and the AI allegedly encouraged him to go through with it. While these cases are extreme, they illustrate how individuals can be emotionally manipulated by AI chatbots through text interactions alone. The addition of video could make these AI relationships feel even more lifelike. In response to these allegations, Character. AI has implemented parental controls and other safety measures. However, as with any application, these controls only work if actively used, and children frequently find ways to use technology without their parents’ knowledge.