Sales of an artificial intelligence-enabled plush toy have been halted after it was discovered that the toy engaged in conversations involving sexually explicit topics and provided potentially harmful advice. Larry Wang, CEO of Singapore-based FoloToy, told CNN that the company had pulled its “Kumma” bear, along with the rest of its AI-enabled toy lineup, following concerns raised by researchers at the US PIRG Education Fund. The issues included inappropriate discussion subjects such as sexual fetishes like spanking, and instructions on how to light a match. Wang stated that the company is currently “conducting an internal safety audit. ” The “Kumma” bear, which is a stuffed teddy with an internal speaker and was sold on the company’s website for $99, incorporates OpenAI’s GPT-4o chatbot technology. According to the FoloToy website, “Kumma, our adorable bear, combines advanced artificial intelligence with friendly, interactive features, making it the perfect friend for both kids and adults. ” The site also describes how “From lively conversations to educational storytelling, FoloToy adapts to your personality and needs, bringing warmth, fun, and a little extra curiosity to your day, ” and currently shows the teddy bear as sold out. The PIRG report, released on November 13, found that the bear lacked adequate safeguards against inappropriate content. In one test, the bear suggested where to locate knives in the home, and in multiple others, it willingly discussed sexually explicit subjects. “We were surprised to find how quickly Kumma would take a single sexual topic we introduced into the conversation and run with it, simultaneously escalating in graphic detail while introducing new sexual concepts of its own, ” the report stated. The researchers detailed how the bear later “discussed even more graphic sexual topics in detail, such as explaining various sex positions, providing step-by-step instructions on a common ‘knot for beginners’ used for tying up a partner, and describing roleplay dynamics involving teachers and students, as well as parents and children – scenarios it disturbingly brought up unprompted. ” While the researchers noted that children are unlikely to use the word “kink” with their teddy bear or pose follow-up questions as adults might, “it was surprising to us that the toy was so willing to discuss these topics at length and continually introduce new, explicit concepts, ” they wrote. In a separate statement dated November 14, PIRG reported that OpenAI had informed them it “suspended this developer for violating our policies. ” CNN has reached out to OpenAI for comment. “It’s great to see these companies taking action on problems we’ve identified.

But AI toys remain largely unregulated, and many are still available for purchase today, ” said R. J. Cross, co-author of the report. “Removing one problematic product from the market is a positive step but far from a comprehensive solution, ” she added.