Pruna AI, a European startup focused on developing compression algorithms for AI models, is launching its optimization framework as open source this Thursday. The company has designed a framework that implements various efficiency techniques, including caching, pruning, quantization, and distillation, to optimize AI models effectively. “Our framework standardizes the process of saving and loading compressed models, combines these compression techniques, and evaluates the performance of your compressed model post-optimization, ” said John Rachwan, co-founder and CTO of Pruna AI, in an interview with TechCrunch. Specifically, Pruna AI’s framework can assess whether significant quality loss occurs after compressing a model and the performance improvements achieved. “To use a metaphor, we are akin to Hugging Face in terms of standardizing transformers and diffusers — establishing how to call them, save, and load them, etc. We are doing the same for efficiency methods, ” he noted. Major AI laboratories are already leveraging various compression techniques. For example, OpenAI has used distillation to develop faster iterations of its core models. This approach likely contributed to creating GPT-4 Turbo, a speedier version of GPT-4. The Flux. 1-schnell image generation model is another example, serving as a distilled variant of the Flux. 1 model from Black Forest Labs. Distillation involves extracting knowledge from a larger AI model through a "teacher-student" framework. Developers issue requests to a teacher model and capture the outputs. These responses may then be compared to a dataset for accuracy, guiding the training of the student model to emulate the teacher's behavior. “For large companies, they typically develop these solutions in-house. In the open-source community, you often find tools focusing on single methods, like one quantization technique for large language models or one caching approach for diffusion models, ” Rachwan explained. “However, there’s a lack of comprehensive tools that integrate and simplify all these methods.

This is the key benefit that Pruna offers. ” While Pruna AI supports any model types—ranging from large language models to diffusion models, speech-to-text systems, and computer vision applications—the company is currently placing greater emphasis on image and video generation models. Among Pruna AI’s current clients are Scenario and PhotoRoom. Besides the open-source version, Pruna AI offers an enterprise solution with advanced optimization capabilities, including an optimization agent. “The most thrilling feature we’ll soon release is a compression agent, ” Rachwan revealed. “You simply provide your model and specify, ‘I need more speed without sacrificing accuracy by more than 2%. ’ The agent then performs its magic, determining the best combination and presenting it to you without any additional work required from the developer. ” Pruna AI charges by the hour for its professional version. “It’s comparable to renting a GPU on AWS or other cloud services, ” Rachwan added. If your model is a critical element of your AI infrastructure, optimizing it can lead to substantial cost savings on inference. For instance, Pruna AI has reduced a Llama model’s size by eight times with minimal loss through its compression framework. The company aspires for clients to view its compression framework as a self-sustaining investment. Recently, Pruna AI completed a seed funding round, raising $6. 5 million. Notable investors include EQT Ventures, Daphni, Motier Ventures, and Kima Ventures.