Alibaba Details Qwen AI Model Evolution From 7B to 2.4T

Alibaba Cloud's AI model family, Qwen, has undergone rapid development since its initial unveiling in April 2023, culminating in the release of open-weight versions with up to 2.4 trillion parameters. The journey began on April 7, 2023, with the demonstration of a chatbot named Tongyi Qianwen, whose name translates to 'truth from a thousand questions' and draws inspiration from the philosopher Mencius. This initial model was slated for integration into Alibaba's various business applications, including DingTalk and the Tmall Genie voice assistant, as announced by then-CEO Daniel Zhang at the Alibaba Cloud Summit in Beijing on April 11, 2023. A significant milestone was reached on August 3, 2023, when Alibaba open-sourced Qwen-7B and Qwen-7B-Chat. This release was positioned as a direct competitor to Meta's Llama 2. Qwen-7B was pretrained on an extensive dataset of over 2.2 trillion tokens and featured a 2,048-token context window. Its license permitted free commercial use for entities with fewer than 100 million monthly active users. The multimodal capabilities of the Qwen family emerged shortly thereafter, with the launch of Qwen-VL, the first vision-language branch, in late August 2023. Further public accessibility was granted on September 13, 2023, when Tongyi Qianwen was opened to the general public, indicating regulatory approval within China. The team also published a technical report detailing the Qwen models on arXiv in September 2023. The year concluded with an expansion in model scale, as Alibaba released its 72B and 1.8B parameter models for download around December 1, 2023, offering a range of sizes from laptop-compatible to frontier-scale open weights. The year 2024 marked Qwen's ascent as a preferred choice for developers, with three major generations released within eight months. The Qwen1.5 series, launched on February 5, 2024, was presented as a beta version of Qwen2. This iteration included dense models ranging from 0.5 billion to 72 billion parameters, boasting a stable 32K context window. The evolution continued with the release of Qwen2, which introduced models up to 72B parameters and expanded context windows to 128K tokens. The most significant development in 2024 was the release of the Qwen1.5-72B-Chat model, which achieved a score of 82.1 on the MT-Bench benchmark, surpassing other open-weight models like Llama-3-70B-Instruct and Mixtral-8x22B-Instruct. The Qwen team also announced the development of a 2.4 trillion parameter model, though its weights are not yet publicly available. This model demonstrated advanced reasoning capabilities, including the ability to process and analyze video content, a feature that sets it apart from many contemporary AI models. The Qwen series continues to be a significant contribution to the open-source AI landscape, providing developers with powerful and versatile tools for a wide range of applications.