Google Unveils Gemini 1.5 Pro With 1 Million Token Context Window

Google announced Gemini 1.5 Pro, a new generation of its flagship AI model, on February 15, 2024. The most significant advancement is its expanded context window, capable of processing up to 1 million tokens. This represents a tenfold increase over the 100,000 tokens offered by Gemini 1.0 Pro, allowing the model to analyze and reason over substantially larger volumes of data, including entire codebases, multiple lengthy documents, or hours of video.

This enhanced context window enables Gemini 1.5 Pro to perform complex tasks that were previously infeasible. For instance, it can analyze up to 1 hour of video, 11 hours of audio, codebases with over 30,000 lines, or documents exceeding 700,000 words. Google demonstrated this capability by having Gemini 1.5 Pro identify a specific object within a 40-minute silent film, a task requiring deep understanding of visual context over an extended period.

The model also incorporates a Mixture-of-Experts (MoE) architecture, which Google states makes it more efficient and faster to train and serve compared to previous dense models. While the exact details of the MoE implementation were not fully disclosed, the company indicated it contributes to improved performance and scalability. Gemini 1.5 Pro is currently available in a limited preview for developers and enterprise customers, with broader availability planned for later in 2024.

Google highlighted that Gemini 1.5 Pro maintains the multimodal capabilities of its predecessor, understanding and processing text, images, audio, and video. The expanded context window is expected to unlock new applications in areas such as long-form content analysis, code comprehension, and detailed historical research. The company emphasized its commitment to responsible AI development, with safety and ethical considerations integrated into the model's design and deployment.