Google Unveils Gemini 1.5 Pro With 1 Million Token Context Window

Google announced Gemini 1.5 Pro on February 15, 2024, a new generation of its flagship AI model that significantly expands its context window to 1 million tokens. This substantial increase allows the model to process and analyze vastly larger amounts of information, including entire books, hours of video, or extensive codebases, in a single prompt. The previous standard for many large language models was around 32,000 to 128,000 tokens.

This enhanced context window is powered by Google's new Mixture-of-Experts (MoE) architecture, which makes the model more efficient and performant. The MoE approach allows Gemini 1.5 Pro to selectively activate relevant parts of its neural network for specific tasks, leading to faster processing and reduced computational costs compared to dense models of similar size. Google stated that this architecture is a key innovation for scaling future AI models.

In demonstrations, Gemini 1.5 Pro successfully processed a 402-page PDF document, a 44-minute silent film, and 11 hours of audio in a single prompt, showcasing its advanced comprehension capabilities. The model also demonstrated a 5.7x improvement in performance over Gemini 1.0 Pro on standard benchmarks, particularly in long-context retrieval tasks. Google highlighted that the model maintains its performance across different modalities, including text, images, audio, and video.

Google plans to make Gemini 1.5 Pro available to developers and enterprise customers through Google AI Studio and Vertex AI starting February 15, 2024, with a limited preview. The company emphasized its commitment to responsible AI development, with built-in safety measures and ongoing research into mitigating potential risks associated with advanced AI capabilities. The 1 million token context window is initially available in a private preview, with plans for wider rollout.