Google Unveils Gemini 1.5 Pro With 1 Million Token Context Window

Google announced the public preview of Gemini 1.5 Pro on February 15, 2024, a significant advancement in large language model capabilities, most notably featuring an unprecedented 1 million token context window. This expanded context window allows the model to process and analyze vastly larger amounts of information in a single prompt compared to previous iterations. For instance, Gemini 1.5 Pro can ingest and reason over an entire hour of video, 11 hours of audio, or over 30,000 lines of code. This capability is a substantial leap from the standard 128,000 token context window offered by Gemini 1.0 Pro and other contemporary models.

The enhanced context window is powered by Google's new Mixture-of-Experts (MoE) architecture, which enables more efficient processing of long sequences. This architectural shift allows the model to selectively activate relevant parts of its network for specific tasks, leading to improved performance and reduced computational overhead for handling extensive data. The MoE architecture is a key innovation that underpins the model's ability to manage such a large context without a proportional increase in processing costs or latency.

Gemini 1.5 Pro also demonstrates strong performance across a range of standard benchmarks, including Massive Multitask Language Understanding (MMLU) and BIG-bench Hard (BBH), achieving scores comparable to Gemini 1.0 Pro. The model's multimodal capabilities are retained, allowing it to understand and process text, images, audio, and video simultaneously. Google highlighted its ability to perform complex reasoning tasks, such as identifying specific moments in a video or extracting information from lengthy documents, with remarkable accuracy.

Developers can access Gemini 1.5 Pro through the Google AI Studio and Vertex AI platforms. Google is also offering an experimental 2 million token context window for select customers through an application process, further pushing the boundaries of AI's information processing capacity. This move positions Gemini 1.5 Pro as a powerful tool for applications requiring deep comprehension of extensive datasets, from code analysis and long-form content summarization to video and audio transcription and analysis. The company emphasized its commitment to responsible AI development, with safety filters and evaluations integrated into the model's deployment.