Google Unveils Gemini 1.5 Pro With 1 Million Token Context Window

Google announced the public preview of its Gemini 1.5 Pro large language model on February 15, 2024, featuring a groundbreaking 1 million token context window. This significant expansion allows the model to process and analyze substantially larger volumes of information compared to previous iterations. The previous standard context window for many advanced models hovered around 32,000 to 128,000 tokens, making the 1 million token capacity a tenfold increase for some applications.

The 1 million token context window means Gemini 1.5 Pro can ingest and reason over the equivalent of approximately 1,500 pages of text, or over an hour of video, or over 30,000 lines of code. This capability is particularly transformative for tasks requiring the understanding of lengthy documents, complex codebases, or extensive audio and video content. For instance, developers can now feed an entire codebase into the model to identify bugs or suggest optimizations, and researchers can analyze lengthy research papers or historical archives in their entirety.

Google highlighted specific use cases, including the ability to analyze a 402-page document and answer detailed questions about its content, or to process a 44-minute YouTube video and summarize its key points. The model also demonstrated its capacity to sift through 11 hours of audio to identify a specific spoken phrase. This enhanced context window is powered by a new Mixture-of-Experts (MoE) architecture, which Google states makes the model more efficient and scalable. The MoE approach allows the model to selectively activate different parts of its neural network for different tasks, leading to faster processing and reduced computational cost for certain operations.

While the 1 million token context window is available in a limited preview, Google also announced that a 128,000 token context window is now generally available for Gemini 1.5 Pro. This broader availability still represents a significant leap from earlier models and offers substantial utility for a wide range of applications. The company emphasized its commitment to responsible AI development, noting that safety filters and responsible AI practices are integrated into the model's deployment. The introduction of Gemini 1.5 Pro with its expanded context window positions Google at the forefront of advancements in large language model capabilities, particularly in handling and understanding vast amounts of data.