Google Updates Android Bench with New LLMs

Google updated its Android Bench benchmark on May 15, 2024, to evaluate the performance of large language models (LLMs) in Android app development. The benchmark now includes eight new models, aiming to provide developers with a clearer understanding of which AI agents excel at various coding tasks. This update incorporates a new framework designed for easier use and invites developers to submit their own test results and feedback to influence the benchmark's future development.

The Android Bench suite assesses LLMs across 100 specific Android development tasks. Since its initial launch in March, Google has expanded the metrics to include cost and efficiency, alongside support for open-weight models. The latest additions to the leaderboard feature prominent LLMs such as Claude Fable 5, Claude Sonnet 5, Claude Opus 4.8, GLM 5.2, Kimi K2.7 Code, MiniMax M3, Qwen 3.7 Plus, and Qwen 3.7 Max. These additions aim to reflect the rapidly evolving landscape of AI coding assistants.

While LLMs are increasingly popular for code generation, their effectiveness varies significantly across different development tasks. Android Bench seeks to differentiate between useful AI outputs and less reliable ones by providing a standardized evaluation. The inclusion of these new models allows for a more comprehensive comparison of current AI capabilities in the context of Android development, although the benchmark's data indicates that Google's own Gemini model continues to show performance gaps compared to some of the leading models in this specific area.