Supabase Releases Open Source AI Coding Benchmark

Supabase has open-sourced Supabase Evals, a benchmark and framework designed to test and score AI agents on real-world Supabase development tasks. This initiative aims to evaluate the performance of AI coding assistants when building applications on the Supabase platform. The framework allows for the testing of various coding agents, including Claude Code, Codex, and OpenCode, against authentic development challenges. These challenges encompass a range of activities such as constructing database schemas, debugging failed Edge Functions, and rectifying broken Row Level Security (RLS) policies. The results are then scored to provide a quantitative measure of the AI's efficacy.

Supabase Evals powers a public leaderboard available at supabase.com/evals, enabling users to compare the performance of different AI agents. Additionally, it supports an internal regression suite that is monitored on a daily basis to ensure consistent performance and identify any regressions. The benchmark is deployable locally and is available under the Apache-2.0 license. Running the benchmark requires a Docker daemon, provider API keys, and specific ports (54321–54329) to be free. The framework is designed to run locally via pnpm.

The Supabase Evals harness operates by evaluating AI agents across three key dimensions: products, topics, and stages. The 'products' dimension includes core Supabase services such as database, auth, storage, edge-functions, realtime, cron, queues, vectors, and data-api. The 'topics' dimension covers essential development areas like RLS, security, migrations, SQL, SDK usage, observability, self-hosting, testing, and declarative schema management. The 'stages' dimension categorizes the development process into build, deploy, investigate, and resolve phases. Supabase has curated a scenario set that touches each of these dimensions at least once, drawing from actual support tickets, bug reports, and GitHub issues to ensure relevance.

Scenarios are divided into two distinct suites. The benchmark scenarios are designed to assess the breadth of an AI agent's capabilities and are publicly available. In contrast, regression scenarios focus on known failure modes, are refreshed daily, and do not impact the published benchmark scores. Each scenario is executed within a realistic environment. The framework provisions a hosted-like stack and a local CLI project within containers, enabling the AI agents to interact with the actual Supabase Management Console (MCP) server and CLI. A platform-lite runtime provides a Management API-compatible interface, supported by the @supabase/lite library. The scoring mechanism combines deterministic checks with a 'LLM-as-a-judge' approach, and agents are granted one retry before their performance is graded. Each evaluation directory contains a PROMPT.md file detailing the task and frontmatter, along with an EVAL.ts file that defines the scoring logic.