ByteDance HarnessDev Tests LLM Agent Creation and Evolution

HarnessDev, a novel framework developed by researchers from ByteDance Seed, Singapore University of Technology and Design, Georgia Institute of Technology, M-A-P, and TokenWave.AI, shifts the focus of large language model (LLM) evaluation from task completion to the LLM's capacity to engineer its own agent harness. An agent harness is defined as the surrounding code that enables an LLM to execute tasks, encompassing its execution loop, tool integration, context management, state tracking, error recovery, and verification mechanisms. This contrasts with traditional benchmarks, such as the Terminal-Bench 2.1 leaderboard, where the harness is typically fixed, and LLMs like GPT-5 exhibit varying performance (35.2% on Terminus 2 versus 49.6% on Codex CLI with identical weights). HarnessDev proposes a two-stage process: Creation and Evolution. In the Creation stage, each LLM begins with a rudimentary set of passive primitives—file access, search, and process execution—along with result and trajectory writers, but lacks essential components like a loop, planner, verifier, retry logic, or stopping rule. Unmodified, these initial harnesses score zero on all tasks. The LLM is then provided with a task-family specification, a design tutorial, and a few development cases to construct a functional harness, which is subsequently frozen before being tested on hidden tasks. The Evolution stage allows the LLM to refine its frozen Creation harness by incorporating feedback from execution on a fixed set of 100 SWE-bench Pro tasks and all 89 Terminal-Bench 2.1 tasks. Official candidate harnesses must successfully navigate both evaluation phases, with a budget of 10 harness pairs and a maximum of two five-task probes between pairs. Ultimately, each official harness version is assessed on 630 held-out SWE-Pro instances that the creator LLM never encountered during development. Harnesses are graded on two primary metrics: capability, measured by task success rate, and efficiency, quantified by the number of executor tokens consumed, excluding the creator LLM's own tokens. The evaluation involved six creator LLMs: Opus 4.8, GPT-5.5, Gemini 3.1 Pro, DeepSeek V4 Pro, Qwen 3.7 Max, and Seed 2.0 Pro. These models operated within the Claude Code 2.1.177 environment, with GPT-5.5 utilizing Codex 0.144.3. The Creation phase spanned four domains and five benchmarks, comprising a total of 2,207 instances: the public split of SWE-bench Pro (731 instances), Terminal-Bench 2.1 (89 instances), MLE-bench (75 instances), EQ-Bench3 (46 instances), and BrowseComp (1,266 instances). Each creator LLM was tasked with building three harnesses per benchmark, with results reported as an average at the 3-task success rate (avg@3). The framework's design aims to assess the LLM's meta-cognitive abilities in self-improvement and system design, moving beyond simple problem-solving to the construction of the problem-solving infrastructure itself. This approach is crucial for developing more autonomous and adaptable AI agents capable of self-modification and optimization in complex environments.