Anthropic Releases Claude Plugin Evaluation Workflow

Anthropic has introduced a new plugin evaluation workflow for Claude Code, a tool designed to help developers assess the effectiveness of their plugins. The `claude plugin eval` command allows users to run a plugin against realistic prompts, grade the output generated by Claude, and compare it to a scenario where the plugin is not active. This workflow addresses three critical questions that were previously difficult for plugin developers to measure: whether a specific skill or function within the plugin is triggered, if the plugin's performance remains consistent when edited or when a new model version is used, and whether the plugin's output is superior to that of a bare model without the plugin. The evaluation system is deployable and compatible with Claude Code version 2.1.269 or later. It functions with any directory containing a `plugin.json` or `.claude-plugin/plugin.json` manifest file, or a skills-directory plugin. Each evaluation run and the grading process involve actual model calls, which are billed to the user's plan or API account. An evaluation suite is structured within an `evals/` directory inside the plugin. Each individual case is organized as a subdirectory containing a `prompt.md` file and a `graders/` folder. The content of the `prompt.md` file is directly fed to Claude, with `@path` mentions not being expanded. Frontmatter within `prompt.md` can be used to configure parameters such as `max_turns` (defaulting to 10), `timeout_seconds` (defaulting to 300), the specific `model` to be used, `tags` for categorization, and `allowed_tools`. The graders are defined in markdown files, where frontmatter specifies the `type` of grader, an optional `weight` for its influence on the overall score, and an optional `arm` for conditional grading. Anthropic has defined six distinct grader types. Four of these graders are free to use as they are computed directly from the transcript and local files: `regex` (regular expression matching), `tool_used` (checking if a tool was invoked), `tool_order` (verifying the sequence of tool usage), and `file_exists` (confirming the presence of a file). The remaining two graders, `llm` and `baseline`, incur costs as they involve calls to a judge model. The `llm` grader scores Claude's reply against prose-based criteria defined by the developer, while the `baseline` grader compares the output against a predefined reference answer. The `claude plugin eval init` command assists developers by reading the plugin, prompting for a description of a desired outcome, proposing test cases and graders, executing initial trials, and generating the necessary files. In continuous integration (CI) environments, the `--bare <name>` option can be used to generate a blank template for a specific plugin. A key metric in this evaluation process is the difference, denoted as Δ (delta). By default, each case is executed twice: once with the plugin's functionality enabled (`with-arm`) and once without it (`without-arm`). The Δ value represents the contribution of the plugin to the outcome. If a case achieves a perfect score of 1.0 in both the `with-arm` and `without-arm` evaluations, it indicates that the plugin was not the determining factor in the case's success. The documentation provides an example output illustrating a single case with a `WITH` score of 1.00, a `W/OUT` score of 0.33, resulting in a Δ of +0.67 across six runs.