OpenAI's GPT-Red Model Outperforms Human Red-Teamers

OpenAI detailed GPT-Red this week, an internal automated model designed to identify prompt injection vulnerabilities in the company's AI systems. This initiative addresses the limitations of human red-teaming, which is described as time-intensive and difficult to scale, and the saturation of existing robustness evaluations by advanced models. The increasing complexity of AI agents, which interact with external data through browsers, applications, and files, expands the potential attack surface for malicious instructions embedded within this data.

GPT-Red functions as an iterative attacker, similar to a human red-teamer, by sending prompts, analyzing responses, and refining its approach to achieve specific adversarial goals. OpenAI trained GPT-Red at a significant compute scale, comparable to its largest post-training runs, with a primary focus on safety. Two key deployment strategies are in place: GPT-Red is isolated from deployed models to prevent its capabilities from falling into the wrong hands, and it performs dual roles in identifying vulnerabilities before deployment and generating attacks during the training process.

The model's training utilizes self-play reinforcement learning, where GPT-Red and a group of defender Large Language Models (LLMs) train concurrently across various red-teaming scenarios. The reward system incentivizes GPT-Red to elicit failures, such as successful prompt injections, while defender models are rewarded for resisting attacks and completing their intended tasks. This dual reward structure prevents defender models from simply refusing all actions, as they must still fulfill their operational objectives. Each training environment includes a threat model defining GPT-Red's control over elements like file content, web banners, or tool outputs, and specifying criteria for successful attacks.