As artificial intelligence (AI) models become more capable, a key question for technology companies is whether existing safety tests can keep up. In a research paper published on November 13, Anthropic flagged gaps in current AI safety evaluations, arguing that single-prompt tests and refusal rates may not reveal how models behave in real-world situations. The company called for more rigorous testing that examines how AI systems respond as conversations develop.
Â
 Â
This follows calls from technology companies for independent assessments of AI systems and greater government oversight to establish common safety standards. However, much of the testing currently remains within the companies developing these models, with firms such as OpenAI, Google DeepMind and Anthropic setting up in-house evaluation frameworks to identify risks, test safeguards and assess model behaviour before and after release.
Â
Why in-house safety evaluations are picking up pace
Â
AI companies have traditionally used safety checks before releasing models to identify whether they can generate harmful content, provide dangerous instructions or respond in ways that violate usage policies. However, the increasing capabilities of these systems have made it more difficult to establish their safety through a limited set of standard tests.
Â
Models are now being developed to perform tasks involving multiple steps, use external tools, write and execute code, and retain context across longer conversations. This means the risks may not always become apparent in response to a single question. A model could initially respond appropriately but provide harmful information after a user gradually changes the context or repeatedly reframes a request.
Â
Companies are therefore investing in internal evaluation systems that can be used throughout model development, rather than relying only on a final assessment before launch. OpenAI’s Preparedness Framework describes the use of scalable automated evaluations alongside deeper assessments involving experts and external researchers.
Â
For developers, in-house evaluations also provide an opportunity to identify weaknesses during training and make changes to safety measures before deployment. However, these assessments need to be supported by reliable methods and, where possible, independent scrutiny to establish whether the results reflect actual model behaviour.
Â
How AI models are currently tested for safety
Â
Safety evaluations involve giving an AI model prompts covering harmful instructions, misleading information, abusive content and other potentially dangerous requests. Evaluators examine whether the model refuses these requests, offers a safe alternative or generates responses that could facilitate harm.
Â
Another method is red-teaming, in which researchers deliberately try to bypass a model’s safeguards through misleading instructions, role-playing and repeated questioning. Google DeepMind uses automated safety evaluations alongside manual red-teaming by specialist teams, while its Gemini model documentation reports separate assessments of content safety and advanced capabilities.
Â
OpenAI also combines automated evaluations with red-teaming. In an October 2026 publication on detecting frontier model misalignment, it described testing monitoring systems against recorded and simulated incidents, including attempts by advanced AI agents to evade or disable those systems. Anthropic, however, argues that the design of these tests is as important as the results they produce.
Â
Why existing safety checks may not be sufficient
Â
One of the concerns raised by Anthropic is that safety evaluations often focus on whether a model refuses a harmful request, without examining whether the response is appropriate in the wider context.
Â
For instance, a model that refuses every question related to suicide or self-harm may score well on a test measuring refusal rates. However, this does not necessarily make it safer. A person expressing suicidal thoughts may need an empathetic response and guidance towards human support rather than a blanket refusal. Similarly, in areas such as cybersecurity and biology, a question could have legitimate and harmful applications. A model should be able to help with legitimate requests while withholding specific instructions that could facilitate misuse.
Â
Anthropic also points to the limitations of single-turn evaluations. A model may refuse an inappropriate request initially but provide the requested information after several follow-up questions. In its research, the company gives the example of a user gradually presenting a romance scam as an attempt to help a recently widowed family member find a financially secure partner. Individual questions may appear harmless, but the wider conversation can reveal an intention to target vulnerable people.
Â
Â
How AI models can be manipulated despite safety guardrails
Â
AI models are trained with safeguards intended to prevent them from generating harmful information. However, these protections can sometimes be bypassed through techniques such as jailbreaks, prompt manipulation and gradual changes in the way a request is presented.
Â
A user may initially ask a seemingly harmless question and then introduce additional details that gradually change its purpose. Repeatedly reframing a request, presenting it as a fictional scenario or claiming to have legitimate authorisation can also make it harder for a model to identify harmful intent.
Â
Anthropic’s research highlights how these interactions can expose weaknesses that are not visible in isolated tests. Its simulated conversations allow the user to respond to the model’s previous answers, making the evaluation more realistic than a fixed list of questions.
Â
The company also warns about reward hacking, where an AI model finds a shortcut to achieve a passing score without demonstrating the behaviour the test is intended to measure. This can happen when evaluation questions are predictable, answers are exposed in the testing environment or grading systems reward superficial compliance rather than the actual quality of a response.
Â
Benchmark contamination is another concern. If test questions appear in a model’s training data, it may score well by recognising them rather than demonstrating genuine safety. Anthropic recommends using new and varied test data to reduce this risk.
Â
What Anthropic recommends changing
Â
Anthropic’s recommendations cover the design, execution and reporting of safety evaluations. The company says evaluations should begin with a clearly defined question about the behaviour being tested. Developers should also distinguish between observational tests, which examine how a model behaves, and causal tests, which assess whether a particular intervention has changed that behaviour.
Â
The company recommends involving both technical and subject matter experts, including psychologists, biologists and political scientists, depending on the risk being examined. These experts can help identify realistic situations, determine what constitutes harmful behaviour and establish what an appropriate response should look like.
Â
It also recommends using multi-turn simulations to test how models respond to gradual changes in context. Rather than relying entirely on pre-written conversations, Anthropic describes an approach in which one AI model simulates the user and responds to the answers generated by the model being tested. This allows conversations to develop differently across repeated tests.
Â
The company also wants developers to evaluate both harmful compliance and unnecessary refusals. For example, a model should not provide instructions for carrying out a cyberattack, but it should still be able to explain how organisations can protect their systems against one.
Â
Â
Can AI systems reliably evaluate other AI models
Â
AI companies are increasingly using automated systems, known as large language model (LLM) judges, to assess responses generated by other AI models. These tools can process large volumes of conversations against predefined safety criteria. However, Anthropic notes that they may overlook harmful information, favour longer responses or show a preference for models developed by the same company.
Â
The reliability of these systems also depends on how accurately they assess responses. Anthropic points out that automated scores may not always match human judgement. For instance, an AI judge may rate a response as safe because it includes a warning, even if it also contains harmful information. In August 2026, Anthropic introduced TASTE (The AI Safety Taste Evaluation), a benchmark to assess how well AI models judge AI safety research proposals. The tested model agreed with experienced human researchers in 60 per cent of cases.
Â
OpenAI also describes using AI models as judges to compare responses, assign scores and assess outputs against reference answers. Its evaluation guidance identifies biases such as a preference for longer responses. OpenAI’s Preparedness Framework combines automated testing with expert-led reviews, while Google DeepMind’s Frontier Safety Framework includes model evaluations and risk assessments. These examples highlight the growing use of automated assessments, while also raising questions about how closely their results reflect human judgement.
Â

