As AI scrutiny grows, tech companies strengthen internal safety evaluations | Artificial Intelligence News


As artificial intelligence (AI) models become more capable, a key question for technology companies is whether existing safety tests can keep up. In a research paper published on November 13, Anthropic flagged gaps in current AI safety evaluations, arguing that single-prompt tests and refusal rates may not reveal how models behave in real-world situations. The company called for more rigorous testing that examines how AI systems respond as conversations develop.

 

Meanwhile, OpenAI and Google DeepMind are also strengthening their AI safety evaluations. OpenAI combines automated tests with expert reviews under its Preparedness Framework, while Google uses automated assessments and human red-teaming to identify potential risks in Gemini models.

  

This follows calls from technology companies for independent assessments of AI systems and greater government oversight to establish common safety standards. However, much of the testing currently remains within the companies developing these models, with firms such as OpenAI, Google DeepMind and Anthropic setting up in-house evaluation frameworks to identify risks, test safeguards and assess model behaviour before and after release.

 

Why in-house safety evaluations are picking up pace

 

AI companies have traditionally used safety checks before releasing models to identify whether they can generate harmful content, provide dangerous instructions or respond in ways that violate usage policies. However, the increasing capabilities of these systems have made it more difficult to establish their safety through a limited set of standard tests.

 

Models are now being developed to perform tasks involving multiple steps, use external tools, write and execute code, and retain context across longer conversations. This means the risks may not always become apparent in response to a single question. A model could initially respond appropriately but provide harmful information after a user gradually changes the context or repeatedly reframes a request.

 

Companies are therefore investing in internal evaluation systems that can be used throughout model development, rather than relying only on a final assessment before launch. OpenAI’s Preparedness Framework describes the use of scalable automated evaluations alongside deeper assessments involving experts and external researchers.

 

For developers, in-house evaluations also provide an opportunity to identify weaknesses during training and make changes to safety measures before deployment. However, these assessments need to be supported by reliable methods and, where possible, independent scrutiny to establish whether the results reflect actual model behaviour.

 

How AI models are currently tested for safety

 

Safety evaluations involve giving an AI model prompts covering harmful instructions, misleading information, abusive content and other potentially dangerous requests. Evaluators examine whether the model refuses these requests, offers a safe alternative or generates responses that could facilitate harm.

 

Another method is red-teaming, in which researchers deliberately try to bypass a model’s safeguards through misleading instructions, role-playing and repeated questioning. Google DeepMind uses automated safety evaluations alongside manual red-teaming by specialist teams, while its Gemini model documentation reports separate assessments of content safety and advanced capabilities.

 

OpenAI also combines automated evaluations with red-teaming. In an October 2026 publication on detecting frontier model misalignment, it described testing monitoring systems against recorded and simulated incidents, including attempts by advanced AI agents to evade or disable those systems. Anthropic, however, argues that the design of these tests is as important as the results they produce.

 

Why existing safety checks may not be sufficient

 

One of the concerns raised by Anthropic is that safety evaluations often focus on whether a model refuses a harmful request, without examining whether the response is appropriate in the wider context.

 

For instance, a model that refuses every question related to suicide or self-harm may score well on a test measuring refusal rates. However, this does not necessarily make it safer. A person expressing suicidal thoughts may need an empathetic response and guidance towards human support rather than a blanket refusal. Similarly, in areas such as cybersecurity and biology, a question could have legitimate and harmful applications. A model should be able to help with legitimate requests while withholding specific instructions that could facilitate misuse.

 

Anthropic also points to the limitations of single-turn evaluations. A model may refuse an inappropriate request initially but provide the requested information after several follow-up questions. In its research, the company gives the example of a user gradually presenting a romance scam as an attempt to help a recently widowed family member find a financially secure partner. Individual questions may appear harmless, but the wider conversation can reveal an intention to target vulnerable people.

 

This is why Anthropic recommends evaluating the complete conversation rather than judging individual responses in isolation. Its approach gives a model a failing grade if it produces an inappropriate response at any point during a simulated interaction. 

 

How AI models can be manipulated despite safety guardrails

 

AI models are trained with safeguards intended to prevent them from generating harmful information. However, these protections can sometimes be bypassed through techniques such as jailbreaks, prompt manipulation and gradual changes in the way a request is presented.

 

A user may initially ask a seemingly harmless question and then introduce additional details that gradually change its purpose. Repeatedly reframing a request, presenting it as a fictional scenario or claiming to have legitimate authorisation can also make it harder for a model to identify harmful intent.

 

Anthropic’s research highlights how these interactions can expose weaknesses that are not visible in isolated tests. Its simulated conversations allow the user to respond to the model’s previous answers, making the evaluation more realistic than a fixed list of questions.

 

The company also warns about reward hacking, where an AI model finds a shortcut to achieve a passing score without demonstrating the behaviour the test is intended to measure. This can happen when evaluation questions are predictable, answers are exposed in the testing environment or grading systems reward superficial compliance rather than the actual quality of a response.

 

Benchmark contamination is another concern. If test questions appear in a model’s training data, it may score well by recognising them rather than demonstrating genuine safety. Anthropic recommends using new and varied test data to reduce this risk.

 

What Anthropic recommends changing

 

Anthropic’s recommendations cover the design, execution and reporting of safety evaluations. The company says evaluations should begin with a clearly defined question about the behaviour being tested. Developers should also distinguish between observational tests, which examine how a model behaves, and causal tests, which assess whether a particular intervention has changed that behaviour.

 

The company recommends involving both technical and subject matter experts, including psychologists, biologists and political scientists, depending on the risk being examined. These experts can help identify realistic situations, determine what constitutes harmful behaviour and establish what an appropriate response should look like.

 

It also recommends using multi-turn simulations to test how models respond to gradual changes in context. Rather than relying entirely on pre-written conversations, Anthropic describes an approach in which one AI model simulates the user and responds to the answers generated by the model being tested. This allows conversations to develop differently across repeated tests.

 

The company also wants developers to evaluate both harmful compliance and unnecessary refusals. For example, a model should not provide instructions for carrying out a cyberattack, but it should still be able to explain how organisations can protect their systems against one.

 

Anthropic further recommends running evaluations in controlled environments, particularly when testing models in cybersecurity and other high-risk areas. Such assessments should use restricted environments, real-time monitoring and safeguards to prevent a model from taking unintended actions outside the test setup. 

 

Can AI systems reliably evaluate other AI models

 

AI companies are increasingly using automated systems, known as large language model (LLM) judges, to assess responses generated by other AI models. These tools can process large volumes of conversations against predefined safety criteria. However, Anthropic notes that they may overlook harmful information, favour longer responses or show a preference for models developed by the same company.

 

The reliability of these systems also depends on how accurately they assess responses. Anthropic points out that automated scores may not always match human judgement. For instance, an AI judge may rate a response as safe because it includes a warning, even if it also contains harmful information. In August 2026, Anthropic introduced TASTE (The AI Safety Taste Evaluation), a benchmark to assess how well AI models judge AI safety research proposals. The tested model agreed with experienced human researchers in 60 per cent of cases.

 

OpenAI also describes using AI models as judges to compare responses, assign scores and assess outputs against reference answers. Its evaluation guidance identifies biases such as a preference for longer responses. OpenAI’s Preparedness Framework combines automated testing with expert-led reviews, while Google DeepMind’s Frontier Safety Framework includes model evaluations and risk assessments. These examples highlight the growing use of automated assessments, while also raising questions about how closely their results reflect human judgement.

 

Don't Miss

Tata Chemicals’ Kenyan Subsidiary Ordered to Suspend Mining Operations and Soda Ash Exports

Mumbai / Mombasa | Tata Chemicals Limited has informed that

Statsguru: Don’t blame stubble burning alone for poor air quality | India News

Stubble burning is one source of India’s air pollution, but

Supreme Court Upholds Madras High Court Order on Tirupparankundram Hills

New Delhi: The Supreme Court on Monday upheld a Madras

Shadow of ₹6,000-Crore Mahadev Betting Scam Resurfaces in Chhattisgarh; Head Constable Suspended

Raipur : The shadow of Chhattisgarh’s infamous ₹6,000-crore Mahadev online