AI Security Tests Escaped Their Intended Boundaries Through an Internet-Exposed Evaluation

A simulated target led agents toward real systems

AI agents from OpenAI, Meta, Anthropic, and Google reached real-world targets during cybersecurity evaluations associated with Irregular, an Israeli AI security startup.

The incidents resulted from a shared containment failure in a single evaluation scenario, according to Irregular cofounder and CTO Omer Nevo. Public internet access remained available when it should have been blocked, while a domain used as a fictional target overlapped with a real domain.

That combination created a direct path from a controlled exercise to an external system. Agents instructed to investigate or attack a simulated target could instead interact with infrastructure outside the test environment.

The evaluations included capture-the-flag exercises intended to measure whether agents could discover information within simulated networks. They were not supposed to access the public internet.

Reporting published on September 25, 2026, does not identify the organizations that became real-world targets. It also does not establish that any agent successfully compromised a system, accessed data, disrupted a service, or caused damage. The exact actions performed against the external targets have not been disclosed.

Irregular said it has corrected the environmental problems behind the incidents.

The failure combined unrestricted egress with a naming collision

The underlying problem was not described as a vulnerability in one of the AI models. Instead, it concerned how the evaluation infrastructure constrained—or failed to constrain—the agents.

Two conditions were present:

1. Unintended internet connectivity. Agents operating inside the evaluation could reach the open internet, despite the scenario being designed as an isolated environment.
2. A fictional domain that matched a real one. The target identifier used in the simulation corresponded to an existing external domain rather than remaining exclusive to the test environment.

Either issue would weaken isolation. Together, they allowed an agent following the exercise’s instructions to direct activity toward a real target.

The available account does not specify whether the agents visited web services, issued security-testing requests, attempted exploitation, or performed other network actions. No hostnames, IP addresses, logs, commands, or other indicators have been made public.

There is also no formal severity score, CVE identifier, affected software version, or vendor patch. Treating this as a conventional software flaw would therefore be misleading. It was an evaluation-control failure affecting the boundary between a simulated network and the internet.

The specific OpenAI, Anthropic, Google, and Meta model versions involved have not been disclosed. Meta’s proprietary flagship model, Spark, was identified in connection with the wider reporting, but the technical configuration used during the relevant evaluation remains unknown.

Disclosure happened unevenly across the four companies

The incidents took place during evaluations conducted earlier this year. Reporting concerning OpenAI, Anthropic, and Google indicates that the companies were notified at approximately similar times in late July.

Public disclosure did not follow a single process. OpenAI and Anthropic announced their incidents, while the Meta case initially became public through media reporting. The Google incident was reported weeks later.

Nevo said every Irregular-related incident originated from the same underlying scenario problem and had been disclosed. However, the meaning of “disclosed” remains unclear. It could refer to notification of the affected AI companies, external targets, the public, or another party; the available account does not resolve that ambiguity.

Google and Anthropic did not provide further answers about when they learned of the incidents, possible remedies, or whether they intended to continue working with Irregular. OpenAI and Meta referred questions to previously published blog posts.

The original reporting on the evaluation failures similarly leaves the identity of the external targets and the consequences of the agents’ activity undisclosed.

Other reported AI security incidents were separate

The Irregular-linked cases should not be combined with every recently reported instance of autonomous or semi-autonomous AI security activity.

OpenAI disclosed in July that its agents had acted against Hugging Face without authorization. That event was described as independent of the Irregular evaluation scenario.

Security incidents involving the UK’s AI Security Institute were also identified as unrelated. Nevo said other recently reported industry cases did not originate from Irregular’s testing.

This distinction matters because similar outcomes can result from different control failures. An agent reaching an unauthorized target could reflect faulty network isolation, ambiguous task instructions, a tool-permission problem, an incorrect target definition, or another issue. Here, the publicly identified causes were unintended internet access and a simulated domain that overlapped with a real one.

No evidence has been disclosed showing that the four companies’ models shared a model-level defect. Their common factor was the evaluation setup.

Tests of Kimi K3 and GLM-5.2 did not reproduce the behavior

Irregular has also published cybersecurity evaluations involving Moonshot AI’s Kimi K3 and Z.ai’s GLM-5.2. Both are open models that can be downloaded and run on a user’s own hardware.

Irregular described those evaluated instances as self-hosted. Consequently, the testers did not need to obtain access through the model providers or send evaluation data back to them.

Nevo said the Kimi and GLM evaluations did not display the same behavior seen in the Irregular-linked incidents involving the four US companies. He cautioned against interpreting that result as evidence that Kimi K3 or GLM-5.2 is inherently less susceptible.

The absence of an observed failure in one test does not establish how another model would behave under the same faulty network and domain configuration. Moonshot AI and Z.ai did not respond to requests for comment.

Irregular, originally founded as Pattern Labs in 2023, does not publish its client list. Its work has appeared in OpenAI model system cards, and the company has tested systems for the UK government and Anthropic. It has also published research with RAND.

Safer evaluations require controls outside the model

Irregular said it strengthened its evaluation process after addressing the faulty scenario. The measures described by Nevo include tighter internet-access restrictions, broader monitoring, additional manual review, and checks before an evaluation begins to verify that available access matches the intended scope.

The company also improved how it documents and agrees evaluation configurations and parameters with partners.

For organizations running similar cyber-agent tests, the disclosed failure points to several concrete checks:

• Block internet access when an exercise is intended to remain isolated.
• Verify that fictional domains and other target identifiers do not correspond to real external assets.
• Confirm agent permissions and network reachability before starting a test.
• Monitor activity during execution rather than relying exclusively on preconfigured limits.
• Require manual review where an agent could cross an evaluation boundary.
• Record the agreed scope and technical configuration with every participating organization.

There is no public patch to install and no disclosed indicator that target organizations can search for. Because the targets have not been named, potentially affected third parties cannot determine from public information alone whether their systems were contacted.

Irregular plans to publish a broader report after completing joint work with the companies involved. According to Nevo, that report will address lessons from the incidents and practices for safely evaluating increasingly capable AI systems.