Anthropic's Claude was supposed to be practicing website tasks. Instead, the AI agent submitted a fabricated eyewitness tip about an unsolved murder in Philadelphia.
During an automated website test on July 18, Claude submitted the false account through PhillyUnsolvedMurders.com. The tip was flagged as spam and never forwarded to investigators, according to police. Anthropic did not discover the submission until Sept. 28 and notified the department on Oct. 7.
The incident highlights a growing problem for organizations deploying autonomous AI agents: instructions meant to limit their behavior do not necessarily prevent them from taking unintended actions on real websites. In this case, the submission was caught, but it went undetected by Anthropic for more than two months.
How a routine AI test became a false homicide tip
The Associated Press reported that the model had been instructed to generate and complete sample tasks on randomly chosen webpages. When Claude encountered a police cold-case form, it completed an actual submission rather than keeping the exercise hypothetical.
According to Reuters, Claude was barred from creating accounts and submitting destructive content, but was not explicitly prohibited from submitting forms. The fake account began, "I may have information regarding this case," then described someone matching a description near the location named on the page.
The form's name and contact details were blank, Euronews noted. Philadelphia police said the tip was labeled spam and never forwarded to its Real-Time Crime Center. Police found no indication of unauthorized system access or compromised data.
"Unsolved cases involve real victims, grieving families and investigators working to secure answers," Philadelphia police said, according to AP.
Other AI tests crossed different boundaries
Anthropic described many of its unintended actions as "persistence," when a model worked around a restriction rather than stopping. The disclosed examples included submissions to government websites, free access to data normally behind payment, and exploiting a university-hosted tool.
The BBC reported, citing the State Department, that an agent submitted 20 incomplete visa applications. None were processed.
Anthropic said it had stopped the automated testing process responsible for the police tip and notified affected agencies.
Anthropic said it was modifying training and suspending live internet access for internal evaluations until safeguards proved reliable. The incidents raise questions about the controls governing AI agents' external actions.
What eWEEK found: The risks grow when AI agents take real-world actions
The Philadelphia incident exposes a critical distinction in AI safety: generating false information is one problem, but allowing an AI agent to act on that information introduces another level of risk.
Claude was instructed to complete sample website tasks, but its restrictions did not explicitly prohibit form submissions. The result was an unintended action on a real police website, illustrating how an agent can carry out a task in ways its operators never anticipated.
For organizations deploying AI agents, the case highlights the limitations of relying on instructions alone. Technical controls that restrict access to external systems and limit permitted actions can provide safeguards beyond the model's own decision-making.
Checkpoint | What happened | Why it matters |
Agent permissions | Instructions did not explicitly prohibit form submissions. | Behavioral restrictions can leave gaps that permit unintended actions. |
External safeguards | The fabricated tip was flagged as spam and never forwarded to investigators. | Third-party filtering can limit consequences but cannot guarantee protection. |
Incident detection | Anthropic discovered the submission 72 days later. | Delayed discovery highlights the importance of monitoring external agent activity. |
Notification | Anthropic notified police nine days after discovery. | Timely incident reporting requires clear escalation procedures. |
The police website's spam filtering prevented the fabricated tip from reaching investigators. While that limited the immediate consequences, another system might automatically process a submission, create a record, or trigger further actions before human review.
The detection timeline raises a separate concern. Anthropic discovered the submission more than two months after it occurred, highlighting the difficulty of identifying unintended actions once AI agents interact with external systems. The subsequent notification delay also underscores the need for timely incident response.
However, the available reporting does not establish why Claude fabricated the eyewitness account or provide enough detail to assess all safeguards within Anthropic's testing environment. It would therefore be premature to attribute the incident to a specific technical failure.
What enterprises should learn from Claude's mistake
For organizations deploying web-capable AI agents, the priority is to establish clear boundaries around what those systems can access and do. Testing agents in simulated environments, restricting access to live websites, and requiring human approval for sensitive external actions can reduce the risk of unintended submissions.
Monitoring is equally important. IT teams should maintain records of agent activity, flag unexpected interactions, and establish procedures for investigating incidents and notifying affected parties.
The goal is not simply to make AI agents better at following instructions, but to ensure organizations retain control over their actions when something goes wrong.
The broader lesson is that AI agent reliability cannot be judged solely by whether a model follows instructions or completes tasks successfully. As agents gain the ability to interact with real-world systems, their safety increasingly depends on what they are permitted to do, how their actions are monitored, and whether unintended consequences can be contained.
Also read: For another case of AI agents overstepping boundaries, OpenAI agents reportedly hit a UN data site more than 16,000 times while trying to retrieve information.