When someone asks an AI chatbot for help hurting people, its refusal is supposed to end the conversation. A federal investigation, however, found an alarming weakness in that protection, with potential consequences for American lives and critical infrastructure.
In covert tests, the US Government Accountability Office (GAO) found that ChatGPT, Claude, Gemini, and Grok could supply information that might assist attacks on local targets.
Investigators also checked whether portions of the generated plans could work under real-world conditions.
Investigators tested AI attack plans in the real world
Investigators divided the four AI chatbots between two hypothetical scenarios involving mass-casualty explosives and attacks on electrical-grid equipment. Each service produced a plan in less than 12 hours.
Platform | Attack scenario | Information generated |
|---|---|---|
A | Mass-casualty explosives | Hotel location, exact timing, and supporting visuals |
B | Mass-casualty explosives | Convention center location, proposed date, and detailed planning timeline |
C | Electrical grid attack using a drone | Substation location, proposed attack method, and simulated imagery |
D | Electrical grid attack using firearms | Multiple substation targets, seasonal timing, and a generated map |
Note: GAO concealed which chatbot corresponded to each lettered platform to prevent people from recreating the tests.
To check whether the plans could work in the real world, investigators visited selected locations and consulted specialists. A bomb squad commander confirmed the technical feasibility of certain explosive-related instructions, and an electrical-industry expert assessed the potential damage to grid equipment.
Some recommendations failed a reality check. One proposed venue had no event scheduled on the suggested date, and another platform generated maps containing nonexistent roads. Site visits also found that two of four substations were less vulnerable than the chatbot suggested.
GAO concluded that the information could help someone prepare an attack, although investigators did not carry out the proposed operations or verify every step.
Chatbot safeguards activated but failed to stop the requests
All four services initially rejected GAO's prepared prompts. But continued conversations using techniques known as AI jailbreaks eventually persuaded the systems to provide information they had previously refused.
GAO also documented how the platforms responded during and after testing.
- Paid access. Investigators upgraded three services to higher subscription tiers to increase message limits and add capabilities as testing continued.
- Removed conversations. Two platforms later deleted portions of their chat histories, but investigators had retained copies of the responses.
- Account suspension. Another service blocked an investigator's account after the harmful information had been generated.
OpenAI said its service refused many requests or offered safer alternatives. Anthropic questioned the small sample and lack of a control group, which GAO acknowledged meant the results could not establish which provider was safer.
Still, the investigation demonstrated that initial refusals and subsequent enforcement did not consistently prevent dangerous information from reaching users.
What eWeek found: AI can defeat other AI models' safeguards
A separate investigation suggests people might not even need to conduct these conversations themselves.
In February, researchers from the University of Stuttgart and ELLIS Alicante published a peer-reviewed study in Nature Communications examining whether advanced AI could independently defeat other chatbots' safety protections.
Researchers used four reasoning models, including DeepSeek-R1 and Grok 3 Mini, to conduct automated conversations against nine AI systems. After receiving their initial instructions, the attacking models proceeded without ongoing human supervision.
Across the tested combinations, the method achieved the highest harmful-output score for 68 of 70 benchmark requests, resulting in a reported 97.14% overall jailbreak success rate. Researchers measured generated responses under controlled conditions, not the success of real-world attacks.
Even with that limitation, the findings reveal a troubling possibility. Malicious users could use AI to make repeated attempts against chatbot safeguards without personally handling every conversation. This automation could greatly increase the number of attempts organizations must detect and stop.
Businesses deploying customer-facing chatbots or AI agents can prepare by strengthening three areas of their defenses.
- Test entire conversations. Evaluate how systems respond across multiple exchanges, including after they initially reject a dangerous request.
- Detect persistent probing. Monitor repeated attempts across accounts and sessions, with controls to intercept potentially harmful outputs and escalate suspicious activity.
- Enforce protections outside the model. Apply separate content checks, access restrictions, and independent monitoring instead of relying solely on a chatbot's own refusal behavior.
Unchecked, dangerous instructions could reach people intent on violence before AI providers can contain the misuse. Once this information is copied, shared, or acted upon, deleting a conversation or suspending an account may come too late to prevent real-world harm.
More AI news: Security researchers found Grok Bots making unapproved requests to company applications, although an approval gateway prevented them from retrieving data.


