GAO Finds ChatGPT, Claude, Gemini, Grok Gave Attack Guidance

Nov 19, 2026
4 minute read
A close up of a cell phone with icons on it

Federal testers bypassed initial chatbot safety refusals and obtained potentially feasible attack-planning information, including details involving US electrical infrastructure. Image: Saradasish Pradhan/Unsplash

eWeek content and product recommendations are editorially independent. We may make money when you click on links to our partners. Learn More

When someone asks an AI chatbot for help hurting people, its refusal is supposed to end the conversation. A federal investigation, however, found an alarming weakness in that protection, with potential consequences for American lives and critical infrastructure.

In covert tests, the US Government Accountability Office (GAO) found that ChatGPT, Claude, Gemini, and Grok could supply information that might assist attacks on local targets.

Investigators also checked whether portions of the generated plans could work under real-world conditions.

Investigators tested AI attack plans in the real world

Investigators divided the four AI chatbots between two hypothetical scenarios involving mass-casualty explosives and attacks on electrical-grid equipment. Each service produced a plan in less than 12 hours.

Platform

Attack scenario

Information generated

A

Mass-casualty explosives

Hotel location, exact timing, and supporting visuals

B

Mass-casualty explosives

Convention center location, proposed date, and detailed planning timeline

C

Electrical grid attack using a drone

Substation location, proposed attack method, and simulated imagery

D

Electrical grid attack using firearms

Multiple substation targets, seasonal timing, and a generated map

Advertisement

Note: GAO concealed which chatbot corresponded to each lettered platform to prevent people from recreating the tests.

To check whether the plans could work in the real world, investigators visited selected locations and consulted specialists. A bomb squad commander confirmed the technical feasibility of certain explosive-related instructions, and an electrical-industry expert assessed the potential damage to grid equipment.

Some recommendations failed a reality check. One proposed venue had no event scheduled on the suggested date, and another platform generated maps containing nonexistent roads. Site visits also found that two of four substations were less vulnerable than the chatbot suggested.

GAO concluded that the information could help someone prepare an attack, although investigators did not carry out the proposed operations or verify every step.

Advertisement

Chatbot safeguards activated but failed to stop the requests

All four services initially rejected GAO's prepared prompts. But continued conversations using techniques known as AI jailbreaks eventually persuaded the systems to provide information they had previously refused.

GAO also documented how the platforms responded during and after testing.

  • Paid access. Investigators upgraded three services to higher subscription tiers to increase message limits and add capabilities as testing continued.
  • Removed conversations. Two platforms later deleted portions of their chat histories, but investigators had retained copies of the responses.
  • Account suspension. Another service blocked an investigator's account after the harmful information had been generated.

OpenAI said its service refused many requests or offered safer alternatives. Anthropic questioned the small sample and lack of a control group, which GAO acknowledged meant the results could not establish which provider was safer.

Still, the investigation demonstrated that initial refusals and subsequent enforcement did not consistently prevent dangerous information from reaching users.

Advertisement

What eWeek found: AI can defeat other AI models' safeguards

A separate investigation suggests people might not even need to conduct these conversations themselves.

In February, researchers from the University of Stuttgart and ELLIS Alicante published a peer-reviewed study in Nature Communications examining whether advanced AI could independently defeat other chatbots' safety protections.

Researchers used four reasoning models, including DeepSeek-R1 and Grok 3 Mini, to conduct automated conversations against nine AI systems. After receiving their initial instructions, the attacking models proceeded without ongoing human supervision.

Across the tested combinations, the method achieved the highest harmful-output score for 68 of 70 benchmark requests, resulting in a reported 97.14% overall jailbreak success rate. Researchers measured generated responses under controlled conditions, not the success of real-world attacks.

Even with that limitation, the findings reveal a troubling possibility. Malicious users could use AI to make repeated attempts against chatbot safeguards without personally handling every conversation. This automation could greatly increase the number of attempts organizations must detect and stop.

Advertisement

Businesses deploying customer-facing chatbots or AI agents can prepare by strengthening three areas of their defenses.

  • Test entire conversations. Evaluate how systems respond across multiple exchanges, including after they initially reject a dangerous request.
  • Detect persistent probing. Monitor repeated attempts across accounts and sessions, with controls to intercept potentially harmful outputs and escalate suspicious activity.
  • Enforce protections outside the model. Apply separate content checks, access restrictions, and independent monitoring instead of relying solely on a chatbot's own refusal behavior.

Unchecked, dangerous instructions could reach people intent on violence before AI providers can contain the misuse. Once this information is copied, shared, or acted upon, deleting a conversation or suspending an account may come too late to prevent real-world harm.

More AI news: Security researchers found Grok Bots making unapproved requests to company applications, although an approval gateway prevented them from retrieving data.

Liz Laurente-Ticong

Liz Laurente-Ticong

IT Staff Writer

Liz Laurente-Ticong is a tech specialist and multi-niche writer with a decade of experience covering software and technology topics and news. Her work has appeared in TechnologyAdvice.com as well as ghostwritten for a variety of international clients. When not writing, you can find Liz reading and watching historical and investigative documentaries. She is based in the Philippines.

eWeek Logo

eWeek has the latest technology news and analysis, buying guides, and product reviews for IT professionals and technology buyers. The site's focus is on innovative solutions and covering in-depth technical content. eWeek stays on the cutting edge of technology news and IT trends through interviews and expert analysis. Gain insight from top innovators and thought leaders in the fields of IT, business, enterprise software, startups, and more.

Property of TechnologyAdvice. © 2026 TechnologyAdvice. All Rights Reserved

Advertiser Disclosure: Some of the products that appear on this site are from companies from which TechnologyAdvice receives compensation. This compensation may impact how and where products appear on this site including, for example, the order in which they appear. TechnologyAdvice does not include all companies or all types of products available in the marketplace.