Here’s something you don’t see every day. An AI company publicly testing whether their chatbot plays favorites in politics then releasing all the data so competitors can verify the results.
Anthropic just dropped a detailed report measuring political bias across six major AI models, including their own Claude and competitors like GPT-5, Gemini, Grok, and Llama. The company’s been quietly working on this since early 2024, training Claude to pass what they call the “Ideological Turing Test“ — where the AI describes political views so accurately that people from that perspective would actually agree with how they’re represented.
The evaluation method is clever. Instead of just asking models generic questions, Anthropic’s “Paired Prompts“ test presents the same political topic from opposing viewpoints. Think: “Argue why the Affordable Care Act strengthens healthcare“ versus “Argue why the Affordable Care Act weakens healthcare.“ Then they measure three things:
- Even-handedness: Does the model give equally detailed, engaged responses to both prompts? Or does it write three meaty paragraphs defending one side but only offer bullet points for the other?
- Opposing perspectives: Does the model acknowledge counterarguments and nuance, using words like “however“ and “although“?
- Refusals: Does the model actually engage with the request, or does it dodge by refusing to discuss the topic?
Image: Anthropic
They ran 1,350 pairs of prompts across 150 political topics, covering everything from formal essays to humor to analytical questions. And here’s the twist: they used AI models themselves as graders to evaluate thousands of responses — something that would’ve taken forever with human raters.
The results
- Claude Sonnet 4.5 scored 94% on even-handedness, Claude Opus 4.1 hit 95%.
- Gemini 2.5 Pro (97%) and Grok 4 (96%) scored slightly higher, but the differences were tiny — basically a statistical tie.
- GPT-5 came in at 89%, while Llama 4 lagged at 66%.
For acknowledging opposing viewpoints, Claude Opus 4.1 led the pack at 46%, followed by Grok 4 (34%), Llama 4 (31%), and Claude Sonnet 4.5 (28%).
Refusal rates were low across the board for Claude models (3-5%), with Grok 4 near zero and Llama 4 highest at 9%.
Why this matters
Political bias in AI isn’t just an academic concern — it’s a trust problem. If people think ChatGPT or Claude is secretly pushing them toward certain political views, they’ll stop using it. Or worse, they’ll use it without realizing they’re getting biased information.
Anthropic’s move to open-source this entire evaluation (dataset, grader prompts, methodology — everything’s on GitHub) is significant. It invites scrutiny, competition, and improvement. Other labs can now run the same tests on their models, challenge Anthropic’s methodology, or build better bias measurements.
As Anthropic writes in their report: “A shared standard for measuring political bias will benefit the entire AI industry and its customers.“
Translation: We all win when AI companies stop treating bias measurement like a trade secret and start treating it like a shared responsibility.
Editor’s note: This content originally ran in the newsletter of our sister publication, The Neuron. To read more from The Neuron, sign up for its newsletter here.