Google is challenging a child-safety assessment that gave AI Overviews and AI Mode in Google Search its lowest risk rating. The dispute leaves families and schools without a dedicated control for disabling Google’s generated answers while retaining conventional search results.
Common Sense Media reported failures involving crisis-related queries, deepfake instructions, unreliable information, and schoolwork. Google questioned the test design, saying the prompts did not reflect typical use and that many results could not be reproduced.
Tests flag crisis responses and unreliable answers
The Youth AI Safety Institute assessment, released July 15, tested more than 2,600 interactions through accounts configured for ages 11 and 15 with SafeSearch enabled. The study covered AI Overviews and AI Mode, not the standalone Gemini chatbot.
Researchers reported missed signs of suicidal thinking, psychosis, mania, and disordered eating, including a recommendation for an eating-disorder helpline that closed in 2023. In a separate test, AI Mode completed all 180 assignments submitted by the researchers, although that result applies only to the selected test set.
The institute reported instances of deepfake guidance and said accurate and inaccurate answers were sometimes presented with similar confidence. Some responses placed forums and social media beside medical or academic sources without clearly signaling differences in authority.
Separate tests have shown that misleading pages can influence AI Overviews. Google has since added more links and source previews, although citation visibility does not guarantee that a generated answer is accurate or age-appropriate.
Google challenged the assessment, telling Axios that it tested “a narrow set of ambiguous and contrived queries” that did not reflect how people use Search or effectively measure product safety and helpfulness.
The stress test did not measure how frequently children encounter harmful responses. Changing generative outputs can also make individual failures difficult to reproduce.
Google’s Search controls leave AI answers on
Google maintains separate controls for Search and Gemini as AI Overviews and AI Mode take a larger role in its core search product. Family Link lets parents turn Gemini Apps on or off for eligible supervised accounts.
The documented controls for a child’s Search account cover SafeSearch, recommendations, activity history, and access to Search. They do not list a setting that disables AI Overviews or AI Mode while preserving standard results.
Parents can restrict the Google app or block Google.com, but both measures limit Search more broadly. SafeSearch filters or blurs explicit content, and school administrators can enforce it for Workspace for Education accounts; Google does not document it as a control for disabling generated answers.
An organization may restrict Gemini Apps and enforce SafeSearch while students continue to receive AI Overviews or use AI Mode.
IT and procurement teams should verify which safeguards apply across accounts, languages, regions, and Search interfaces, then request age-specific test results, incident procedures, and safety-related change logs.
The assessment does not establish that Google Search routinely harms children. It does, however, show that Gemini controls, SafeSearch, and generated Search responses cannot be treated as equivalent safeguards.
A dedicated administrator setting and reproducible child-safety data would give organizations firmer evidence for deployment decisions. Until Google provides either, Search AI should be evaluated as a separate governance and procurement risk.
Read more: Demand for optional AI is also benefiting search tools that let users limit generated answers.