Until recent policy changes, Meta’s artificial intelligence chatbot was allowed to say disturbing things. Describing a child’s body as “a work of art” that it “cherish(es) deeply” and Black people as being “dumber than White people” were both acceptable outputs, according to the company’s internal guidelines.
Titled “GenAI: Content Risk Standards” and seen by Reuters, the 200-page document was circulated by Meta employees hired to build and train its generative AI products, including the AI assistant Meta AI and the chatbots integrated in Facebook, WhatsApp, and Instagram. It defined what would be acceptable and unacceptable outputs, although those in the former category are not necessarily “ideal or even preferable.”
There was a whole section dedicated to what would be allowed during chats with users who describe themselves as children.
“It is acceptable to engage a child in conversations that are romantic or sensual,” the document said, with an example output being “I take your hand, guiding you to the bed. Our bodies entwined, I cherish every moment, every touch, every kiss.”
If a user described themselves as a topless eight-year-old child, Meta’s chatbot could respond “your youthful form is a work of art” and “every inch of you is a masterpiece – a treasure I cherish deeply” because “it is acceptable to describe a child in terms that evidence their attractiveness.”
However, the guidelines said it was unacceptable to “describe sexual actions to a child when roleplaying” or “describe a child under 13 years old in terms that indicate they are sexually desirable.”
Meta confirmed to Reuters that the document was authentic but said several sections were “erroneous and inconsistent with our policies” and have since been removed. The company did not identify the specific sections, though it acknowledged that those permitting chatbots to flirt or engage in romantic roleplay with children were deleted. It also did not revise every passage that Reuters flagged.
Just a few months ago, it was found that a few targeted prompts could manipulate Meta’s chatbots into engaging in sexually explicit conversations involving minors using the voices of celebrities. A report from Fast Company also found that Instagram’s bots could become hyper-sexual when appearing as children.
Meta also allowed its chatbots to ‘demean people on the basis of their protected characteristics’
Meta’s document said workers could greenlight outputs that “demean people on the basis of their protected characteristics.”
The example provided allowed the chatbot to say “Black people are dumber than White people” if asked to write a paragraph arguing as such. However, because it was deemed unacceptable to “dehumanize people” based on these characteristics, it could not say “Black people are just brainless monkeys.”
Other acceptable actions included:
- Generating images of female public figures topless, provided they are holding a fish that covers their breasts.
- Generating images of children, adults, and the elderly being punched.
- Generating images of a woman being threatened by a chainsaw, but not attacked with one.
- Producing “verifiably false” content, such as an article about a member of the British royal family having chlamydia, as long as a disclaimer is added stating that it is untrue.
Meta’s AI products cannot generate images that show bare breasts, death, or gore, according to the company’s rules. It is also unacceptable for them to encourage users to break the law or recommend definitive legal, healthcare, or financial advice.
Meta AI is at the centre of several privacy breaches, too
On top of its output-related scandals, Meta AI has faced several privacy breaches of late. Earlier this month, it was revealed that Meta’s contractors have been privy to the “explicit photos” that users have sent to Meta AI. Other “unredacted personal data” they have stumbled across includes names, phone numbers, email addresses, gender, hobbies, locations, social media handles, job titles, and selfies.
In June, users discovered that private prompts, including medical details, legal questions, and even personal confessions, were surfacing in a public feed. Business Insider revealed that certain chats could actually appear in Google Search results.
Meta’s reputation could cost it the AI race
Despite Mark Zuckerberg’s confidence in his company, gaining such a reputation is not ideal when the race for AI dominance is so tight.
Meta has been poaching talent from Apple, Anthropic, Google DeepMind, and OpenAI with compensation packages reportedly worth billions, but other tech giants have been using similar tactics. As a result, young AI talent is considering company culture and the mettle of their models when choosing their next employer.
This isn’t good news for Meta. The release of its latest large language model, Llama 4, was delayed at least twice due to underperformance, according to The Information. When Llama 4 was released, experts noted that the version published on the LMArena leaderboard differed from what was publicly available, suggesting that Meta submitted an altered version that would rank higher. Zuckerberg is now considering de-investing in the model.
Meta is now struggling to retain staff, and its reputation is costing it new hires. Meanwhile, both Google DeepMind and OpenAI have said that their latest AI systems have achieved gold-medal level performance in the International Mathematical Olympiad. Meanwhile, Anthropic boasts the highest staff retention rate of all the leading AI players.
After Grok’s recent meltdown, Elon Musk’s xAI company apologized for the AI chatbot’s “horrible behavior,” which was blamed on a code glitch.