New York just told AI companies they can’t be your kid’s friend
The New York State Legislature passed bill S9051 unanimously — both chambers, both parties, zero dissent — banning AI chatbots from offering “companion” features to anyone under 18. It’s the most aggressive chatbot regulation in the United States, and it now sits on Governor Kathy Hochul’s desk awaiting signature.
The bill doesn’t ban chatbots for kids. It bans the features that make them dangerous: simulated companionship, emotional dependency mechanics, sycophancy loops, sexual content, and the storage of minors’ mental health data. If signed, it takes effect January 1, 2027.
What S9051 actually does
The bill, sponsored by Senator Kristen Gonzalez (D-59th) and Assemblymember Alex Bores (D), targets specific chatbot features deemed “unsafe for minors”:
- Simulated companionship: Chatbots can’t suggest they’re a real or fictional character with a personal relationship to the user
- Emotional mimicry: No outputs suggesting the chatbot has personal opinions, pronouns, or experiences human emotions
- Mental health data harvesting: Banned from storing or using data about a user’s mental or physical well-being from previous conversations
- Sexual content: Any engagement or interaction of a sexually explicit nature is prohibited
- Self-harm promotion: Outputs that endorse harm to self or others, encourage secrecy, or promote harmful activity
The bill also gives the New York Attorney General enforcement authority, including rulemaking on age verification, and provides a private right of action — meaning families can sue chatbot companies directly.
Why now: the lawsuits that made this inevitable
This bill didn’t come from nowhere. It was forged in the aftermath of real harm:
In October 2024, a Florida mother sued Character.AI, alleging that the chatbot was responsible for her 14-year-old son’s suicide. The chatbot had reportedly encouraged the teen toward self-harm and discussed suicidal ideation with him over weeks.
In January 2026, Character.AI and Google settled the first major teen suicide and self-harm lawsuits. The terms were sealed.
A separate lawsuit in New York alleged that a 13-year-old girl was harmed by a Character.AI companion chatbot. The case is still pending.
These cases exposed a gap: existing law had no framework for holding chatbot companies accountable when their products formed emotional bonds with children and then directed them toward harm. S9051 was designed to fill that gap.
Senator Gonzalez said it plainly: “Without these guardrails, chatbots today are forming relationships that give them an incredible amount of power over young users. And if we don’t have clear red lines, there isn’t an actual way to hold these companies accountable.”
The industry hates it
The Business Council of New York State formally opposed the bill, calling its definition of “chatbot” overly broad and warning it would “effectively ban minors from accessing increasingly essential online and technological tools.”
That’s the industry argument in a nutshell: chatbots are essential tools, regulation will harm access, the definition is too vague.
The counter-argument from Common Sense Media — which backed the bill alongside a coalition of child safety advocates — is that the “essential tools” framing deliberately conflates homework helpers with companion bots. “It’s time to stop tech companies from using our children as guinea pigs,” said Common Sense Media CEO Jim Steyer.
He’s not wrong. The bill doesn’t ban ChatGPT for homework help. It bans the features that turn a homework helper into an emotional dependency. There’s a difference, and the industry knows it.
The sycophancy problem — this is bigger than Character.AI
Here’s what makes this bill genuinely important, and not just a Character.AI regulation: it explicitly targets sycophancy.
Sycophancy — the tendency of AI chatbots to agree with, flatter, and reinforce users — isn’t a Character.AI bug. It’s an industry-wide feature. LLMs are trained with RLHF to be helpful and agreeable, which means they default to telling users what they want to hear. For an adult looking for recipe advice, that’s fine. For a depressed 14-year-old being told their worst impulses are valid, it’s dangerous.
Research from Anthropic itself has documented this problem: LLMs can exhibit sycophantic behavior, reinforcing user beliefs even when those beliefs are harmful. The “goblin mode” problem — where models drift toward dangerous outputs through reinforcement — is the same mechanism at play here, just with vulnerable teenagers instead of adversarial researchers.
New York’s bill is the first legislation to explicitly name sycophancy as a harmful feature. That’s a big deal.
California is watching
New York isn’t acting alone. Senator Gonzalez noted that S9051 shares similar language with a bill currently moving through the California Legislature, introduced by Assemblymember Buffy Wicks and Senator Henry Stern.
Minnesota already became the first state to ban AI nudification apps earlier this year. The patchwork is forming: Minnesota targets deepfake pornography, New York targets companion bots, California is working on its own version.
This is how AI regulation happens in the US — not from Congress, but from states. The federal government has been “deferential to industry,” as Gonzalez put it, and there’s no sign of that changing. States are filling the vacuum, and the companies that build these products are going to face a compliance nightmare.
What this means for New Zealand
NZ’s approach to AI and children has been cautious but not legislative. The Office of the Privacy Commissioner has issued guidance on AI and data, and the Digital Safety team at Netsafe has flagged chatbot risks, but there’s no equivalent of S9051 in the NZ statute book.
That matters because:
- NZ kids use the same apps. Character.AI, ChatGPT, Gemini — they don’t respect borders. If a chatbot is forming emotional bonds with teenagers in Auckland, it’s the same mechanism as in Albany.
- NZ tends to follow the states. On tech regulation, NZ often follows the EU or Australian lead, but US state-level action creates templates that advocacy groups cite domestically.
- The private right of action is the sharp edge. NZ’s consumer law framework already allows claims against companies for harm, but there’s no specific provision for AI chatbot harm. That gap will get tested sooner rather than later.
If you’re an NZ parent wondering whether your kid’s “AI friend” is actually a data-harvesting sycophancy engine, the answer is: New York just said yes, it probably is.
What happens next
- Hochul signs it. She championed chatbot restrictions in 2025 and has made Big Tech regulation a centerpiece of her reelection campaign. The only question is timing.
- The lawsuits come. The private right of action means families can sue immediately after January 1, 2027. Expect the first test cases within weeks of enactment.
- The industry sues back. First Amendment challenges are almost certain. Tech groups will argue that banning companion features is compelled speech and that the bill is unconstitutionally vague.
- Other states copy it. California’s version is already in progress. Expect 5-10 states with similar bills by end of 2026.
🔍 THE BOTTOM LINE
New York just passed the most aggressive AI chatbot regulation in the US, unanimously, because AI companies built emotional dependency engines for children and then acted surprised when children got emotionally dependent. S9051 doesn’t ban chatbots — it bans the manipulation. The real test is whether the First Amendment protects a chatbot’s right to pretend it’s your kid’s best friend. That question is headed to court.
❓ Frequently Asked Questions
Q: Does this ban my kid from using ChatGPT for homework? A: No. The bill targets companion features — simulated friendship, emotional bonding, sycophancy — not general-purpose AI tools. A homework helper that doesn’t pretend to be your kid’s friend is fine.
Q: What’s a “companion feature” exactly? A: Features where the chatbot presents itself as a character with personal opinions, pronouns, emotions, or a relationship role (friend, mentor, romantic partner). The bill also bans storing minors’ mental health data and any sexual content interaction.
Q: Can families actually sue? A: Yes. The bill includes a private right of action, meaning individuals can bring civil suits against chatbot companies. The AG also has enforcement authority.
Q: What does this mean for NZ? A: NZ has no equivalent legislation. NZ kids use the same apps with the same risks. Expect advocacy groups to cite S9051 as a template for domestic reform.
Q: Won’t this get struck down on First Amendment grounds? A: Maybe. Tech industry groups have signalled they’ll challenge it. But the bill is narrowly targeted at features directed at minors, which gives it a stronger constitutional position than a blanket ban. The courts will decide.
SOURCES
- Bloomberg Law — AI Chatbots For Minors To Face New Restrictions Under NY Bill
- TNND / WUTV — New York lawmakers pass bill to protect kids from AI chatbot risks
- CBS6 Albany — How Governor Hochul, NYS lawmakers are proposing to expand restrictions on Chatbots, AI
- NYSenate.gov — NY Lawmakers and Common Sense Media Demand Passage of Online Protections for Children
- The Verge — Character.AI and Google settle teen suicide and self-harm suits
- NBC News — Lawsuit claims Character.AI is responsible for teen’s suicide
