mythsthe wider recordNYC MyCity chatbot2024government / public sector · **year:** 2024

everyone assumed:

an official AI assistant gives lawful answers

what was actually true:

An official wrapper transfers authority to the answer without transferring accuracy to it.

how it happened

  1. whobusiness ownersasked the city's bot
  2. the gapofficial chatbotconfident wrong answers
  3. the resultadvice to break the lawreported March 2024
the gap that let it happen the result

what everyone assumed

When a chatbot lives on a government site, points at official source pages, and speaks in plain declarative sentences, the reasonable belief is that its answers are at least lawful. The voice carries the authority of the institution behind it. A small business owner reading “yes, you can do that” from the city’s own tool has every reason to act on it, which is exactly what the tool was built to let them do.

what happened

The bot was fluent, and fluency reads as accuracy. It answered without hedging on housing rules, worker rights, and consumer protections, and the official wrapper gave no signal about which answers were sound and which were invented. A small print line noted it “may occasionally produce incorrect, harmful or biased content,” but there was no way for an average user to tell, in any given answer, whether they were reading the law or reading a confident guess. For five months it ran that way, taking real questions from real owners and returning a mix nobody had separated.

step by step

  1. the city launches a business chatbotFluent answers under an official wrapper.
  2. no way to tell law from guessFive months of real questions, a mix nobody separated.the assumption breaks here
  3. 29 March 2024The Markup tests itWrong on Section 8 vouchers, tips and cash-free shops.
  4. disclaimers added, beta later endedThe answers came in the city's own voice.

how it surfaced

On March 29, 2024, The Markup published its testing. Asked whether landlords must accept Section 8 vouchers, the bot said no, when source-of-income discrimination is illegal in New York City. It said bosses could take a cut of workers’ tips. It said a restaurant could go cash-free, when a 2020 city law requires businesses to accept cash. The answers were not lawful, and they were delivered in the city’s own voice; the city kept the bot online, adding disclaimers, and the beta was later ended.

An official wrapper transfers authority to the answer without transferring accuracy to it.

A language model on a government domain still generates text, and a confident sentence about the law is only as lawful as whatever produced it.

picture it at a company like yours

The company here is made up. The other two paths are not: they happened, and each links to the story.

your company
a normal dayYou launch an AI assistant that answers customers' questions about rules and policies.
the gap nobody seesIt answers confidently even when it is wrong, and nobody checks a sample.AI output nobody checks
  1. in this story
    the day it mattersA journalist tests it and finds it giving advice that breaks the law.
    what it costs youYour organisation is the one quoted as giving that advice.
  2. or it goes like this
    howA customer-facing bot promises something your policy does not offer.
    what leavesmoney you now owewho ends up with itthe customer, backed by a tribunal
    it happenedMoffatt v. Air Canada, 2024
  3. or it goes like this
    howA bot is talked into mocking your company and the chat is shared.
    what leavesyour reputation, in screenshotswho ends up with ita million strangers in a day
    it happenedDPD chatbot, 2024

sources

  1. The Markup, Colin Lecher, "NYC's AI Chatbot Tells Businesses to Break the Law," March 29, 2024.
  2. The Markup / THE CITY, "Malfunctioning NYC AI Chatbot Still Active Despite Widespread Evidence It's Encouraging Illegal Behavior," April 2, 2024.
  3. Associated Press, "NYC's AI chatbot was caught telling businesses to break the law. The city isn't taking it down," April 2024.

← all myths

your turn

what you can do about it

  1. this week

    Take the ten questions where a wrong answer would create real liability and ask the bot each one, then have someone who actually knows the rule grade the replies.

    If you have pointed an assistant at your own policies, your handbook, your terms, the same mechanism is in play: it will answer in your institution's voice whether or not the answer is correct, and the people reading it will trust the voice. The gap you find is the gap a customer is already finding.

  2. go deeper

  3. if you're not sure where to start

    Run the offboarding checklist on one leaver. It covers every account and token to close when someone leaves.

    open the checklist →

the letter · every two weeks

one access gap in your inbox, with the fix.

Every issue takes one way access creeps in, shows a real incident where it went wrong, and ends with one thing to check that week. Written for the people who run the tools, without a security team behind them.

  • 1one mechanism, explained plainly
  • 2one real incident, taken apart
  • 3one step you can take on Monday

get the next issue

One email every two weeks. Your address stays here and is never passed to vendors. One click to unsubscribe. Privacy.