how it happened
- whouserslogged out unexpectedly
- the gapAI support botinvented a policy
- the resultpublic cancellationsanswers now labelled AI
what everyone assumed
A support bot answers in the company’s name, on the company’s domain, in the company’s tone, so its answers are the company speaking. That is a reasonable thing to believe, because for years it was true: the thing that answered a support ticket was a person reading from a real policy. When the answer arrives with the same confidence and the same letterhead, there is no visible seam where the policy stops and the guess begins.
what happened
Here the quiet stretch was short, the hours a confident wrong answer ran before anyone outside could check it. The bot, known as Sam, was asked to explain the logouts. It produced a clean, plausible reason: a one-device-per-subscription rule presented as a deliberate security design. The rule did not exist. The real cause was a backend change. There was no such policy. But the answer carried no label saying a model had written it, so the people reading it had no way to tell an invented rule from a real one.
step by step
- a backend change logs users outPeople ask support why.
- the support bot invents a policyIt describes a one-device rule that does not exist, unlabelled as AI.the assumption breaks here
- users take it as officialSome post that they and their workplaces are cancelling.
- the cofounder apologisesAI replies in support email will be labelled from then on.
how it surfaced
Users took the answer as official. Some posted on Reddit that they were cancelling, and that their workplaces were dropping the tool. The assumption retired itself in public, in screenshots. Cofounder Michael Truell apologised, explained that a backend security change had caused the logouts, and said that AI responses used for email support would from then on be clearly labelled as AI.
A bot can produce text in the company’s voice without producing the company’s policy.
The voice and the authority are not the same thing, and a confident answer is not evidence that the answer is grounded in anything.
picture it at a company like yours
The company here is made up. The other two paths are not: they happened, and each links to the story.
- in this storythe day it mattersIt invents a company policy to explain a bug, and customers believe it.what it costs youCustomers cancel over a rule you never made, and your founder writes the apology.
- or it goes like thisit happenedMoffatt v. Air Canada, 2024howA customer-facing bot promises something your policy does not offer.what leavesmoney you now owewho ends up with itthe customer, backed by a tribunal
- or it goes like thisit happenedDPD chatbot, 2024howA bot is talked into mocking your company and the chat is shared.what leavesyour reputation, in screenshotswho ends up with ita million strangers in a day
sources
your turn
what you can do about it
this week
Look at what your support automation is allowed to assert without a human in the loop, and whether a customer can tell which answers were written by a model.
Somewhere in your company an automated reply is answering customers right now, on your domain, in your name. The question the quiet hours ask is a small one: when it states a policy, is it reading from a real source, or is it composing a plausible one.
go deeper
if you're not sure where to start
Run the offboarding checklist on one leaver. It covers every account and token to close when someone leaves.
open the checklist →