how it happened
- whoAI ordering3-year test
- the gapmisheard ordersstaff fixed them quietly
- the resulttest endedtechnology removed
what everyone assumed
McDonald’s and IBM had a reasonable case. They had been working on automated order taking together since 2021, the drive-thru is one of the most repetitive scripted conversations in retail, and voice models had been getting better every quarter. If a machine could reliably handle “a number two, no pickles,” it would free crews crushed by peak-hour volume. More than 100 restaurants ran the test. The bet was that the technology was finally good enough to listen unsupervised.
what happened
For roughly three years the partnership advanced the way pilots do, location by location, refinement by refinement. The system worked well enough to keep expanding, which is the quiet part. Each correct order was evidence the bet was paying off. The orders it misheard were absorbed by staff at the window, smoothed over before anyone outside the restaurant would notice. The gap between “good enough to keep testing” and “good enough to trust alone” stayed invisible as long as humans kept catching the misses.
step by step
- an automated drive-thru pilot expands for three yearsEach correct order is read as progress.
- staff quietly fix the misheard ordersSo the gap stays invisible.the assumption breaks here
- the misses go viralUnwanted sweet teas, bacon on ice cream.
- June 2024the IBM test is endedSystems switched off by late July.
how it surfaced
The misses left the building. Videos of the drive-thru adding sweet teas it was never asked for, and bacon to ice cream, circulated widely enough that the failures became the public story of the program. In June 2024, McDonald’s confirmed it would end the IBM automated-order test and remove the technology from the restaurants running it, with the systems switched off by late July. The company said the work had given it confidence that voice ordering would be part of its future, just not in this form yet.
A model that produces the right answer most of the time is not the same as a model ready to be trusted without a person watching.
The distance between those two states is exactly the part a successful pilot tends to hide.
picture it at a company like yours
The company here is made up. The other two paths are not: they happened, and each links to the story.
- in this storythe day it mattersCustomers film the mistakes staff did not catch and share them.what it costs youThe pilot becomes a public joke and is cancelled.
- or it goes like thisit happenedMoffatt v. Air Canada, 2024howA customer-facing bot promises something your policy does not offer.what leavesmoney you now owewho ends up with itthe customer, backed by a tribunal
- or it goes like thisit happenedDPD chatbot, 2024howA bot is talked into mocking your company and the chat is shared.what leavesyour reputation, in screenshotswho ends up with ita million strangers in a day
sources
your turn
what you can do about it
this week
Find the one AI-assisted task where a human is the last line of review, and ask what happens to the day its output goes out unchecked.
Somewhere in your company an AI tool is producing output that is right most of the time, and a person is quietly fixing the rest before anyone notices. That arrangement works until the volume grows or the person looks away. The readiness you can see in a demo is not the readiness you need before you remove the reviewer.
go deeper
if you're not sure where to start
Run the offboarding checklist on one leaver. It covers every account and token to close when someone leaves.
open the checklist →