mythsthe wider recordMata v. Avianca2023legal

everyone assumed:

ChatGPT's case citations can be trusted

what was actually true:

A model produces the shape of a fact with the same fluency whether or not the fact exists.

how it happened

  1. wholawyersasked ChatGPT for cases
  2. the gapinvented casesnever checked
  3. the resultsanctions$5,000 and bad faith finding
the gap that let it happen the result

what everyone assumed

A fluent, specific, correctly formatted citation reads as a real one, because that is what real ones look like. The assumption is that a tool confident enough to produce a docket number has a docket number behind it.

what happened

None of the six cases existed. Opposing counsel could not find them; neither could the court. Asked to produce them, the lawyers supplied more fabricated text the model generated. The confidence never wavered, because confidence was all the model had.

step by step

  1. lawyers use a chatbot for legal researchIt returns six cases with full citations.
  2. none of the six existNeither opposing counsel nor the court can find them.the assumption breaks here
  3. asked for the cases, they supply more fabricated textThe model stays confident.
  4. June 2023sanctions and a $5,000 fineThe court finds bad faith.

how it surfaced

In June 2023 a federal judge sanctioned the lawyers in Mata v. Avianca, finding bad faith and fining them $5,000. It became the canonical example of AI fabrication reaching a court of record.

A model produces the shape of a fact with the same fluency whether or not the fact exists.

Confidence is not evidence, and polish is not accuracy, however identical they look on the page.

picture it at a company like yours

The company here is made up. The other two paths are not: they happened, and each links to the story.

your company
a normal daySomeone uses AI to research and draft a formal document with references.
the gap nobody seesThe AI invents some references, written just as confidently as the real ones.AI output nobody checks
  1. in this story
    the day it mattersThe other party asks for copies of the sources, and they do not exist.
    what it costs youYou face sanctions or a lost case, and the embarrassment is public.
  2. or it goes like this
    howA customer-facing bot promises something your policy does not offer.
    what leavesmoney you now owewho ends up with itthe customer, backed by a tribunal
    it happenedMoffatt v. Air Canada, 2024
  3. or it goes like this
    howA bot is talked into mocking your company and the chat is shared.
    what leavesyour reputation, in screenshotswho ends up with ita million strangers in a day
    it happenedDPD chatbot, 2024

sources

  1. Mata v. Avianca, 678 F.Supp.3d 443 (S.D.N.Y. 2023)

← all myths

your turn

what you can do about it

  1. this week

    Name where AI-written claims and citations leave the company unchecked.

    Someone in your company is using AI to draft something with citations, figures, or claims this week, and the tool states the false ones as smoothly as the true ones. Mata is what happens when nobody verifies before it ships. Where AI output leaves the building unchecked is the exposure worth naming.

  2. go deeper

  3. if you're not sure where to start

    Run the offboarding checklist on one leaver. It covers every account and token to close when someone leaves.

    open the checklist →

the letter · every two weeks

one access gap in your inbox, with the fix.

Every issue takes one way access creeps in, shows a real incident where it went wrong, and ends with one thing to check that week. Written for the people who run the tools, without a security team behind them.

  • 1one mechanism, explained plainly
  • 2one real incident, taken apart
  • 3one step you can take on Monday

get the next issue

One email every two weeks. Your address stays here and is never passed to vendors. One click to unsubscribe. Privacy.