how it happened
- whoexpert witnessdrafted with GPT-4o
- the gapfabricated citationslooked like real studies
- the resulttestimony excludedJanuary 2025
what everyone assumed
A sworn declaration from a subject-matter expert is supposed to be the reliable part of the record. The citations are there precisely so the claims can be checked, and the signature under oath is supposed to mean someone already did. It is a reasonable belief, the one the whole system of expert testimony rests on: the expert vouches, the footnotes anchor it, and the court can build on it.
what happened
The declaration was prepared to support the Minnesota Attorney General’s defense of a state law on deepfakes in political campaigns. The expert, Professor Jeff Hancock of Stanford, later explained that he drafted text from his own bullet points, left placeholders where citations should go, and ran the draft through GPT-4o. At each placeholder, the model generated citations that looked real. The placeholders were filled, the document was signed, and the manufactured sources rode into the court file looking exactly like the genuine ones around them.
step by step
- an expert drafts with citation placeholdersFor a declaration defending Minnesota's political deepfake law.
- GPT-4o fills the placeholdersWith citations that look real.the assumption breaks here
- opposing counsel checks themTwo studies do not exist; a third has an attribution error.
- 10 January 2025the testimony is excludedNo amended declaration allowed.
how it surfaced
Opposing counsel checked the citations and found that two of the referenced studies did not exist, with a further attribution error in a third. On January 10, 2025, U.S. District Judge Laura M. Provinzino granted the motion to exclude Hancock’s testimony and declined to let him file an amended declaration. The judge noted the irony that an expert on AI and misinformation had, as reported, fallen victim to over-reliance on AI in a case about the dangers of AI. The fabricated citations were generated by GPT-4o; this is reported as the expert’s own account and the court’s ruling, offered as general information, not legal advice.
A signature and a credential do not verify a citation.
The check the document appears to have already passed is a check someone still has to actually run, line by line, against the real sources.
picture it at a company like yours
The company here is made up. The other two paths are not: they happened, and each links to the story.
- in this storythe day it mattersThe other side checks the references and two do not exist.what it costs youThe report is thrown out and the expert's credibility goes with it.
- or it goes like thisit happenedMoffatt v. Air Canada, 2024howA customer-facing bot promises something your policy does not offer.what leavesmoney you now owewho ends up with itthe customer, backed by a tribunal
- or it goes like thisit happenedDPD chatbot, 2024howA bot is talked into mocking your company and the chat is shared.what leavesyour reputation, in screenshotswho ends up with ita million strangers in a day
sources
- "Misinformation Expert's Citation to Fake, AI-Generated Sources...", Eugene Volokh, Reason / The Volokh Conspiracy
- Misinformation expert used AI to draft testimony containing misinformation about AI, Minnesota Reformer
- Minnesota court rejects expert testimony tainted by AI-generated citations, PPC Land
your turn
what you can do about it
this week
Open the sources in one AI-assisted document before anyone relies on it.
Somewhere in your organization, a document with sources in it is about to be relied on, and the sources are being trusted because of who signed it, not because anyone opened them. The quiet months here are short and the lesson is cheap: when a filing, a report, or a brief was drafted with AI assistance, the one place to look this week is the citations, checked against the originals before the document leaves the building.
go deeper
if you're not sure where to start
Run the offboarding checklist on one leaver. It covers every account and token to close when someone leaves.
open the checklist →