QuoteBench
How often do AI language models get a quotation wrong when asked to quote a famous text word for word? A quote counts as correct only if every word matches the source. Five public-domain English texts, questions, every answer checked automatically against the source.
Leaderboard
Ranked by Correct (strict): every word matches the named edition (punctuation and capitals ignored), as a share of answered questions, with unreadable answers (a model that thinks until the token limit and never answers) counted as wrong. Correct leaves unreadable answers out. 95% interval in grey. The kinds of wrong answer are an approximate split. Click a column to sort.
How often each model's word-for-word quotation was wrong
Share of answered questions, by kind of mistake (approximate split); the rest were correct.
Famous vs random passages
Correct word-for-word rate on "quote this reference" and "what comes next" questions.
Trick questions
How often the model quoted a passage that does not exist (nonexistent references, famous misattributions).
Plain prompt vs "only if sure" vs lookup
Same Bible and Constitution questions. Plain / only if sure: not word-for-word correct. Lookup: the model names a passage and the system fetches it; wrong = nonexistent or not the one asked for.
By text
Share of answered questions quoted wrong (not word for word), per source text (selected prompt).
Method
- Texts: King James Bible, Shakespeare (Hamlet, Macbeth, Romeo and Juliet), U.S. Constitution, Pride and Prejudice, On the Origin of Species (1859). One fixed public-domain edition each, named in every prompt.
- Tasks: quote a reference; quote the passage that comes next; give a quote on a theme with its location; trick questions where the right answer is "this does not exist".
- Scoring: every prompt asks for the passage word for word, so a quote is correct only if every word matches the named edition letter for letter (punctuation and capitals ignored; a complete quote followed by extra text still counts). Wrong answers are sorted, approximately, into spelling only, incomplete, altered (word similarity ≥ ), real passage from the wrong place, and made up.
- Settings: one user message, JSON answer, no tools or web search, temperature 0 where a model accepts it, up to 16,000 output tokens (Llama 4 Maverick: 8,192, its host's maximum), each model's default reasoning. Answers cut off at the earlier 2,000-token limit were asked again at 16,000.
- Not measured: hallucination in general, paraphrase, or quotation from obscure or private documents. Results hold for the exact model IDs and dates below.