🎩 Mr. Chatterbox

by Trip Venturella

We introduce the Mr. Chatterbox model family, a family of historical language models trained on 19th-century British text. The family consists of four fully-trained base models — Mr.Chatterbox-345M, Mr.Chatterbox-500M, Mr.Chatterbox-1.2B and Mr.Chatterbox-2.7B — as well as Mr.Chatterbox-2.7B-Chat, a chat-tuned version of 2.7B trained on fine-tuning data drawn from the corpus. We are also releasing the fine-tuning dataset, Mr.Chatterbox-SFT, and the training code on GitHub.

Read about it Chat model Base models Fine-tuning data Code on GitHub

Models

The family consists of four fully-trained base models, each trained to Chinchilla-optimality for its size.

model	parameters	training tokens	weights
345M	350M	7.0B	Mr.Chatterbox-345M
500M	490M	9.8B	Mr.Chatterbox-500M
1.2B	1.18B	23.6B	Mr.Chatterbox-1.2B
2.7B	2.71B	54.1B	Mr.Chatterbox-2.7B
2.7B-Chat	2.71B	—	Mr.Chatterbox-2.7B-Chat

Pretraining corpus

Corpus composition by decade

Share of training material by publication decade, measured by character volume. The corpus tilts towards the late Victorian period.

decade	share	tokens	documents
1800s	1.1%	0.6B	4,908
1810s	1.4%	0.8B	6,203
1820s	1.9%	1.1B	7,653
1830s	5.2%	2.8B	24,575
1840s	13.3%	7.3B	50,765
1850s	13.7%	7.5B	55,402
1860s	12.8%	7.0B	57,775
1870s	14.4%	7.9B	65,382
1880s	16.4%	9.0B	86,114
1890s	19.7%	10.8B	107,064

218.4B characters (≈ 54.7B tokens) across 465,841 documents. Shares are by character volume; tokens are converted at the corpus-wide rate of characters per token. The 0.7% of documents dated outside 1800–1899 are excluded.

Sources

The pretraining corpus draws on text from these sources, restricted to 1800–1899:

HathiTrust
Institutional Books 1.0
British Library (blbooks)
Project Gutenberg
Internet Archive (National Library of Scotland, UK Medical Heritage Library, Wellcome Collection, European libraries)
Encyclopaedia Britannica
NCSE
Old Bailey Proceedings
Hansard
House of Commons “Blue Books”

Institutional Books provided by the Institutional Data Initiative with source material contributed by Harvard Library, available at institutional.org/institutional-books.

Held-out perplexity

All four models were scored on the same 2,000 sequences of 2,048 tokens each.

Sample: drawn uniformly at random (seed 42) from the 141,185 packed sequences in the held-out split. Its source mix matches the corpus: 66.7% HathiTrust, 8.4% Wellcome, 4.3% British Library, and so on.
Scoring: fp32 weights, scored one sequence at a time. Loss on the [year][source][ocr] prefix tokens is masked, as in training.
Confidence intervals: 95%, from the per-sequence loss.
Bits per byte (BPB): total loss (in bits) divided by the UTF-8 bytes of the predicted text. Unlike perplexity, it doesn’t depend on the tokenizer.

size	mean loss (nats)	perplexity	95% CI	BPB
345M	2.965 ± 0.011	19.39	18.99 – 19.80	1.117
500M	2.846 ± 0.010	17.22	16.87 – 17.57	1.072
1.2B	2.632 ± 0.010	13.91	13.63 – 14.19	0.992
2.7B	2.488 ± 0.012	12.04	11.76 – 12.33	0.937

Supervised fine-tuning corpus

Mr.Chatterbox-2.7B-Chat was fine-tuned on 19,800 conversations containing 46,719 assistant replies; 7,016 of the conversations are multi-turn. Almost every reply is real 19th-century text. In 56.6% of turns, both the question and the reply were taken from period text (catechisms, novel dialogue, play dialogue, etc.). Most of the remaining user turns were back-translated: written to fit a real Victorian passage. The data is released as Mr.Chatterbox-SFT, and the methods that built it are documented, with code, in the GitHub repository.

The replies

reply type	replies	share
Period text, verbatim apart from light clean-up	45,422	97.2%
Period text with a small edit (a name or fact substituted)	671	1.4%
Written by Mr. Chatterbox itself (safety and race fine-tuning rounds)	334	0.7%
Two period passages joined	290	0.6%
Written by the project	2	0.0%

Most replies come from HathiTrust (41%) and Project Gutenberg (36%), with the rest from catechism and question books held as scans (10%), the Internet Archive (5%), Hansard, NCSE, the Wellcome Collection, the British Library and others. 1,359 replies come from text published after 1899; they are flagged in the data.

What the conversations cover

category	conversations	share
Facts and opinions	7,224	36.5%
Behaviour (greetings, abuse, jailbreaks, questions about the bot)	4,599	23.2%
Instructional (advice, descriptions, letters, poems, speeches)	3,036	15.3%
Multi-turn conversations	2,416	12.2%
Topic shifts and keeping track of people	1,503	7.6%
Wit and reasoning	1,022	5.2%

Privacy, content and licence

Real visitor prompts: many prompts were typed by visitors to the Mr. Chatterbox chatbot, who were not told their words might be published. Short or stock prompts (“Tell me a joke”, “Good evening, sir”) are kept verbatim; every other visitor prompt is replaced by a stand-in that does the same job but carries no identifying detail, and real names are swapped for invented ones. These prompts are therefore not identical to the ones the model was trained on.
Removed: 138 conversations containing text from Institutional Books 1.0, whose terms forbid redistribution, and one conversation whose prompts could not be rewritten safely. The released text is the clean version, not the copy with simulated OCR noise the model was also trained on.
Content warning: the replies are period text and reproduce 19th-century attitudes on race, empire, religion, gender and class. Many user turns are deliberately abusive, sexual, bigoted or jailbreak attempts: the data exists to teach the model how to respond to them.
Licence: CC BY-NC-SA 4.0, matching the most restrictive licensed source. The full source list, record format and citation are on the dataset card.

Benchmarks

Mr.Chatterbox 2.7B-Base and Mr.Chatterbox 2.7B-Chat were both evaluated by us using the Chronologic benchmark for historical language models, released by AICHR (Artificial Intelligence for Cultural and Historical Research). Chronologic tests whether a model can answer as a writer of a given date would. It has 866 questions drawn from texts published between 1831 and 1930. We scored the released 2.7B base model and the 2.7B chat model with the benchmark’s own likelihood scorer. Only the model loader was adapted, for our tokenizer and chat format.

model	overall Brier ↓ [95% CI]	cloze acc.	constrained generation acc.	knowledge & inference acc.
talkie-1930-13b-base	0.1574 [0.1551, 0.1596]	39.5%	24.9%	44.0%
Mr. Chatterbox 2.7B (base)	0.1588 [0.1567, 0.1610]	34.4%	27.1%	44.4%
talkie-1930-13b-it	0.1593 [0.1569, 0.1616]	37.4%	23.8%	25.1%
Qwen 7B ft (ckpt-3500)	0.1601 [0.1579, 0.1621]	36.2%	14.9%	34.4%
Mr. Chatterbox 2.7B-Chat	0.1603 [0.1581, 0.1626]	35.3%	27.4%	23.8%
Gemma 4 31B base	0.1606 [0.1585, 0.1626]	23.0%	13.3%	40.6%
Qwen 72B	0.1607 [0.1586, 0.1627]	10.8%	8.8%	38.0%
Qwen 3.5 35B-A3B base	0.1623 [0.1603, 0.1643]	12.6%	8.0%	37.1%
Talkie web base	0.1627 [0.1606, 0.1646]	18.2%	13.6%	34.1%
Qwen 7B	0.1628 [0.1608, 0.1648]	8.6%	9.5%	37.0%
GPT-2 (124M, our run)	0.1635 [0.1615, 0.1656]	16.0%	16.8%	34.2%

Rows in bold are ours. The other rows are the benchmark authors’ published results. Brier is calibrated, as on the leaderboard (median of 1,000 bootstrap draws); lower is better.

The 2.7B base model is second on the leaderboard, behind talkie-1930-13b-base. Their 95% intervals overlap.
Our corpus ends in 1899, but 38% of the eval questions (329) come from texts published 1900–1930. Talkie 1930 was trained on text up to 1930. Even so, overall accuracy barely moves across the line (35.2% before 1900, 34.3% after), because most questions test period language and reasoning rather than knowledge of a particular text. The cutoff does show on the questions that need post-1899 knowledge: the base model answers none of the 16 post-1900 knowledge questions correctly.
Dialogue fine-tuning roughly halves knowledge-and-inference accuracy (44.4% → 23.8%). Talkie shows a similar drop from base to instruction-tuned (44.0% → 25.1%). In our case the loss is almost entirely on the abstention questions, where the right answer is to decline. The base model declines on 56 of 62, the chat model on 7, because Mr. Chatterbox is trained to stay in character and answer. On the knowledge questions themselves, the chat model does slightly better than the base (35 of 94 correct, against 27).
Overlap with our corpus: at least 19 of the benchmark’s 213 source books (9%) are also in our pretraining corpus, matched by library barcode. Other editions were not checked, so this is a lower bound.

Cost and compute

Cost of the released models: about $5,300 in GPU time, plus about $210 for storage and data transfer.
Fine-tuning: a 2.7B fine-tuning run takes about 1.3 GPU-hours on one B200.

model	wall-clock	B200 GPU-hours
345M	~5.1 h (includes instance setup and loading the corpus)	~41
500M	~6.4 h	~51
1.2B	~20.5 h	~164
2.7B	~74 h	~592
total	106.2 h	~850

Acknowledgements

Thank you to:

The Cosmos Institute for their generous support — and a shoutout to Matt Mandel, Kush Kansagra, and all of the “cosmonauts” who helped make it happen.
The Talkie team — David Duvenaud, Alec Radford, and especially Nick Levine — for answering questions, making introductions, and providing inspiration.
Ted Underwood and Teddy Roland at UIUC for their insight, intelligence, and data.
Tim Buckingham for some early conversations.
Lauren Klein at Emory University for reproducing the Mr. Chatterbox pipeline, turning it into a semester-long class (!!), and inviting me to guest lecture.
Ethan Mollick and Simon Willison for bringing attention to the project.
Joe Weisenthal for an important early endorsement.
Bogo Giertler for his enthusiasm.
Jeff Carpenter and James Lahsher for their advice and conversations.
Finally, and most importantly, I want to thank my wife, Emily, for putting up with my endless requests to test out the model, and my son, Robin, for keeping me awake during those 2AM training runs. Thanks to you all!