Train a language model in this tab
A transformer with 0 parameters learns to write from Alice’s Adventures in Wonderland while you read, and trains on your device. Its forward pass, backward pass and optimizer are written for this page, 562 lines for the graphics card and 441 lines of JavaScript for the processor, with no library. No server is called.
Loss
Cross entropy per character, where lower is better. The line is the text it trains on. The dots are the last 10% of the book, which it never trains on. A model that guesses evenly among the 68 characters scores 4.22.
When it learns to copy
An induction head is an attention head that finds the last place the current letter appeared and predicts the letter that followed it there. Anthropic found that these heads form suddenly during training, at the same point as a bump in the loss, and argued that they may be how models use most of their context (Olsson and others, 2022 ↗). The test below is a string of 31 random letters written twice. No model can predict the first copy; a model with an induction head predicts most of the second.
Switch heads off
Select heads to switch them off, and the test runs again without them. An induction head depends on a head in an earlier layer that puts its weight on the previous letter, so both scores are shown for each head.
Ordinary prose holds few exact repeats inside a short context, so 1 in 2 training sequences repeats a passage from the book of 10 or more characters, with a different length each time. In a test on the processor with a smaller model and every sequence repeated, two heads crossed 0.3 at step 1,700, and the second copy cost 2.8 less than the first. Choose None to train on the book alone.
The score is the attention weight a head puts on the letter after the earlier occurrence, averaged over the test strings: 1 means all of it, and a head that spreads its weight evenly scores close to 0. One line per head. The model counts as having an induction head when one head scores 0.3 or more and the second copy costs at least 1 less than the first.
What it expected, letter by letter
A sentence from the held out text. Each letter is darker the more probability the model gave it before seeing it. Select a letter to see the five characters it rated highest at that point; the bar under each earlier letter shows the attention weight on it.
Select a letter
Point at one, or focus the sentence and use the arrow keys.
Embeddings and attention
The left view places each character by the 64 numbers it has learned for it. The right view shows all 8 attention heads on the sentence above.
Where each character sits
Each character’s 64 numbers, reduced to the 2 directions that vary most. Characters it treats alike end up close.
Every attention head
Rows are positions in the sentence, columns are the earlier positions each one weighs. Select a head to show it in the sentence.
Darker cells carry more weight. When a letter is selected, its row is shaded in every head.
Features it has learned
A sparse autoencoder rewrites each position’s 64 numbers as a few active features out of 256. It is the method Anthropic used to find readable features in a small transformer (Bricken and others, 2023 ↗) and later in Claude 3 Sonnet (Templeton and others, 2024 ↗). Here it trains on the model’s last layer, as the model is now.
The feature’s direction is added to the model’s last layer at every position. Strength is a multiple of the feature’s largest activation in the book. Both samples use the same random seed, so every difference comes from the feature.
Write with it
Type a start, and the bars show its probabilities for the next character. Continue adds 120 characters at the temperature set above.
Next character
How the writing changed
A sample is kept each time the step count doubles, all from the same prompt. The first row is the model before any training.
- Samples appear here as it trains.
How it is built
The model, its training and every view on this page are written for it, in five files.
What it is
A decoder transformer with 2 layers, 4 attention heads per layer, 64 numbers per character and a context of 64 characters: 0 parameters, trained on your device in batches of 8 sequences. With WebGPU the model is larger and every step runs on the graphics card; a worker on the processor reads the weights about once a second for the views on this page.
What it shows
How the probability of the right next character rises as the model trains, and what changes in its writing along the way: spacing, spelling, punctuation and dialogue. The sample, the sentence and the views update as it trains.
In your project
A model this size trains and runs in a visitor’s browser, so their text stays on their device and no server is called. The limit: it learns the spelling and rhythm of a text, not its facts, and it needs at least 2,000 characters to start.
Comparing the backward pass with central differences on a small copy.
Comparing the gradients from your graphics card with the JavaScript reference.
Two runs from the same seed.
8 sequences of 64 characters per step.