MTMalte Therkildsen Lab LanguageP42 · Language model StartingStep 0Loss 4.22

Train a language model in this tab

A transformer with 0 parameters learns to write from Alice’s Adventures in Wonderland while you read, and trains on your device. Its forward pass, backward pass and optimizer are written for this page, 562 lines for the graphics card and 441 lines of JavaScript for the processor, with no library. No server is called.

Temperature
Paste at least 2,000 characters. The text stays in this tab.

Less sure More sureWaiting for the first sample

Loss

Cross entropy per character, where lower is better. The line is the text it trains on. The dots are the last 10% of the book, which it never trains on. A model that guesses evenly among the 68 characters scores 4.22.

Training text
4.22
Held out text
4.22
Characters per second
0
Runs on
0

When it learns to copy

An induction head is an attention head that finds the last place the current letter appeared and predicts the letter that followed it there. Anthropic found that these heads form suddenly during training, at the same point as a bump in the loss, and argued that they may be how models use most of their context (Olsson and others, 2022 ↗). The test below is a string of 31 random letters written twice. No model can predict the first copy; a model with an induction head predicts most of the second.

Strongest head
None yet
Its score
0.00
Loss on the first copy
4.22
Loss on the second copy
4.22
Formed
Not yet

Switch heads off

Select heads to switch them off, and the test runs again without them. An induction head depends on a head in an earlier layer that puts its weight on the previous letter, so both scores are shown for each head.

First copy
Second copy
Repeated passages in training

Ordinary prose holds few exact repeats inside a short context, so 1 in 2 training sequences repeats a passage from the book of 10 or more characters, with a different length each time. In a test on the processor with a smaller model and every sequence repeated, two heads crossed 0.3 at step 1,700, and the second copy cost 2.8 less than the first. Choose None to train on the book alone.

The score is the attention weight a head puts on the letter after the earlier occurrence, averaged over the test strings: 1 means all of it, and a head that spreads its weight evenly scores close to 0. One line per head. The model counts as having an induction head when one head scores 0.3 or more and the second copy costs at least 1 less than the first.

What it expected, letter by letter

A sentence from the held out text. Each letter is darker the more probability the model gave it before seeing it. Select a letter to see the five characters it rated highest at that point; the bar under each earlier letter shows the attention weight on it.

Bars show all heads averaged.

Select a letter

Point at one, or focus the sentence and use the arrow keys.

0%Average probability of the right letter in this sentence

Embeddings and attention

The left view places each character by the 64 numbers it has learned for it. The right view shows all 8 attention heads on the sentence above.

Where each character sits

Each character’s 64 numbers, reduced to the 2 directions that vary most. Characters it treats alike end up close.

Every attention head

Rows are positions in the sentence, columns are the earlier positions each one weighs. Select a head to show it in the sentence.

Darker cells carry more weight. When a letter is selected, its row is shaded in every head.

Features it has learned

A sparse autoencoder rewrites each position’s 64 numbers as a few active features out of 256. It is the method Anthropic used to find readable features in a small transformer (Bricken and others, 2023 ↗) and later in Claude 3 Sonnet (Templeton and others, 2024 ↗). Here it trains on the model’s last layer, as the model is now.

Trains on 9,600 positions of the book, on the processor. The other views wait until it is done.

Write with it

Type a start, and the bars show its probabilities for the next character. Continue adds 120 characters at the temperature set above.

Tab adds the top guess

Next character

How the writing changed

A sample is kept each time the step count doubles, all from the same prompt. The first row is the model before any training.

  • Samples appear here as it trains.

How it is built

The model, its training and every view on this page are written for it, in five files.

What it is

A decoder transformer with 2 layers, 4 attention heads per layer, 64 numbers per character and a context of 64 characters: 0 parameters, trained on your device in batches of 8 sequences. With WebGPU the model is larger and every step runs on the graphics card; a worker on the processor reads the weights about once a second for the views on this page.

What it shows

How the probability of the right next character rises as the model trains, and what changes in its writing along the way: spacing, spelling, punctuation and dialogue. The sample, the sentence and the views update as it trains.

In your project

A model this size trains and runs in a visitor’s browser, so their text stays on their device and no server is called. The limit: it learns the spelling and rhythm of a text, not its facts, and it needs at least 2,000 characters to start.

Gradient check
Running

Comparing the backward pass with central differences on a small copy.

Graphics card against reference
Running

Comparing the gradients from your graphics card with the JavaScript reference.

Determinism
Running

Two runs from the same seed.

Speed
0steps/s

8 sequences of 64 characters per step.

How Language is built
  1. The page reads the book, keeps its first 90% for training and the last 10% as held out text, and gives every distinct character a number. A step takes a batch of sequences from the training part; with the repeats on, 1 in 2 sequences copies a passage of 10 or more characters from earlier in the same sequence.
  2. The model is a decoder transformer: an embedding for each character and each position, then layers of attention with several heads and a two layer network, each with layer normalization and a residual connection, and a last layer that gives a probability for every character.
  3. With WebGPU, compute shaders run the forward pass, the backward pass and Adam, an optimizer, for every step on the graphics card. Without it, the same model in JavaScript trains in workers on the processor, each with a part of the batch, and the page adds their gradients.
  4. Three checks run when the page opens: the backward pass against central differences on a small copy, the gradients from the graphics card against the JavaScript reference, and two runs from the same seed, which must match.
  5. About once a second a worker reads the weights and computes the views: a sample, the loss on held out text, the induction test, the letter by letter sentence and the attention heads.
  6. Find features trains a sparse autoencoder on the model’s last layer in a worker. Turning a feature up adds its direction to that layer at every position while the model writes.
  7. Training stops while the tab is hidden. The page uses no library, and nothing is sent to a server.