Skip to the content

SoftwareCaseP42

A model learns to write in this tab.

Language

0steps

0.00loss on the passages it trains on

0characters read a second

The model starts when this plate is in view

likelypossiblea long shot

01 What it writes after the word Alice

A transformer learns to write from Alice's Adventures in Wonderland while this plate is in view, on one thread of your processor.

TransformerWeb WorkerNo libraryOpen Language

Each character is a guess at the next one. The brightness says how sure the model was of it.

0.00

02 The judge

Loss is how surprised the model is by the character that really comes next. Lower is better, and the gap between the line and the dots is what it memorised.

Cross entropyAdamW

Attention is how the model reads back. Each head learns its own way to look at what came before.

03 Where each head looks

Each square is one head on the line above.

Ablate a head in the Lab

Language learns one book. The last plate says what a larger model has.

Has

  • A transformer written by handAttention, the backward pass, AdamW and sampling in plain JavaScript, with no library.
  • The modelA few layers of attention heads and a short context.
  • Training in this tabA batch of passages a step, with a rate that warms up and falls.
  • A held out tenthThe last tenth of the book is never trained on, so the dots measure what it learned and the line what it read.

Does not have

  • WordsIt reads and writes one character at a time, from the characters of one book.
  • MeaningIt learns how the book's sentences are spelled and paced, and nothing about what they say.
  • The graphics card on this pageThe Lab trains a larger model through WebGPU where the browser has it.
  • Saved weightsA reload starts from random numbers again, and nothing goes to a server.
04 What was left out

The book is Lewis Carroll's, from Project Gutenberg. The Lab page turns heads off, steers the model and finds its features with a sparse autoencoder.

How it works
Next case Mail domains connected and checked