SoftwareCaseP42
A model learns to write in this tab.
Language
0steps
0.00loss on the passages it trains on
0characters read a second
The model starts when this plate is in view
likelypossiblea long shot
A transformer learns to write from Alice's Adventures in Wonderland while this plate is in view, on one thread of your processor.
TransformerWeb WorkerNo libraryOpen LanguageEach character is a guess at the next one. The brightness says how sure the model was of it.
0.00
Loss is how surprised the model is by the character that really comes next. Lower is better, and the gap between the line and the dots is what it memorised.
Cross entropyAdamWAttention is how the model reads back. Each head learns its own way to look at what came before.
Language learns one book. The last plate says what a larger model has.
Has
- A transformer written by handAttention, the backward pass, AdamW and sampling in plain JavaScript, with no library.
- The modelA few layers of attention heads and a short context.
- Training in this tabA batch of passages a step, with a rate that warms up and falls.
- A held out tenthThe last tenth of the book is never trained on, so the dots measure what it learned and the line what it read.
Does not have
- WordsIt reads and writes one character at a time, from the characters of one book.
- MeaningIt learns how the book's sentences are spelled and paced, and nothing about what they say.
- The graphics card on this pageThe Lab trains a larger model through WebGPU where the browser has it.
- Saved weightsA reload starts from random numbers again, and nothing goes to a server.
The book is Lewis Carroll's, from Project Gutenberg. The Lab page turns heads off, steers the model and finds its features with a sparse autoencoder.
How it works