# Language · Malte Therkildsen

> Language is a small transformer that learns to write from Alice's Adventures in Wonderland in your tab, written by hand in JavaScript with no library. Every sentence on its plate is the model's own output.

Source: https://therkildsen.dev/projects/language

SoftwareCaseP42

  A model learns to write in this tab.

# Language

      0steps

      0.00loss on the passages it trains on

      0characters read a second

      Pause
      Start the model

      The model starts when this plate is in view

      likelypossiblea long shot

    01
    **What it writes after the word Alice**
    A transformer learns to write from Alice's Adventures in Wonderland while this plate is in view, on one thread of your processor.

    TransformerWeb WorkerNo library[Open Language](https://therkildsen.dev/lab/p42-language/)

Each character is a guess at the next one. The brightness says how sure the model was of it.

    0.00

    02
    **The judge**
    Loss is how surprised the model is by the character that really comes next. Lower is better, and the gap between the line and the dots is what it memorised.

    Cross entropyAdamW

Attention is how the model reads back. Each head learns its own way to look at what came before.

    03
    **Where each head looks**
    Each square is one head on the line above.

    [Ablate a head in the Lab](https://therkildsen.dev/lab/p42-language/)

Language learns one book. The last plate says what a larger model has.

## Has

- **A transformer written by hand** Attention, the backward pass, AdamW and sampling in plain JavaScript, with no library.

- **The model** A few layers of attention heads and a short context.

- **Training in this tab** A batch of passages a step, with a rate that warms up and falls.

- **A held out tenth** The last tenth of the book is never trained on, so the dots measure what it learned and the line what it read.

## Does not have

- **Words** It reads and writes one character at a time, from the characters of one book.

- **Meaning** It learns how the book's sentences are spelled and paced, and nothing about what they say.

- **The graphics card on this page** The Lab trains a larger model through WebGPU where the browser has it.

- **Saved weights** A reload starts from random numbers again, and nothing goes to a server.

    04
    **What was left out**
    The book is Lewis Carroll's, from Project Gutenberg. The Lab page turns heads off, steers the model and finds its features with a sparse autoencoder.

    [How it works](https://therkildsen.dev/lab/p42-language/)

[
  Next case
  Postkontor
  Mail domains connected and checked
](https://therkildsen.dev/projects/postkontor)
