Quire, a PDF reader written from nothing
Quire reads a PDF file, finds its pages and draws them on a canvas, and reads their text back out. It was written from an empty file and the specification, and it stands on two of the engines before it: Thaw inflates its compressed streams, and Glyph reads the TrueType fonts embedded in the files. Below, it draws a page as slowly as you like, and the numbers further down hold it against pdf.js, the reader in Firefox.
·
·
Everything on the canvas is drawn by Quire: the paths, the images, the gradients and the letters of every embedded TrueType and Type3 font. Where a font’s outlines are in a format Quire does not read, the browser draws the letters in a font like it, and the panel says so. A file you open is read in this page and not sent anywhere.
Read from the end
A PDF is a heap of numbered objects: dictionaries, arrays, names, strings, numbers and streams of bytes. Their places are listed only in the last lines of the file. There, after startxref, is the place of the cross reference table, which gives the byte offset of every object, and the trailer, which names the catalog the pages hang from. So a reader starts at the end of the file and works backwards.
A file that has been edited and saved again keeps its old bytes and adds the changes after them, with a new table that points back to the one before. Newer files pack their table into a compressed stream and many small objects into object streams. Quire reads all of these. When the table is wrong, which happens in real files more than it should, Quire scans the whole file for obj and builds the table again, as every reader has to.
These are the objects of the file drawn above, printed from what Quire parsed and not from the bytes. A reference is a button that goes to its object. A stream shows its dictionary and then its contents, inflated by Thaw where it was compressed, so a page’s content stream reads as the program it is.
A page is a program
The content of a page is a list of operators in reverse Polish notation: the operands first, then the operator. re adds a rectangle to the path, f fills it, cm changes the coordinate system, q and Q save and restore the whole graphics state. Text is the same: BT starts it, Tf sets the font and size, Tj shows a string, and a text matrix moves along as each character is placed.
Quire runs that program against a canvas. Its matrices multiply in the order the specification writes them, which is the opposite of the order canvas code is usually written in, and the most common bug while writing Quire was a page drawn mirrored, upside down or a page away. A form is a small program of its own that a page can call, with its own resources, and a transparency group is drawn on a canvas of its own first and laid onto the page with its opacity. The slider above stops the program after any operator, so you can watch a page being built in the order its writer wrote it.
Four kinds of font
A font in a PDF is a dictionary that maps its codes to glyphs, gives the width of each glyph, and often contains the font program itself. A TrueType font goes to Glyph, the renderer from the note before last, which already reads outlines: Quire asks it for the outline of each glyph and fills it as a path. Fonts cut down for a PDF lack tables a font on its own must have, so Glyph learned a loose mode for them. A Type3 font is glyphs written as small content streams, and Quire draws each as a program, like a form.
Getting the text back out is a different question from drawing it. A code in a string is only a number. A ToUnicode map, when the file has one, says which characters it stands for. When it has none, Quire goes through the names of the glyphs, the standard encodings and the font’s own tables, and, for a font with none of these, reads the font’s character map backwards. The checkbox above lays what it read over the page, a box for each character.
Held against pdf.js
Four files from three different writers, the samples above, drawn by Quire and by pdf.js in the same Chromium at the same scale. The pixels are compared one by one, and the text of each page is compared as a count of each character, so a different order does not count.
$ node web/notes/quire/held.mjs 4 PDFs from three writers, every page drawn by Quire and by pdf.js 4.10.38 in the same Chromium at scale 1.5 pixels differ when a channel is off by more than a quarter of its range; "as paths" is pdf.js set to draw its fonts as outlines, the way Quire does, instead of handing them to the browser's text engine file pages characters differ pixels differ as paths Quire pdf.js chromium.pdf 2 3,080 3 2,260,032 0.71% 0.02% 84 ms 95 ms libreoffice.pdf 1 602 0 1,090,584 0.31% 0.00% 14 ms 49 ms reportlab-plain.pdf 1 385 3 1,127,859 0.01% 1.17% 45 ms 63 ms reportlab.pdf 2 425 3 2,255,718 0.01% 0.76% 25 ms 62 ms characters: 4,489 of the 4,495 pdf.js read, Quire read the same pixels: 0.29% differ, and 0.46% with pdf.js drawing its fonts as paths
Most of the pixels that differ are the edges of letters. pdf.js hands a font to the browser’s own text engine, which smooths edges its own way. Told to draw its fonts as paths, as Quire does, it differs from Quire on far fewer pixels for the files from Chromium and LibreOffice. For reportlab it is the other way round, because its standard fonts are not embedded and both readers ask the browser for a font like them, which is what pdf.js stops doing when it is told to draw paths. The only characters that differ are the same ligature in three of the files, which pdf.js unfolds into the two letters it is made of.
Held against pdf.js’s own tests
pdf.js keeps hundreds of PDFs in its repository to catch its own mistakes, broken and odd ones on purpose among them. The command fetches every one from the tag of the pdf.js version in use, and both readers read the text of every page.
$ node web/notes/quire/suite.mjs
pdf.js 4.10.38's test suite: 577 files in its repository, read by Quire and by pdf.js in 23 s
pdf.js read 577; of those, Quire read 571 and refused 6
pages: Quire found 1,097, pdf.js 1,097
text, in the first fifty pages of each: of the 458,821 characters pdf.js read, Quire read 451,347 the same (98.37%), and every one in 508 files
the reasons for each refusal:
6 The file is encrypted, and Quire does not decrypt
Every file Quire could not open is encrypted, and Quire does not decrypt. Much of the text that differs is in fonts that keep their characters only inside a font program Quire does not read, where pdf.js reads the names out of the font itself. Running this suite found most of what is in this note: a trailer whose catalog is the wrong object, text that stands outside the page, code page decoded as Latin one by the decoder in Node, and a page whose crop box was all that told three pages apart.
What it does not have
Quire does not decrypt, so a file with a password, or one locked against copying, is refused. It reads no Type1 or CFF outlines and lets the browser draw those letters in a similar font. It decodes JPEG through the browser and leaves JPEG , JBIG2 and fax images out. It leaves out soft masks and the shadings made of meshes, and text used as a clipping path is drawn but does not clip. It draws the page and not what lies on it: annotations, form fields, links and the outline are not read. Each thing it leaves out on a page is named in the panel beside it.
The source is in the portfolio’s repository under web/notes/quire/: pdf.js for the file, its objects and its filters, content.js for the painter, fonts.js and encodings.js for fonts and their characters, held.mjs and suite.mjs for the commands, and corpus.mjs for how the samples were written.