Lexicon format
The lexicon is version-controlled as source CSVs and compiled to JavaScript maps at build time. Four columns describe each word; a three-digit encoding ties them together.
The source CSV
| Column | Type | Notes |
|---|---|---|
Word | string | Source spelling, always lowercase |
PoS | string | Pipe-separated CLAWS7 tags, e.g. NN2|VVZ |
Encoding | string | Three-digit code; 000 = unchanged |
euspelling | string | Pipe-separated new spellings; [] = unchanged. Lowercase except in abbreviation rows, whose expansions (April) keep their capitals |
Word,PoS,Encoding,euspelling night,NN1|NNT1|VV0,101,niht aahs,NN2|VVZ,012,aahs|aahz does,NN2|VDZ,202,does|duz
Abbreviations, contractions, and phrases share the same four-column shape in their own files. Abbreviations are encoded 900, which marks a row the converter never respells. Contractions and phrases take the same codes as single words, because they reform the same way (according to is 101, becoming according tu). Contractions add a wrinkle: the PoS field encodes a sequence of grammatical words (spaces separate positions, pipes separate alternative analyses).
All four are in the repository, and they are the only lexicon files edited by hand: euspell_lexicon.csv, abbreviations, contractions, and phrases.
The compiled form
build/compile-lexicon.js turns each CSV into an auto-generated JS module — a Map keyed by lowercase word:
export const data = new Map([
['aahs', { pos: ['NN2', 'VVZ'], encoding: 12, spellings: ['aahs', 'aahz'] }],
// …~205000 entries
]);lexicon.get(word.toLowerCase())). The encoding is stored as an integer and compared as one (entry.encoding >= 200). matchCase()restores the original word's ALL-CAPS, Title Case, or lowercase afterwards.CLAWS7 tags — the relevant subset
Part-of-speech tags come from the CLAWS7 tagset. The ones that drive disambiguation, each shown with words that carry that tag and no other:
| Tag | Meaning | Example |
|---|---|---|
VV0 | Base-form verb | begin, explain, write |
VVZ | 3rd-singular present | begins, explains, writes |
VVD | Past tense | began, went, wrote |
VVN | Past participle | begun, seen, swum |
VVG | Present participle | discussing, remembering, suggesting |
NN1 | Singular common noun | child, elephant, idea |
NN2 | Plural common noun | children, elephants, ideas |
JJ | Adjective | difficult, happy, tall |
Many words carry more than one tag. Where the reform treats the readings differently, each gets its own spelling and the disambiguator picks between them; otherwise the word keeps one spelling whatever its role:
| Word | Tags | Euspelling |
|---|---|---|
| reads | NN2|VVZ | reads / readz |
| read | NN1|VV0|VVD|VVN | read / redd |
| blessed | JJ|VVD|VVN | blessd / blessed |
| tears | NN2|VVZ | taers / tears / taerz / tearz |
| run | NN1|VV0|VVD|VVN | Unchanged |
Full reference: ucrel.lancs.ac.uk/claws7tags.html
After editing a CSV
Nothing reads the CSVs at runtime. Every tool loads the compiled dist/ modules, so an edit to a CSV changes nothing until it is recompiled — which is what these commands are for, in a checkout of the repository:
npm run build:lexicon # data/*.csv → dist/*.js — after any lexicon edit npm run build # the above, plus the Rollup content bundle npm run gen:svm # only if the NN2|VVZ training data changed
The retrained model, unlike the compiled lexicon, is committed, at src/disambig/vvz-svm.js, so that a build needs no Python.