Embeddings Are Overpriced? A Frozen 16 MB Vocabulary for Small and Ternary Language Models

AI summary

Overview

Research area: Natural Language Processing — specifically, efficient language-model architecture, token embedding design, and low-bit (ternary) model deployment.

Technical level: Intermediate. The paper is plainly written and well structured, but it assumes familiarity with token embeddings, validation loss in nats, quantisation formats (GGUF, Q8_0), and zero-shot benchmarks like LAMBADA and PIQA.

Scope: A controlled, single-author ablation that replaces Gemma 3 270M's 167.8M-parameter learned embedding table with a frozen 16.8 MB table of 512-bit codes and measures the quality, size, speed, and extensibility trade-offs against a learned table and a ternary variant.

What This Paper Is About

In small language models the vocabulary is the largest single component: in Gemma 3 270M, 167.8M of 268.1M parameters (62.6%) are the tied embedding table, and because common quantisation methods leave that table in 16-bit, it also dominates the compressed file. The paper asks what that table actually buys, replacing it with a frozen 16.8 MB set of 512-bit codes derived once from a pretrained embedding and then training the same Gemma-shaped body from scratch with and without it. The goal is to determine whether a fixed, rule-generated vocabulary can match a learned one at a fraction of the parameters and bytes.

Key Contributions

1. A controlled comparison of learned versus frozen vocabularies. Three arms share the same body architecture, the same 2.05B training tokens, the same batches in the same order, and the same seed; only the vocabulary (and, in the ternary arm, the body type and schedule) changes.

2. A frozen table construction and interface. A 512-bit code per token, obtained by centring Gemma 3 270M's embedding matrix, taking the top 512 principal directions, and keeping the sign of each projection, plus small trained 512↔640 interface maps (0.65M parameters) and a fixed log-unigram prior ("the seal") costing 1 MB with no trainable parameters.

3. A working ternary deployment. A BitNet b1.58 body over the frozen table packs into a 46.2 MB file and runs on a laptop CPU via a released AVX2 kernel at 615 to 740 tokens per second in 62 MB of peak memory.

4. Post-training vocabulary extension and full release. 10,000 new biomedical words are added to the frozen table with no training at a 3% quality cost, where the learned table degrades by 42.9%; all weights, data, logs, and per-question evaluation results are released.

Main Findings

• The learned table's loss advantage is real but modest: Arm A (learned, float) reaches 3.155 validation loss, Arm B (frozen, float) 3.358, and Arm C (frozen, ternary) 3.439. B minus A is +0.203 nats, which the paper describes as about 23% lower perplexity for the 2.6 times larger trainable parameter count.

• On benchmarks the frozen table is better or tied on three of four tasks: B beats A on LAMBADA by 1.6 points (23.2 versus 21.6), ties on HellaSwag (30.4 versus 30.4) and PIQA (+0.9 points, range includes zero), and loses only on ARC-Easy by 3.4 points (42.5 versus 45.9).

• The advantage is a capacity the learned table grows into, not a head start that fades: B starts 0.23 nats ahead at step 200 because its codes already carry Gemma's meanings; A overtakes at step 600 and the gap stabilises at 0.20 nats from about step 3,000 to the end.

• The frozen table wins decisively on rare words: on tokens seen 10 to 99 times in training, B is 2.34 nats better than A and C is 1.7 nats better. On the 161 most common tokens, which are 52% of all text, the gap is only 0.035 nats. Of Gemma's 262,144 tokens, 52,782 never appear in the 2.05B training tokens and only 147,187 appear ten or more times.

• The ternary model ties the float learned-table model on two benchmarks: C (22.5 LAMBADA, 59.5 PIQA) is within the paired error range of A on LAMBADA and PIQA, is 0.9 points behind on HellaSwag and 4.9 behind on ARC-Easy, and is within 0.8 to 2.0 points of B everywhere and 0.08 nats in loss.

• Large speed and memory gains on a laptop CPU: on an Intel Xeon W-11955M with 8 threads, C generates 615 to 740 tokens per second in 62 MB of peak memory from a 46 MB file, against 116 tokens per second and 471 MB for A at GGUF Q8_0 — 5.3 to 6.4 times faster in 7.6 times less memory. Perplexity on 300k validation tokens is 23.864 for the f16 and Q8_0 files and 23.97 for Q4_K_M, so 8-bit loses essentially nothing.

• The floor on the float model is the vocabulary: llama.cpp keeps the 262,144 by 640 table in 16-bit, so 250 MB of the 253 MB Q4_K_M file is the vocabulary; A cannot go below 446 MB peak at any quantisation.

• New words are accepted without training, at a cost: adding the 10,000 most frequent biomedical words that Gemma's tokenizer splits into two or more pieces replaces 3.2% of tokens on 300 held-out PubMed articles at a cost of +3.3% (B) and +3.1% (C) nats per character, while the learned table degrades by +42.9%. At 2,000 tokens the costs are +2.0% (B) and +1.9% (C) against +20.2% (A).

• Cross-lingual behaviour is mixed: B beats A overall in Arabic, Bulgarian, Spanish, Russian, and Vietnamese and is worse in the other fifteen of twenty languages tested. Setting the prior of never-seen tokens to the median of seen tokens makes B better than A on never-seen tokens in Thai, Arabic, Russian, French, and Greek, while A stays ahead in Hindi, Japanese, and Chinese.

• Two expected wins did not materialise: brand-new control tokens (<tool_call>, <think>, <sql>, citation and field markers) are placed correctly by all arms from in-context examples (at five shots B 88%, C 75%, A 73%), and emoji prediction on SemEval-2018 task 2 (20 classes) is at chance for all three models at 2.6 to 5.7% against 5%.

• The codes retain most of the pretrained geometry: the 512 directions carry 92.3% of the embedding's variance, code agreement correlates with human similarity ratings on WordSim-353 at 0.617 against 0.614 for the float embedding, and 67% of each token's 10 nearest float neighbours are among its 10 nearest code neighbours.

Methodology in Plain English

The author builds three versions of the same model and trains each one from scratch on identical data so that any difference can be attributed to the vocabulary.

The body is Gemma 3 270M's exact architecture with random initialisation: 18 layers, width 640, a 2,048-wide gated GELU feed-forward network, 4 query and 1 key-value head of size 256 with QK-norm, five local layers per global layer, and RMSNorm.

The first arm keeps the ordinary learned table of 262,144 by 640 vectors tied to the output head. The second swaps in the frozen table: each token's code is the sign of its projection onto the top 512 principal directions of Gemma's own embedding, so the code is a string of 512 bits that never changes. Small trained linear maps move between the 512-bit code space and the 640-wide body, and a fixed per-token log-unigram prior is added to the output scores. The third arm takes the frozen table and replaces every linear layer in the body with BitNet b1.58's ternary BitLinear, using the schedule BitNet's authors published for their smallest model, with the unigram prior made trainable.

All arms see 2.05B tokens of FineWeb-Edu in sequence length 1,024, 262,144 tokens per optimiser step over 7,820 steps, drawing the same batches in the same order from the same seed. Evaluation uses validation loss on 512 held-out sequences plus zero-shot LAMBADA, HellaSwag, ARC-Easy, and PIQA via lm-evaluation-harness, with 95% bootstrap ranges over questions and paired comparisons on the same questions.

For the speed test, the author wrote a single-file AVX2 CPU kernel for the ternary arm and ran the float arm through llama.cpp on the same machine with the same thread count, then verified the kernel against the PyTorch model (identical argmax at every position on 64 validation tokens, loss 4.495 versus 4.483).

Why This Matters

Impact on research. The paper isolates a component that is usually treated as a fixed cost. Prior BitNet reporting quotes memory "non-embedding" and quantisers like GPTQ and AWQ leave embed_tokens and lm_head untouched, so the table silently dominates small-model budgets. A controlled three-arm comparison with paired error bars and a full release of per-question results gives the field a reproducible baseline for asking whether embedding tables need to be trained at all.

Real-world applications:

• On-device assistants: a 46 MB model generating 615 to 740 tokens per second in 62 MB of RAM fits comfortably on a phone or a laptop where a 471 MB working set would not.
• Domain adaptation without retraining: clinical, legal, or technical vocabularies can be added after training at a few percent quality cost, instead of being impossible without gradient updates.
• Cheap multilingual coverage: tokens never seen during training still carry meaning through their codes once the prior is corrected, which matters for long-tail languages.
• Reproducible small-model research: every checkpoint every 800 steps, the training data with checksums, and all evaluation answers are released, so a lab without a large compute budget can extend the tables rather than re-run the study.

Industry relevance. The result challenges the assumption that a learned embedding is a necessary cost of having a vocabulary. For anyone shipping sub-billion-parameter models, the trade is explicit: 0.28 nats of loss for a file roughly one-tenth the size and five to six times the generation speed, with the benchmark damage confined largely to ARC-Easy.

Future Directions

• Scale the data budget. Two billion tokens is described as Chinchilla-optimal for the 101M-parameter arms but under-trains the learned table; the author expects the loss gap at 20B tokens to be larger than 0.20 nats and the benchmark picture to possibly change with it.

• Separate the ternary arm's three variables. Arm C differs from B in body type, BitNet's schedule, and the trained prior. The prior's share was measured in a partial run stopped at step 1,833 (0.003 to 0.022 nats better than B at every matched check), but the schedule's share was not isolated.

• Bootstrap the frozen table from a model you train yourself. The codes are derived from Gemma 3 270M's embedding and inherit its knowledge and its licence terms; the untested alternative is to train a small model with a learned table once and freeze its codes for everything after.

• Extend the code space beyond text. The author teases a follow-up in which one frozen 512-bit table serves text, images, audio, and video simultaneously, with no learned embedding anywhere and the whole model still fitting on a CPU.

• Test more seeds and more hardware. The error bars are over test questions rather than training runs, and the CPU comparison covers one laptop and two different code bases (the author's kernel for C, llama.cpp for A).

Target Audience

Researchers and engineers working on small or on-device language models, particularly those interested in embedding design, extreme quantisation, ternary/BitNet architectures, and CPU inference. It is also useful for practitioners deciding whether to spend a parameter budget on a vocabulary or on the body, and for anyone who needs to extend a deployed model's vocabulary without access to training compute. Readers without background in perplexity, quantisation formats, or zero-shot benchmarks will need to pick up those concepts first.

Authors’ abstract

In a small language model the vocabulary is the single largest part. In Gemma 3 270M the embedding table holds 168M of 268M parameters, and because quantisation methods leave the embedding in 16-bit, it is also most of the file once the rest of the model is compressed. We ask what that table actually buys. We replace it with one frozen table of 512-bit codes, 16 MB, with no trainable parameters, derived once from a pretrained embedding by principal-component projection and sign binarisation, and we train the same Gemma-shaped body from scratch on the same 2.05B tokens with and without it. The learned table buys 0.20 nats of validation loss for 2.6 times the trainable parameters, but on LAMBADA, HellaSwag and PIQA the frozen-table model is better or tied; only ARC-Easy favours the learned table. With a ternary BitNet b1.58 body the whole model is a 46 MB file, still ties the float model on LAMBADA and PIQA, and on a laptop CPU generates 615 to 740 tokens per second in 62 MB of memory, five to six times the speed of the float model's 8-bit GGUF in llama.cpp at a seventh of the memory. The frozen table wins by 1.7 to 2.3 nats on rare words, accepts 10,000 new words after training with a 3% cost where the learned table collapses by 43%, and all of it is released with the weights, the data and every evaluation answer.