The German Token Tax: Why Per-Million-Token Prices Compare the Wrong Thing

Every model-selection meeting I sit in in Germany converges on the same spreadsheet: dollars per million tokens, one row per model, sorted ascending. The argument is then about the distance between $2.00 and $2.50.

The spreadsheet is measuring the wrong unit. A token is not a unit of work, and how much work one holds depends on the language you write in and on which tokenizer release you are talking to. On the officially parallel German and English texts of the EU AI Act, the German version costs 28.4% more tokens on OpenAI’s current tokenizer. That is a bigger number than most of the price differences being argued over.

This arithmetic — not sovereignty, not jurisdiction — is the unglamorous case for European tokenizers. It is also a weaker case than it looks. I measured two of them. One works. The other makes German more expensive than OpenAI’s tokenizer does.

What I measured

Both language versions of Regulation (EU) 2024/1689 (the AI Act, CELEX 32024R1689), fetched from EUR-Lex and stripped to running text. The pair is officially parallel rather than machine-translated: Regulation No 1 of 1958, still the Union’s language regulation, requires at Art. 4 that “Regulations and other documents of general application shall be drafted in the official languages”, and at Art. 5 that the Official Journal “shall be published in the official languages”.

After extraction:

EN   90,482 words   583,502 characters
DE   82,404 words   649,328 characters

The first surprise is there. German uses 8.9% fewer words and 11.3% more characters — compounding does that, since Konformitätsbewertungsstelle is one word where English spends three. So “tokens per 1,000 words”, the metric most write-ups reach for, inflates the German figure by about a tenth before any tokenizer is involved. Everything below is per document, which is what the invoice measures.

Five tokenizers, no special tokens added, whole document in both languages:

Re-running the whole thing on the GDPR (32016R0679) gives 1.288 against the AI Act’s 1.284 on o200k_base. The ratio is a property of the language pair, not of the document.

Where the surcharge comes from

It decomposes cleanly into a length term and an efficiency term. Characters per token, same act:

                        chars/token EN   chars/token DE   efficiency   length   total
o200k_base                        5.26             4.56        1.154    1.113   1.284
cl100k_base                       5.24             3.51        1.495    1.113   1.664
Mistral Tekken v7                 5.01             4.19        1.197    1.113   1.332
EuroLLM-22B-Instruct-2512         4.67             4.25        1.098    1.113   1.222
Teuken-7B-instruct-v0.6           4.82             4.86        0.990    1.113   1.102

The length term is fixed at 1.113 for every row: German is simply longer on the page, and no tokenizer changes that. The efficiency term is the part that is anybody’s fault, and Teuken-7B is the only tokenizer here that has actually removed it — 0.990 means German gets marginally more characters per token than English does.

You can see it in one word. Risikomanagementsystem, which the AI Act uses constantly:

o200k     5   R · isik · oman · agement · system
cl100k    6   R · is · ik · oman · agements · ystem
EuroLLM   5   ▁Ris · ik · oman · agements · ystem
Teuken    3   ▁Risiko · management · system

English “risk management system” is three tokens on all four. Überwachungsbehörde is five tokens on o200k_base (Über · wach · ungs · beh · örde) and two on Teuken.

Two results that do not flatter the argument. Grundrechte is two tokens on o200k_base while “fundamental rights” is three — German is cheaper there, and the surcharge is an aggregate, not a rule. And the folk explanation is wrong: ä, ö, ü and ß are each two bytes in UTF-8 but each a single token in o200k_base. Umlauts are not the problem. Morphology is.

What it costs in euros

One pass over the act, input tokens only, at published list rates converted at the ECB euro reference rate of 1.1567 USD/EUR (14 August 2026):

model                 $/M in    EUR (EN)   EUR (DE)   surcharge
gpt-5.6-terra           2.00      0.1917     0.2462     +0.0545
gpt-5.4                 2.50      0.2396     0.3078     +0.0682
Mistral Medium 3.5      1.50      0.1509     0.2010     +0.0501

Now put those two OpenAI rows against each other, which is the comparison the meeting is actually having. Moving from gpt-5.4 to gpt-5.6-terra is a 20% list-price cut. Running the German document on the cheaper model costs €0.2462; running the English document on the dearer one costs €0.2396. Switching model does not recover what switching language costs. The decision the spreadsheet is optimising is smaller than the variable it does not contain.

The same trap with no German in it

The clearest statement of the problem comes from a vendor, about English. Anthropic’s pricing documentation notes that Claude 4.7 and later “use a newer tokenizer that contributes to their improved performance… This tokenizer produces approximately 30% more tokens for the same text.”

Claude Opus 4.6 and Claude Opus 4.7 both list at $5/MTok input. Same headline price, roughly 30% more tokens per document: a price rise of about a third that appears nowhere in the price column. I cannot verify the 30% myself — Anthropic does not publish its tokenizer, so this is their figure, not my measurement. That it comes from the vendor is rather the point.

Where the European argument breaks

I expected the European tokenizers to win and they mostly did not.

EuroLLM-22B-Instruct-2512 has the second-best German/English ratio at 1.222, and that ratio is misleading. In absolute terms it needs 152,875 tokens for the German act against o200k_base’s 142,410 — 7.3% more. Its ratio looks good because its English is 12.8% worse, not because its German is better. A ratio is not a bill.

Teuken-7B genuinely wins on German: 133,476 tokens, 6.3% below o200k_base, on a 250,680-entry vocabulary covering the 24 official EU languages. It pays for it with 9.3% more tokens on English. For a German-language workload that is the right trade; for the mixed German-and-English reality of most enterprises it is a wash. The v0.6 instruct model is CC-BY-NC-4.0 — the Apache-2.0 commercial-v0.4 release carries a tokenizer that produced byte-identical counts in my runs, so the licensing constraint is on the weights, not the arithmetic.

And fewer tokens is not the objective. Ali et al. trained 24 models to test exactly this and found that “the common tokenizer evaluation metrics fertility and parity are not always predictive of model downstream performance”. Their stronger finding cuts the other way, at training time: English-centric tokenizers on multilingual models cause “additional training costs of up to 68%”.

Three things I could not test. Llama 3.1 and Gemma 3 are gated on Hugging Face — OSError: You are trying to access a gated repo — and accepting a licence on Enver’s account was outside what this run may do. utter-project/EuroLLM-9B is gated too, while the 22B sits open under Apache-2.0. And no commercial provider’s hosted German throughput was measured; this is tokenizer arithmetic, not a benchmark.

What I would actually do

1. Stop comparing $/MTok. Compare euros per representative document. Tokenise fifty of your own real files in both languages against each candidate’s actual tokenizer and put that column in the spreadsheet instead. It is an afternoon.
2. Treat a tokenizer change as a price change, including within one vendor’s own lineup, and re-run the arithmetic when a model family bumps its tokenizer.
3. Consider a European tokenizer where the workload is genuinely German-dominant — and measure it, because one of the two I tested would have cost more.

Bottom line

The German token tax is real, reproducible at 1.284 across two independent statutes, and smaller than the folklore: not the 15× that separates the extreme language pairs in the literature, and not caused by umlauts. Roughly two-fifths of it is German being longer, which nobody can fix, and three-fifths is tokenizer fit, which Teuken has already fixed and EuroLLM has not.

That is the whole European tokenizer argument, and it is worth making on exactly these terms. It is a procurement argument about a mismeasured unit, and it survives contact with a spreadsheet — which is more than the sovereignty version manages.

The cheapest model on the list is not the cheapest model for your documents, and the list never said it was.

---

Sources

• EUR-Lex: Regulation (EU) 2024/1689 (DE) and Regulation (EU) 2016/679 (DE) — the two parallel corpora.
• Regulation No 1 of 1958, consolidated — Art. 1, 4, 5, why those pairs are official translations.
• openai/tiktoken — o200k_base and cl100k_base, run at version 0.13.0.
• Tokenizers as published: utter-project/EuroLLM-22B-Instruct-2512, openGPT-X/Teuken-7B-instruct-v0.6 and commercial-v0.4, mistralai/Mistral-Small-3.2-24B-Instruct-2506.
• Anthropic pricing documentation — model list prices and the “approximately 30% more tokens” tokenizer note.
• OpenAI API pricing and Mistral API pricing — list rates used in the euro table.
• ECB euro reference exchange rate, USD — 1.1567, 14 August 2026.
• Petrov et al., Language Model Tokenizers Introduce Unfairness Between Languages (NeurIPS 2023) — differences “up to 15 times in some cases”.
• Ali et al., Tokenizer Choice For LLM Training: Negligible or Crucial? — fertility and parity “not always predictive”, and “up to 68%” additional training cost.

Measured on 15 August 2026 with tiktoken 0.13.0, transformers 5.15.0 and sentencepiece, against the tokenizer files published in the repositories above; prices and the ECB rate are list values on that date and will age. Engineering write-up, not procurement advice.