New Own your agent: a live workshop on running your own AI agentOwn your agent Join the waitlist

The tokenizer tax

Talking to an AI model in Icelandic costs more than saying the same thing in English. The price list is the same. The difference is in how the models read.

Models don’t read words. A tokenizer first cuts the text into pieces called tokens, and you pay per token. Tokenizers learn their pieces from the text they see most, which is mostly English. Common English words become one token each. Words in languages with less text online get chopped into fragments.

English 13 tokens

PeopleshouldbenefitfromAI,regardlessofthelanguagetheyspeak.

Icelandic 28 tokens

Fólkáaðnjótagóðsafgervigreind,óháðþvíhvaðatungumálþaðtalar.
The same sentence, cut into tokens by OpenAI’s open o200k tokenizer. Each box is one token, and you pay per box.

Rest of World wrote about this recently, based on a study by CLEAR that calls it the tokenizer tax. Across 200 languages, the worst-served language needed up to 17 times more tokens than the best-served one for the same content, depending on the model. Among the 20 biggest languages on the web, the gap was mostly between 1.5 and 3.1 times.

English13
Malayalam26
Icelandic28
Amharic78
Tokens needed for that one sentence. The English, Malayalam and Amharic counts are from the CLEAR study (GPT-5 tokenizer). The Icelandic count is my own translation, measured with the o200k tokenizer, which gives the same 13 for the English.

Icelandic isn’t in the study, so I measured it. I ran the English and Icelandic versions of my last two notes through the same tokenizer. The texts say the same thing and are almost the same length in characters, but the Icelandic needs about 1.5 times the tokens: 434 against 297 for one note, 253 against 163 for the other.

So Icelandic gets off lightly next to Amharic, thanks to the Latin alphabet. It still counts in three ways:

  • Cost. About 50% more for every prompt and every answer.
  • Memory. The same document or conversation history fills the model’s context window faster.
  • Speed. More tokens in the answer means a longer wait.

This is one tokenizer. Others split text differently, and token prices are falling fast. What I take from it for my own work: chatting in Icelandic is fine, but the instructions an agent reads on every single run are cheaper in English.