A Hindi Script Costs More Tokens Than the Same Script in English (2026)

A Hindi Script Costs More Tokens Than the Same Script in English (2026)

By Ashok Sachdev, Founder of JustShoot · Published 2 October 2026 · Measured 2 October 2026

Short answer: Yes, but much less than it used to. With o200k_base, the tokenizer OpenAI uses for GPT-5, GPT-4.1 and GPT-4o, the same YouTube script took 1.28–1.63x as many tokens in Devanagari Hindi as in English. Romanised Hinglish took 1.51–1.68x. On the older cl100k_base tokenizer used by GPT-4, Hindi took 4.21–5.08x.

A token is the unit an AI model reads, writes and counts against its limits. Every limit you hit in an AI tool — how long a reply can be, how much of your conversation it remembers, how fast a usage quota runs out — is counted in tokens, not words. So if the same script costs more tokens in Hindi, you hit every one of those limits sooner in Hindi.

This page measures that with a public tool you can run yourself. It gives the exact text, the tokenizer name and version, and the raw counts. If you cannot reproduce a number on this page, it is wrong, and we want to know.

Why a language costs more tokens at all

A tokenizer splits text into pieces from a fixed vocabulary it learned from its training data. Common English words are usually a single piece each, because the tokenizer saw them millions of times. Text the tokenizer saw less of gets split into smaller pieces — sometimes single characters, sometimes fragments of a character's underlying bytes. More pieces means more tokens for the same meaning.

This was measured at scale in 2023. Aleksandar Petrov, Emanuele La Malfa, Philip H.S. Torr and Adel Bibi, "Language Model Tokenizers Introduce Unfairness Between Languages", arXiv:2305.15425 (first submitted 17 May 2023, revised 20 October 2023; NeurIPS 2023), ran 2,000 sentences translated into 200 languages (FLORES-200) through many tokenizers and reported each language's token count relative to English. Their abstract reports differences "up to 15 times in some cases".

For Hindi, the extended table in the paper's appendix (page 23) gives a premium of 4.79 on cl100k_base, the tokenizer behind ChatGPT and GPT-4 at the time. On the same tokenizer it lists Bengali at 5.84, Marathi at 5.07, Tamil at 7.65 and Gujarati at 7.69. In other words, in 2023 the same sentence cost almost five times as many tokens in Hindi as in English.

A second 2023 paper reached the same conclusion from the pricing side: Orevaoghene Ahia, Sachin Kumar, Hila Gonen, Jungo Kasai, David R. Mortensen, Noah A. Smith and Yulia Tsvetkov, "Do All Languages Cost the Same? Tokenization in the Era of Commercial Language Models", arXiv:2305.13707 (23 May 2023). It found that speakers of many supported languages "are overcharged while obtaining poorer results".

Both papers measured tokenizers that are now a generation old. So we re-ran the question on the current one.

What we measured

Tool: the gpt-tokenizer npm package, version 4.0.0, installed locally on 2 October 2026. It implements OpenAI's published encodings and needs no API key; nothing on this page was sent to any model.

Encodings: o200k_base and cl100k_base. Which models use which comes from OpenAI's own open-source tiktoken library (tiktoken/model.py, last changed 17 August 2026). It maps gpt-5, gpt-4.1, gpt-4o and the o1/o3/o4-mini models to o200k_base, and gpt-4 and gpt-3.5-turbo to cl100k_base.

Input: two passages written as YouTube intros — a travel vlog (A) and a phone review (B). Each is written three times with the same meaning: English, Hindi in Devanagari, and Hinglish in Roman script. The full text of all six is printed at the bottom of this page.

What this does not cover: Google's Gemini and Anthropic's Claude tokenizers are not published as packages we can run offline, and we have no API key for either, so we did not measure them. Do not read these numbers as applying to them.

The results

Token counts, and each language's count divided by the English count for the same passage:

Passage	Version	Words	o200k_base (GPT-5, GPT-4o)	vs English	cl100k_base (GPT-4)	vs English
A — travel vlog	English	116	134	1.00x	134	1.00x
A — travel vlog	Hindi (Devanagari)	118	172	1.28x	564	4.21x
A — travel vlog	Hinglish (Roman)	118	203	1.51x	228	1.70x
B — phone review	English	103	114	1.00x	114	1.00x
B — phone review	Hindi (Devanagari)	120	186	1.63x	579	5.08x
B — phone review	Hinglish (Roman)	120	192	1.68x	217	1.90x

Three things stand out.

1. The Hindi premium fell sharply between tokenizer generations. On cl100k_base our two passages cost 4.21x and 5.08x English, which brackets the 4.79 that Petrov et al. measured on 2,000 sentences. On o200k_base the same two passages cost 1.28x and 1.63x. The tax is still there, but it is roughly a third of what it was.

2. On the current tokenizer, Hinglish is not cheaper than Hindi. On the old tokenizer, typing Hindi in Roman script was the obvious workaround — 1.70x instead of 4.21x. On o200k_base, romanised Hinglish cost slightly more than Devanagari in both passages (1.51x vs 1.28x, and 1.68x vs 1.63x). The likely reason: the newer vocabulary includes more Devanagari pieces, while romanised Hindi spellings like "dukaanein" or "khareedna" are not standard English words and still get split up. That last sentence is our reading of the counts, not a measured fact about the vocabulary.

3. The size of the premium depends on the text. Passage B has more borrowed English words written in Devanagari — फ़ोन, कैमरा, डिस्प्ले, रिव्यू — and its Hindi premium is higher, though two passages cannot prove that is the reason. Two passages are also a small sample, so treat the ranges here as an illustration, not a national average. The 2,000-sentence measurement is the paper's.

What this means for a creator using AI in Hindi

The practical effect is not the bill — most creators use a subscription, not a per-token API. It is the limits:

• Replies cut off sooner. A tool that caps a single reply at a fixed number of tokens will stop a Hindi script earlier than an English one. On our numbers, a limit that holds a full English script holds roughly 60–80% of the same script in Hindi on GPT-5's tokenizer. On a GPT-4-era tokenizer it held a fifth to a quarter.
• Long chats forget sooner. The "context window" — how much of the conversation a model can see at once — is a token count too. In Hindi it fills up faster, so the model loses your earlier instructions sooner. That is one reason a tool can drift away from your channel's voice in a long session; we covered the other reasons in why ChatGPT forgets your channel's tone.
• Usage quotas run out faster on any plan that meters tokens rather than messages.

Two things that follow from it:

Split long requests. Ask for the Hindi script in one request and the English title, description and tags in a separate one. Each request then has its whole output limit for one job, and a long script is less likely to be cut off halfway through the outro. (Separately, the independent benchmark in which AI model follows a Hindi script brief found that a Hindi request asking for English output is where models break instructions most often — a second reason to split.)

Do not switch to Roman script to save tokens. On the old tokenizer that worked. On the current one it does not, on our two passages. Write in whichever script your audience reads. If you are not sure how much Hindi versus English is actually in your script, paste it into the free Hinglish ratio checker — it reports the exact Hindi–English word split in seconds.

One thing tokens do not tell you is length on screen. Tokens are what a model counts; words are what you speak. For how many words a 10-minute Hindi video needs, use our script length guide instead — it is a different quantity.

Where JustShoot fits

JustShoot writes scripts in 11 languages, including Hindi in Devanagari and Hinglish in Roman script. It runs nine separate agents — research, script, fact-check, storyboard, SEO and the rest — so the script and the English metadata are already separate jobs rather than one long request. That design choice is the "split long requests" advice above, done for you.

Pricing is counted in scripts, not tokens, so the Hindi premium does not change what you pay: Trial ₹0 (7 days, 2 scripts lifetime, no card) · Starter ₹499/mo (3 scripts) · Creator ₹999/mo (7 scripts, most popular) · Pro ₹1999/mo (15 scripts) · Studio custom. No rollover, GST included, monthly only. Details on the pricing page.

FAQ

Why does Hindi use more tokens than English in ChatGPT? Because the tokenizer's vocabulary was learned mostly from English text, so Hindi gets split into smaller pieces. On o200k_base, the tokenizer OpenAI maps to GPT-5 and GPT-4o, our two test scripts cost 1.28x and 1.63x the English token count. On GPT-4's cl100k_base they cost 4.21x and 5.08x.

Is Hinglish cheaper than Hindi in AI tools? Not on the current OpenAI tokenizer, on our test. Romanised Hinglish cost 1.51x and 1.68x English on o200k_base, slightly more than Devanagari Hindi at 1.28x and 1.63x. On the older GPT-4 tokenizer Hinglish was much cheaper than Devanagari (1.70x vs 4.21x).

How many tokens is a Hindi YouTube script? In our test, about 1.5 tokens per word for Hindi on o200k_base (172 tokens for 118 words, and 186 for 120). A 1,500-word Hindi script would be roughly 2,200–2,300 tokens on that tokenizer. Count your own with any o200k_base tokenizer; it needs no API key.

Do Gemini and Claude have the same Hindi token premium? We did not measure them. Their tokenizers are not available as offline packages and we had no API key for either, so we have no figure to give. Assume there is some premium until a vendor publishes one.

Does writing in Hindi make AI scripts more expensive? On per-token API billing, yes, by the ratios above. On subscription tools priced per message or per script, the cost is the same but you reach length and memory limits sooner. Splitting the script and the English metadata into separate requests reduces the cut-off risk.

The exact input text

Reproduce any number above by running these six texts through gpt-tokenizer 4.0.0 (o200k_base or cl100k_base). Word counts split on whitespace.

Passage A — English

Last monsoon I took a night train from Mumbai to Goa with just one backpack and four thousand rupees. Everyone told me it was a bad idea. The rain would ruin everything, the beaches would be empty, and half the shacks would be shut. They were right about the shacks. They were wrong about everything else. In this video I will show you the exact route I took, where I stayed for under eight hundred rupees a night, and the one waterfall that is only worth visiting in July. If you have been waiting for the perfect season to travel, stay till the end, because the perfect season might be the one everybody else is avoiding.

Passage A — Hindi (Devanagari)

पिछले मानसून में मैंने सिर्फ़ एक बैकपैक और चार हज़ार रुपये लेकर मुंबई से गोवा की रात वाली ट्रेन पकड़ी। सबने कहा कि यह बुरा आइडिया है। बारिश सब कुछ बिगाड़ देगी, बीच खाली होंगे, और आधी दुकानें बंद होंगी। दुकानों के बारे में वे सही थे। बाक़ी हर चीज़ के बारे में वे गलत थे। इस वीडियो में मैं आपको वह पूरा रास्ता दिखाऊँगा जो मैंने लिया, मैं आठ सौ रुपये से कम में एक रात कहाँ रुका, और वह एक झरना जो सिर्फ़ जुलाई में देखने लायक है। अगर आप घूमने के लिए सही मौसम का इंतज़ार कर रहे हैं, तो आख़िर तक देखिए, क्योंकि सही मौसम शायद वही है जिससे बाक़ी सब बच रहे हैं।

Passage A — Hinglish (Roman)

Pichhle monsoon mein maine sirf ek backpack aur chaar hazaar rupaye lekar Mumbai se Goa ki raat waali train pakdi. Sabne kaha ki yeh bura idea hai. Baarish sab kuch bigaad degi, beach khaali honge, aur aadhi dukaanein band hongi. Dukaanon ke baare mein woh sahi the. Baaki har cheez ke baare mein woh galat the. Is video mein main aapko woh poora raasta dikhaunga jo maine liya, main aath sau rupaye se kam mein ek raat kahaan ruka, aur woh ek jharna jo sirf July mein dekhne laayak hai. Agar aap ghoomne ke liye sahi mausam ka intezaar kar rahe hain, toh aakhir tak dekhiye, kyunki sahi mausam shayad wahi hai jisse baaki sab bach rahe hain.

Passage B — English

I have used this phone for thirty days as my only camera, and I want to tell you three things the box does not. The battery lasts a full day of shooting, but only if you turn off the always-on display. The main camera is excellent in daylight and noticeably weaker indoors after sunset. And the speaker is loud enough for a small room, not for a crowded street. None of this makes it a bad phone. It makes it a phone you should buy for the right reasons. By the end of this review you will know whether those reasons are yours.

Passage B — Hindi (Devanagari)

मैंने तीस दिन तक इस फ़ोन को अपने अकेले कैमरे की तरह इस्तेमाल किया है, और मैं आपको तीन बातें बताना चाहता हूँ जो डिब्बे पर नहीं लिखी हैं। बैटरी शूटिंग का पूरा दिन चलती है, लेकिन सिर्फ़ तब जब आप ऑलवेज़-ऑन डिस्प्ले बंद कर दें। मेन कैमरा दिन की रोशनी में शानदार है और सूरज ढलने के बाद घर के अंदर साफ़ तौर पर कमज़ोर है। और स्पीकर एक छोटे कमरे के लिए काफ़ी तेज़ है, भीड़ वाली सड़क के लिए नहीं। इनमें से कोई भी बात इसे बुरा फ़ोन नहीं बनाती। यह इसे ऐसा फ़ोन बनाती है जिसे आपको सही वजहों से ख़रीदना चाहिए। इस रिव्यू के आख़िर तक आप जान जाएँगे कि क्या वे वजहें आपकी हैं।

Passage B — Hinglish (Roman)

Maine tees din tak is phone ko apne akele camera ki tarah use kiya hai, aur main aapko teen baatein batana chahta hoon jo dibbe par nahi likhi hain. Battery shooting ka poora din chalti hai, lekin sirf tab jab aap always-on display band kar dein. Main camera din ki roshni mein shaandaar hai aur sooraj dhalne ke baad ghar ke andar saaf taur par kamzor hai. Aur speaker ek chhote kamre ke liye kaafi tez hai, bheed waali sadak ke liye nahi. Inmein se koi bhi baat ise bura phone nahi banati. Yeh ise aisa phone banati hai jise aapko sahi wajahon se khareedna chahiye. Is review ke aakhir tak aap jaan jaayenge ki kya woh wajahein aapki hain.

---

Sources. Petrov, La Malfa, Torr and Bibi, Language Model Tokenizers Introduce Unfairness Between Languages, arXiv:2305.15425v2, 20 October 2023 (NeurIPS 2023), appendix table, page 23. Ahia, Kumar, Gonen, Kasai, Mortensen, Smith and Tsvetkov, Do All Languages Cost the Same? Tokenization in the Era of Commercial Language Models, arXiv:2305.13707, 23 May 2023. OpenAI, tiktoken (tiktoken/model.py, last commit 17 August 2026), for the model-to-encoding map. Token counts: gpt-tokenizer 4.0.0, run locally on 2 October 2026; the passages were written for this test and are printed above. The ratios are my own division of those counts.