Token
By HubAI editorial team · Source checked: 24 October 2026What does it mean?
A unit a model uses to process text or other supported input. In text, a token can be a word, part of a word or punctuation. It is not a fixed number of characters.
Picture this
A long surname may split into several tokens while a common short word uses one. That is why counting words alone does not give an exact API bill.
Where you’ll come across it
Context limits, usage reporting and many API prices. Use the provider’s actual tokenizer or usage records when estimating a specific workload.
What it doesn’t mean
Token counts vary by model and language. A model request, a user message and a token are different units; never compare them as though they were interchangeable.
Words and tokens are not the same
A short exercise · use fictional or non-sensitive material.
- Write two short pieces of text: one with ordinary words and one with long names or mixed languages.
- If your chosen provider offers a tokenizer, compare its counts for that exact model. Do not use confidential text.
- Estimate a workload using those measured counts, including output, rather than assuming a fixed word-to-token ratio.
Have you named the model and counted input and output separately?
Did that make it clearer?
Likes are a private on-page choice, not a public rating. For a correction or complaint, send the pre-filled form to our editorial inbox. Nothing is sent until you submit it.Check the source
Written and explained by HubAI. The reference below lets you go deeper; it is not an endorsement of a particular product.
Technical reference: huggingface.co ↗