Token

A token is a small chunk of text that a language model reads or generates, usually a word or part of a word. Think of it as the basic unit of currency that AI models use to process and produce language. When you send a prompt to a tool like ChatGPT, it doesn't read whole sentences at once. It breaks everything down into tokens first. A single word like 'apple' is one token. A longer or more complex word like 'unbelievable' might be split into three: 'un', 'believ', and 'able'. Even punctuation marks and spaces can be their own tokens. This system helps the model handle a massive vocabulary efficiently, because it can build almost any word from a smaller set of known pieces. The way text is broken down depends on the model's specific tokenizer, which is a separate program trained to find the most efficient way to split text for that particular AI. For example, when you use a model through an API, you're often billed based on the total number of tokens you send in your prompt and receive in the response. A 500-word article might be 700 tokens. I've found that this is often confused with a 'word,' but the difference is crucial for understanding costs and technical limits. A model's context window, which is its short-term memory, is also measured in tokens, not words. This matters because a limit of 8,000 tokens might only let you paste in about 6,000 words of a long document, leaving the rest for the model's reply. Understanding tokens helps you budget your usage, avoid hitting frustrating length limits, and grasp why some words are harder for an AI to handle than others.