Transformer Intuition, Tokens, and Context Windows
Why order matters to language models, and how they actually 'see' text.
What you'll learn
- Explain what a token is and why LLMs don't operate directly on characters or whole words
- Explain what a context window limits
- Describe attention at an intuitive level (which words 'matter' to which other words)
Prerequisites
Explanation
Modern language models are built on an architecture called the transformer, introduced in 2017. Its core idea is attention: when processing one word, the model looks at every other word in the input and decides how much each one should influence its understanding of the current word.
Consider "The trophy didn't fit in the suitcase because it was too big." What does "it" refer to — the trophy or the suitcase? Humans resolve this instantly using context. Attention gives the model a mechanism to do something analogous: for the token "it," it computes a relevance score against every other token, and words like "trophy" and "suitcase" get more weight than filler words like "the." This happens in parallel across the whole input, layered many times, which is part of why transformers can be trained efficiently on huge amounts of text — unlike older architectures that processed text strictly one word at a time.
Before any of that happens, text has to be converted into numbers. LLMs don't operate on raw characters or whole words directly — they operate on tokens, chunks of text produced by a tokenizer that often split words into sub-word pieces. "unbelievable" might become tokens like "un", "believ", "able". Common short words are usually a single token; rare or made-up words get split into more pieces. This matters practically: providers price API usage per token, and every model has a maximum number of tokens it can process at once — its context window. If your input (plus the model's growing response) exceeds that window, older content has to be dropped or summarized, which is a real constraint you'll design around once you build retrieval systems later in this track.
None of this means the model "understands" text the way you do — it means it has learned, from enormous amounts of text, a very effective statistical mechanism for predicting plausible continuations, guided by which parts of the input are most relevant to each other.
Attention, informally
For the token 'it', attention computes a relevance score against every other token in the sentence, then blends their information weighted by that relevance — so 'trophy' and 'suitcase' matter more to resolving 'it' than 'the' or 'was'.
Example
A hand-written stand-in for both tokenization and a simplified attention-style relevance score (real tokenizers and attention are learned/statistical, not keyword-based like this).
function mockTokenize(text) {
// Real tokenizers use learned sub-word vocabularies; this is a simplified stand-in.
return text.toLowerCase().split(/\s+/);
}
function mockRelevance(word, otherWord) {
// A toy stand-in for attention: real attention scores come from learned
// vector comparisons, not string length similarity.
if (word === otherWord) return 0;
const shared = [...word].filter((ch) => otherWord.includes(ch)).length;
return shared / Math.max(word.length, otherWord.length);
}
const tokens = mockTokenize("the trophy did not fit in the suitcase");
console.log(tokens);
console.log(mockRelevance("trophy", "suitcase"));Try it yourself
Tokenize a sentence of your own and inspect the resulting tokens.
Code editor. Press Escape then Tab to leave the editor if keyboard focus becomes trapped. Press Control+Shift+M inside the editor to toggle Tab-key focus trapping.
Guided exercise
Guided exercise
Write a function `countTokens(text)` that returns the number of whitespace-separated tokens in text (a simplified stand-in for real sub-word tokenization).
Checks: Counts two simple words correctly · plus 1 hidden check
Code editor. Press Escape then Tab to leave the editor if keyboard focus becomes trapped. Press Control+Shift+M inside the editor to toggle Tab-key focus trapping.
Stuck? Get a hint.
Independent exercise
Independent exercise
Write a function `fitsInContextWindow(tokenCount, maxTokens, reservedForResponse)` that returns true only if tokenCount plus reservedForResponse is less than or equal to maxTokens.
Checks: Comfortably-sized input fits · Oversized input does not fit · plus 1 hidden check
Code editor. Press Escape then Tab to leave the editor if keyboard focus becomes trapped. Press Control+Shift+M inside the editor to toggle Tab-key focus trapping.
Stuck? Get a hint.
Common mistakes
- Assuming a 'word' and a 'token' are the same thing — one word often becomes multiple tokens.
- Forgetting to budget context window space for the model's response, not just your input.
- Believing attention means the model 'reads' text sequentially like a person — it processes relationships across the whole input at once.
Knowledge check
Takeaway
Text becomes tokens with a hard budget (the context window), and attention is how a transformer decides which tokens matter to which.
Summary
Transformers use attention to weigh how relevant every token is to every other token, enabling efficient parallel processing of context. Text is converted into tokens (often sub-word pieces) before a model sees it, and every model has a fixed context window limiting combined input and output tokens.
References
Your notes
Notes save automatically.
Finished this lesson?
Mark it complete to track your progress and schedule a future review.
AI tutor
The optional AI tutor isn't enabled in this deployment. All lessons, exercises, quizzes, and search work fully without it.