Large Language Models

Self-attention treats its input as a set — reorder the tokens and each one gets the same representation back, just moved. Order has to be injected separately, or "the auditor approved the invoice" and "the invoice approved the auditor" are indistinguishable.

Each token is projected into three vectors: a query saying what it is looking for, a key advertising what it offers, and a value carrying the content it contributes. Query-key dot products score matches, and those scores weight a sum of values.

No — FlashAttention computes exactly the same attention, bit-for-bit equivalent up to floating-point reordering. It is faster because it tiles the computation in fast on-chip memory and never writes the full score matrix out to the GPU's main memory, so it moves far less data.

Multi-head attention gives every query head its own key and value projections. Multi-query attention makes all query heads share a single key/value head. Grouped-query attention sits between them: query heads are split into groups, and each group shares one key/value head.

Total parameters size memory. Every expert's weights must be resident because any token may route to any expert. Active parameters size compute — the arithmetic and latency per token. A sparse model is therefore memory-hungry like a huge model but computes like a small one.

Run the target model's own tokenizer over the fully assembled request, or call a provider token-counting endpoint when no offline tokenizer exists. Character heuristics such as roughly four characters per token are planning guardrails, not exact counts.

Subword tokenization keeps a fixed vocabulary that can still encode any input. Word-level vocabularies fail on words they never saw; character-level ones make sequences several times longer. Subword pieces keep common words whole and split rare ones.

The model never sees characters. Text arrives as subword token IDs, so spelling is a memorized association rather than something readable, and long numbers split into chunks whose boundaries differ between similar-looking values.

BPE starts with every word split into single symbols, counts every adjacent symbol pair across the corpus, merges the single most frequent pair into a new token, and repeats until the vocabulary target is reached. The ordered list of merges is the tokenizer.

Chat models are fed one rendered sequence, not a list of strings. A template wraps every message in role and turn-delimiter tokens and prepends the system prompt and tool schemas, all billed as input and invisible to a plain character count.

A base checkpoint is trained only to continue documents, so a question is most plausibly followed by more document text rather than an answer. Answering, and stopping when finished, come from the post-training stages that produce an instruct checkpoint.

A knowledge cutoff is the point after which no training text was collected. It is fuzzy because crawls trail real events, coverage of recent months is thin, later training stages can add newer data, and the model has no reliable sense of its own horizon.

Next-token cross-entropy maximises the probability the model assigns to each real next token in the corpus. It optimises fit to the data distribution, including that data's errors and style, never truthfulness, helpfulness or task success.

Choose DPO when good preference pairs already exist and the goal is bounded - tone, format, a known bad habit. It needs no sampling loop, no reward model and no rollout infrastructure. Online RL wins when the reward is verifiable or the model must be pushed well past its current behaviour.

PPO-style RLHF trains a second network to predict expected return as a baseline. GRPO samples a group of responses per prompt instead and uses the group's own mean reward as that baseline, removing a whole model from memory and from the failure surface.

Temperature divides the model's raw scores (logits) before they are turned into probabilities. Below 1 it sharpens the distribution toward the top-scoring tokens; above 1 it flattens it, so rarer tokens get picked. At 0 it collapses to always taking the highest-scoring token.

Time-to-first-token is the wait from sending a request until the first output token arrives, and it grows with prompt length. Inter-token latency is the gap between successive tokens after that, and it sets how fast the answer streams.

Top-k keeps a fixed number of highest-probability tokens. Top-p (nucleus) keeps the smallest set whose probabilities sum past a threshold, so the set size shrinks when the model is confident. Min-p keeps every token above a fraction of the top token's probability.

Under a causal mask, a past token's key and value vectors never change as new tokens arrive, so every later step reuses them. That token's query was consumed once, to produce its own output, and is never read again.

Prefill pushes every prompt token through the model in one pass, so each weight read is amortized over many tokens and the math units saturate. Decode produces one token per pass, re-reading all weights and the cache for that single token, so memory bandwidth is the ceiling.

Rescale the model's rotary position encodings so 128K positions fall inside the range it was trained on — uniform position interpolation, or frequency-aware variants such as NTK-aware scaling, YaRN and LongRoPE — then continue training briefly on long sequences so the model adapts.

Advertised length is the maximum input the model accepts without erroring; effective length is how far accuracy actually holds. Attention spreads thinner, long-range dependencies are rare in training data, and more text means more plausible-but-wrong material to attend to.

Models attend most reliably to the beginning and end of a long prompt and least reliably to its middle, producing a U-shaped recall curve. Put instructions and the most critical evidence at the edges, not buried mid-context.

Attention distributions dump surplus probability mass onto the first few tokens regardless of their meaning. Evict those tokens as the window slides and the mass redistributes onto real content, destabilising the model. Pinning them keeps generation coherent indefinitely.

Extension changes the model everywhere, not only past the old limit. Rescaled positions blunt fine-grained local distance signals, and the continued training on long data shifts the model away from short-prompt behaviour, so short-input accuracy typically drops a few points.

Hallucination is a language model stating false or unsupported claims in the same fluent, confident register as correct ones. A prompt instruction cannot fix it because the model has no internal signal separating what it reliably knows from what it is inventing.

In-context learning changes only the next-token probabilities for that one request. Examples and instructions in the prompt condition the output distribution during the forward pass; the parameters stay frozen, so nothing carries over to the next call.

A factual hallucination is wrong about the world; a faithfulness hallucination contradicts or exceeds the source text the model was given, even if the claim happens to be true. The first is fixed by supplying sources, the second by constraining and checking generation against them.

An effort level sets a policy — think shallowly, moderately or deeply — and lets the model decide per request how far to go. A fixed token budget makes you guess a number in advance for a difficulty you cannot see yet.

Training rewarded only whether the final answer checked out — an exact string match, a passing test suite, a proof checker. With the outcome graded and the path free, models learned to spend tokens exploring, verifying and backtracking, because that reliably produced correct answers.

Multiply parameters by bytes per parameter. A 70-billion-parameter dense model needs roughly 140 GB of weights at bf16 (2 bytes), 70 GB at int8 (1 byte) and 35 GB at int4 (half a byte) — before KV cache and activations.

Post-training quantization rounds an already-trained checkpoint, using a short calibration pass to pick scales. Quantization-aware training simulates that rounding inside a training or fine-tuning run so the weights adapt to it: far more expensive, and worth it mainly at 4 bits and below.

Both formats use 16 bits, but bf16 spends more of them on the exponent and fewer on the mantissa. That gives bf16 the same dynamic range as fp32, so large activations and tiny gradients neither overflow nor vanish — at the cost of coarser precision.

Perplexity averages next-token loss over a whole corpus, so damage concentrated on a few decisive tokens disappears into the mean. Task accuracy, structured-output validity and per-slice scores expose quantization damage that a flat perplexity curve hides.

The K marks llama.cpp's k-quant family, where weights sit in super-blocks whose per-block scales and minimums are themselves stored in low precision. The trailing S or M says how many of the most sensitive tensors are promoted to a higher-bit k-quant; the 4 is the base width.

Chinchilla showed that for a fixed training-compute budget, parameters and training tokens should grow together — roughly 20 tokens per parameter — rather than spending almost the whole budget on more parameters, as earlier Kaplan-era guidance implied.

Because inference, not training, is the bill. Over-training a small model costs more once, but every request afterwards runs on fewer parameters — cheaper, faster and denser to batch across hundreds of millions of monthly calls.

The data wall is the point where high-quality text runs out before compute does, so extra FLOPs no longer buy extra fresh tokens. The remaining levers are repeating data across epochs, generating synthetic data, and pulling in other modalities.

The student learns from a strong teacher's full output distribution and worked outputs rather than from raw one-hot next tokens, so each training example carries far more signal. Part of the teacher's compute is effectively transferred into the smaller model.

Treat them as three curves with different economics. Pretraining raises the base capability, reinforcement learning after it sharpens behaviour on checkable tasks, and test-time compute buys accuracy on the hard request — but you pay that one on every call, forever.