How do Llama 2, Llama 3 and Llama 3.1 differ in sizes and context length?

## Why the generation matters more than the number People say "we're running Llama 70B" as if that identified a model. It does not: a Llama 2 70B and a Llama 3.3 70B have the same parameter count and almost nothing else in common — different tokenizer, different context length, roughly an order of magnitude more training data, and very different instruction-following quality. Interviewers ask this question to check that you track the *generation*, because that is what determines context budget, tokenizer economics, and whether a benchmark number you read still applies. ## Llama 2 (2023) Sizes: 7B, 13B, 70B, each in a base and a chat-tuned variant. (A 34B was described in the paper but never released.) Context window: 4,096 tokens. Tokenizer: SentencePiece BPE with a 32,000-token vocabulary. Attention: standard multi-head attention on 7B and 13B; only the 70B used grouped-query attention. Trained on roughly 2 trillion tokens. The 4K window is the fact that dates this generation hardest — it is too small for document QA or long agent transcripts without chunking, and it is the reason most Llama 2 deployments have been retired. ## Llama 3 (2024) Sizes: 8B and 70B. The 13B tier disappeared; Meta's line since has been that a well-trained 8B covers the small slot and 70B covers the mid slot. Context: 8,192 tokens. Tokenizer: a tiktoken-style BPE with a 128,256-entry vocabulary — about four times larger, which compresses text into fewer tokens. That matters commercially even when you self-host, because fewer tokens per document means less KV cache, faster prefill, and more real text inside a fixed window. Attention: grouped-query attention at *every* size, including 8B, with 8 key/value heads. Training data grew to 15 trillion-plus tokens, which is the main reason an 8B Llama 3 outperforms a 13B Llama 2 on most tasks. ## Llama 3.1 (2024) The generation that introduced two headline facts. First, **405B**: a dense model, the largest openly released Llama, aimed at frontier-class quality and at generating synthetic data to train smaller models. Second, **128K context on every size** — 8B, 70B and 405B all advertise 131,072 positions, achieved by scaling the rotary position embeddings and then continuing pretraining on long sequences in stages. Architecture is otherwise a continuation of Llama 3: dense transformer, GQA, no mixture-of-experts. ## The point releases - **Llama 3.2** added a small tier — 1B and 3B text-only models built by pruning and distilling from the larger Llama 3.1 models, intended for phones and edge devices — and a vision tier, 11B and 90B, which attach an image encoder to a Llama 3.1 text backbone. - **Llama 3.3** shipped exactly one model: an instruction-tuned 70B, text-only, 128K context, whose reported quality lands close to 3.1 405B on many benchmarks. For most teams this is the release that removed the reason to run 405B at all. - The **Llama 4** generation (2025) changed the architectural family, moving the flagships to a mixture-of-experts design where only a fraction of the total parameters is active per token. If you cite Llama 4 numbers, be explicit that total and active parameter counts are different figures — a mistake interviewers listen for. ## How to reason about the menu Think in slots rather than in individual model names, because the names churn: 1. **Edge/on-device (1B–3B).** Runs on a laptop or phone; good for classification, extraction, rewriting, routing. Do not expect multi-step reasoning. 2. **Small server (8B).** The high-throughput workhorse: fits comfortably on a single modern accelerator at bf16, cheap to fine-tune with adapters, good enough for RAG answering and structured extraction. 3. **Mid (70B).** The default when quality matters. Needs multi-GPU or aggressive compression to serve at bf16, since weights alone are around 140 GB at two bytes per parameter. 4. **Flagship (405B and the MoE successors).** Justified by evaluation, not ambition. Frequently used offline — for synthetic data generation and as a judge — rather than on the hot path. ## What to actually say in an interview Give the shape of the progression: sizes narrowed and then extended at both ends; context went 4K → 8K → 128K; the tokenizer changed once, at Llama 3, and that change is why token counts differ between generations for identical text. Then say which generation you deployed and why, and what forced the choice — context length, GPU budget, or an eval you ran. Reciting release dates without that judgement reads as memorisation.