Meta Llama

Llama weights are downloadable and usable commercially, but the Llama Community License imposes field-of-use limits through an acceptable-use policy and singles out very large operators for extra permission. Those restrictions fail the Open Source Definition, so the accurate term is open-weight.

Weights cost bytes-per-parameter times parameter count: about 140 GB for 70B at FP16, about 70 GB at 8-bit, and roughly 35-40 GB at 4-bit once block scales are counted. KV cache and activations are extra and are not shrunk by weight quantization.

Ollama wraps llama.cpp for local, single-user use: one command pulls a quantized model and runs it on a laptop, CPU included. vLLM is a GPU server built for concurrent production traffic, using continuous batching and a paged KV cache.

Llama 2 wraps each user turn in [INST] ... [/INST] plain-text markers and nests the system prompt inside the first one with <<SYS>>. Llama 3 drops both, using real special tokens: a role header per turn, closed by <|eot_id|>.

Rank r sets the width of LoRA's low-rank update matrices, fixing adapter capacity and the trainable-parameter count. lora_alpha scales that update — PEFT applies it as alpha/r — so raising alpha strengthens the adapter's effect without adding parameters.

QLoRA loads the frozen Llama base in 4-bit NF4 instead of 16-bit, then trains ordinary LoRA adapters in bfloat16 on top. Weights are dequantized block by block during the forward and backward passes, cutting base-model memory roughly fourfold.

Meta's Llama Community License grants free commercial use unless the licensee's products had more than 700 million monthly active users in the calendar month before that Llama version's release date. Above that line you must request a separate licence from Meta, which Meta may refuse.

Llama 2 shipped 7B, 13B and 70B with a 4K context. Llama 3 narrowed to 8B and 70B at 8K context with a much larger tokenizer vocabulary. Llama 3.1 kept those sizes, added a 405B flagship, and extended all of them to 128K context.

Grouped-query attention lets several query heads share one key/value head, so the KV cache shrinks by that ratio. Llama 3 uses 8 key/value heads at every size, cutting per-token cache memory about fourfold on 8B and eightfold on 70B, which is what makes long contexts and large batches affordable.

Q4 is the nominal bits per weight, _K marks llama.cpp's k-quant block format with per-block scales, and _S/_M/_L choose how much extra precision goes to the most sensitive tensors. Q4_K_M averages roughly 4.8 bits per weight.

Continuous batching schedules per decode iteration instead of per batch: a finished sequence leaves the running batch immediately and a queued request takes its place at the next step, so the GPU never idles waiting for the slowest generation in a fixed batch.

Because the OpenAI chat-completions shape became the de facto wire protocol: exposing /v1/chat/completions lets existing SDKs, gateways and frameworks talk to a local model by changing only the base URL. Compatibility covers the common fields, not every parameter or feature.

Fine-tuning teaches form — output shape, tone, task convention, refusal behaviour. It is a poor way to install facts that change, because updating them means retraining. Retrieval handles changing knowledge; fine-tuning handles how the answer is produced.

Llama 3.2 shipped two different tiers. The 1B and 3B are small text-only models built for on-device and edge use, produced by pruning and distilling larger Llama 3.1 checkpoints. The 11B and 90B are vision models that accept images alongside text.

Llama 3 Instruct closes each turn with <|eot_id|>; the base end-of-sequence token <|end_of_text|> almost never appears in chat output. Stop only on the base EOS and the model runs past its answer and writes the next user turn itself.

Merge when one adapter will serve all traffic and you want zero adapter overhead — merging folds the low-rank update into the weights, producing a plain checkpoint. Keep it unmerged when you need to hot-swap several adapters over one shared base.

Redistribution under the Llama Community License requires shipping a copy of the agreement, a NOTICE file with Meta's copyright text, prominent "Built with Llama" attribution on any product using it, and a derivative model name that begins with "Llama".

bitsandbytes quantizes weights to NF4 in memory as the checkpoint loads, with no calibration data and no separate build artifact. You give up serving throughput — its dequantize-and-matmul path is generally slower than tuned GPTQ or AWQ kernels — and you re-pay the quantization cost on every load.

Both are calibration-based post-training weight quantizers. GPTQ quantizes column by column and adjusts the not-yet-quantized weights to cancel the error it just introduced. AWQ instead measures activations, finds the few salient weight channels, and rescales them so rounding hurts them less.

It stores each sequence's key/value cache in small fixed-size blocks tracked by a block table instead of one contiguous reservation sized for the maximum output length. That removes the huge reserved-but-unused waste, so far more sequences fit in the same VRAM.

The rendered chat template already contains the model's beginning-of-sequence token. Tokenizing that string again with default settings prepends a second one. Pass add_special_tokens=False when encoding pre-rendered text, or let the template tokenize directly.

DPO trains directly on chosen/rejected preference pairs with a classification-style loss against a frozen reference model. PPO-based RLHF first trains a separate reward model, then optimizes the policy by sampling completions and scoring them — far more moving parts.

Training stores far more than weights: a gradient per parameter, two AdamW moment tensors, usually an fp32 master copy, plus activations. That is roughly 16 bytes per parameter, so an 8B model needs well over 100 GB before activation memory is counted.

The Acceptable Use Policy is incorporated into the Llama Community License and bars categories such as illegal activity, weapons development, malware, critical-infrastructure interference, exploitation of minors, and deceptive impersonation. Breach is a licence breach: Meta may terminate, after which you must stop using and delete the materials.

Llama 2 and Llama 3 forbade using the models' outputs to improve any other large language model. The Llama 3.1 Community License dropped that prohibition and expressly permits using outputs to train other models, provided any resulting distributed model's name begins with "Llama".

Llama 3.1 rescales the rotary position embeddings so positions past the original 8,192 fall back into a range the model has seen, then continues pretraining on progressively longer sequences up to 131,072 tokens. The config records this as a rope_scaling block with rope_type llama3 and factor 8.

Compare the quantized build against the same unquantized weights on your own task, not just on perplexity. Perplexity averages over tokens and hides narrow failures, so add task evals for structured output, tool-call validity, code and long context, plus a side-by-side sample review.

Budget three things: weights (about 16 GB for 8B at fp16), KV cache, and activation/framework overhead. Llama 3.1 8B costs roughly 128 KiB of fp16 KV per token, so a single 128k-token context needs about 16 GB of cache on its own.

Start from a task evaluation, not a leaderboard. Pick the smallest size that clears your quality bar, then check the footprint: at bf16 the weights alone run roughly 16 GB at 8B, 140 GB at 70B and over 800 GB at 405B. Most teams land on 70B.