How do repetition, frequency and presence penalties differ in LLM sampling?

## The failure they exist for Autoregressive models fall into loops. Once a phrase appears, it becomes part of the context conditioning the next step, which raises its own probability, which makes it appear again. The feedback is self-reinforcing and it is worse at low temperature, because a sharpened distribution keeps taking the same leading token. "The ancient ruins, the ancient ruins, the ancient ruins" is the canonical shape. The instinct to fix this by lowering temperature is exactly backwards. Loops are a symptom of the model over-committing to its leading candidate; sharpening makes it commit harder. The penalty family attacks the cause directly by modifying scores of tokens that already occurred. ## Presence penalty A presence penalty applies a constant subtraction to the logit of any token that has appeared in the text so far. It is binary in its trigger: appeared once or appeared fifty times, the penalty is identical. That makes it a *topic-broadening* control rather than a loop-breaking one. It pushes the model toward vocabulary it has not used yet, which is useful when you want a list of genuinely distinct items rather than five rephrasings of the first. It will not, on its own, stop a hard loop, because a constant subtraction is a constant the loop can outrun. ## Frequency penalty A frequency penalty subtracts an amount proportional to how often the token has already occurred. The second occurrence is discouraged a little, the tenth a lot. This escalation is exactly what a loop needs: each repetition raises the cost of the next one until the loop cannot sustain itself. It is the right knob for degenerate repetition, and typically the wrong knob for content variety, where presence is gentler. In APIs that expose both, they are independent and additive. ## Repetition penalty The repetition penalty from open-weight tooling works multiplicatively rather than additively. For a token that has already appeared, a positive score is divided by the penalty factor and a negative score is multiplied by it, so in both directions the score moves down. Values live near 1.0 — 1.0 means off, and small increments like 1.05-1.15 are the usual working range, with anything much higher visibly degrading fluency. Two things follow from the multiplicative form. First, the effect size depends on the magnitude of the logit, so it is not uniform across tokens. Second, it does not escalate with count in the way a frequency penalty does; a token seen once and a token seen ten times are scaled by the same factor. ## Where they hurt Every member of this family operates on tokens, with no notion of meaning or structure. That is the whole problem. In code generation, identifiers and keywords are *supposed* to repeat. A penalty strong enough to break prose loops will make the model rename a variable halfway through a function or reach for a synonym of a language keyword, producing code that does not compile. In structured output, repeated field names, brackets and quotes are mandatory. Penalties directly fight the format, and the failures are ugly: mismatched braces, invented key names, truncated arrays. In long-form narrative, a character's name legitimately recurs on every page. A frequency penalty accumulated over thousands of tokens will eventually push the model to stop using it, and the prose degrades into pronoun soup. The practical rule: default all of these to off, apply the smallest value that resolves an observed symptom, scope them to the generation type that has the symptom, and never leave them enabled on code or schema-constrained paths. ## Neighbouring controls Two related controls are worth distinguishing. **No-repeat n-gram blocking** is a hard constraint rather than a soft penalty: it forbids any n-gram from appearing twice, which reliably kills loops but also forbids legitimate repeated phrases, so it is rarely appropriate for open-ended text. **Stop sequences** are not penalties at all — they terminate generation when a given string is produced, which is how you keep a chat model from writing the other speaker's turn. And as with the rest of the sampling surface, availability is not universal: penalties remain live in open-weight serving stacks, while several hosted frontier reasoning endpoints have narrowed or removed the sampling parameters they accept.