skip to content

Tokenizers & Chat Templates

AutoTokenizer turns text into input_ids and an attention mask, and apply_chat_template renders a chat with the model's own special tokens. Interviewers probe it because silent bugs start here.

on this pageshow

questions

6

In Hugging Face Transformers, what does calling a tokenizer on a batch with padding, truncation and return_tensors='pt' return, and what is the attention_mask for?

level: juniorimportance: must knowfreq 40%

answer

  1. a dict-like result, not a list
  2. BatchEncoding holding ids and a mask
  3. 1 for real tokens, 0 for pads
  4. padding and truncation both off by default
  5. model_max_length when max_length is unset

basics

~10 s

It returns a BatchEncoding: a dict-like object whose input_ids and attention_mask are tensors of one shape. The attention_mask is 1 on real tokens and 0 on padding, so the model ignores the padded positions.

solid answer

~40 s

`tokenizer(texts, padding=True, truncation=True, return_tensors="pt")` returns a `BatchEncoding` holding `input_ids` and `attention_mask` (plus `token_type_ids` for BERT-style models), all PyTorch tensors of shape `(batch, seq_len)`. `padding=True` means `'longest'`: every row is padded to the longest row in the batch; `'max_length'` pads to a fixed length instead. `truncation=True` cuts to `max_length`, or to the tokenizer's `model_max_length` when `max_length` is unset. Both are off by default, so with `return_tensors` set a ragged batch cannot become a tensor and the call raises a `ValueError`. The `attention_mask` is 1 for real tokens and 0 for padding; pass it with `model(**batch)` so padded positions do not change the real tokens' outputs.

go deeper

for a junior

Name the two keys you get back, input_ids and attention_mask, and say what a 0 in the mask means. Know that padding and truncation are off until you ask for them.

for a middle

Explain longest against max_length padding, where truncation takes its limit from when max_length is unset, and why the mask must be passed with model(**batch).

for a senior

Show you have debugged it: a checkpoint with no model_max_length that never truncates, a max_length ignored by padding=True, outputs that shift when the mask is dropped.

for a principal

Talk about making preprocessing explicit and reviewable: fixed padding and truncation settings in shared code, checked against each checkpoint's limits, rather than trusting per-checkpoint defaults.

## What the call returns A **tokenizer** in Hugging Face Transformers is the object `AutoTokenizer.from_pretrained(...)` gives you for a model. Calling it on text runs the whole encoding path in one step: split the text into tokens, map each token to an integer **id** from the model's vocabulary, add the model's **special tokens** (`add_special_tokens=True` is the default), and, if asked, pad, truncate and convert the result to tensors. The result is a **`BatchEncoding`**, a dictionary-like object. You read it by key (`batch["input_ids"]`) or by attribute (`batch.input_ids`), and you can unpack it straight into a model call. Its keys are the inputs the model expects: - **`input_ids`** — the token ids, one row per input text. - **`attention_mask`** — 1 where a position holds a real token, 0 where it holds padding. - **`token_type_ids`** — only for models that use them, such as BERT, to mark which sentence of a pair each token belongs to. ```python from transformers import AutoTokenizer tokenizer = AutoTokenizer.from_pretrained("google-bert/bert-base-uncased") batch = tokenizer( ["A short one.", "A noticeably longer second sentence goes here."], padding=True, truncation=True, return_tensors="pt", ) print(batch["input_ids"].shape) # (2, length of the longer row) print(batch["attention_mask"]) # trailing zeros on the short row ``` `return_tensors="pt"` asks for PyTorch tensors; `"np"` asks for NumPy arrays. Without it you get plain Python lists, which may have different lengths. ## Padding: making a ragged batch rectangular Texts tokenize to different lengths, but a tensor needs one length per row. **Padding** appends the tokenizer's `pad_token` id to the short rows. | `padding=` | What happens | |---|---| | `False` / `"do_not_pad"` (default) | No padding; rows keep their own lengths | | `True` / `"longest"` | Pad every row to the longest row in this batch | | `"max_length"` | Pad every row to `max_length` (or `model_max_length`) | Two details catch people. First, with padding off and `return_tensors` set, a ragged batch raises a `ValueError` that suggests `padding=True` and `truncation=True`. Second, `padding=True` together with `max_length` but no truncation logs a warning that `max_length` is ignored; use `padding="max_length"` for fixed-length rows. `pad_to_multiple_of=8` rounds the padded length up, which some GPUs prefer. ## Truncation and max_length **Truncation** removes tokens beyond a limit. Truncation is off by default. - `truncation=True` (strategy `"longest_first"`) cuts to `max_length`; when `max_length` is unset it uses the tokenizer's **`model_max_length`**, read from the checkpoint's tokenizer configuration. - If the checkpoint never set `model_max_length`, the tokenizer stores a huge sentinel value and `truncation=True` silently cuts nothing. - Passing only `max_length`, with padding and truncation left alone, switches truncation on, for backward compatibility. - Without truncation, an input longer than `model_max_length` only logs a warning that the sequence is longer than the model's maximum; the model then fails or misbehaves at forward time. - `truncation_side` defaults to `"right"`, so the end of the text is what gets dropped. Truncated tokens are gone from the output entirely. They are not kept and masked. ## Sentence pairs Some tasks feed the model two texts at once, such as a question and a passage. Pass both lists, `tokenizer(questions, passages, truncation="only_second", padding=True, return_tensors="pt")`, and each pair is encoded into one row with the model's separator tokens between the two texts. The truncation strategy decides what gets cut: `"longest_first"` (what `True` means) trims whichever text is longer, token by token, while `"only_second"` trims only the passage and keeps the question whole. For BERT-style models the `token_type_ids` then mark which text each token came from. ## The attention mask The **attention mask** tells the model which positions to attend to. Padding exists only to make the batch rectangular, so it must not influence the real tokens; the model reads the mask and gives padded positions no attention weight. Pass the whole `BatchEncoding` with `model(**batch)` so the mask travels with the ids. Feeding `input_ids` alone makes the model treat pad ids as text, and the outputs for the short rows change. Special tokens such as `[CLS]` and `[SEP]` are real tokens and get a 1. If you need to know which positions are special tokens, ask for `return_special_tokens_mask=True`, a separate output. ## Mistakes interviewers listen for 1. Setting `return_tensors="pt"` on a batch without `padding`, then being surprised by the `ValueError`. 2. Passing only `input_ids` to the model and dropping the mask. 3. Assuming `truncation=True` always enforces a limit, when the checkpoint may define none. 4. Confusing the attention mask with the special-tokens mask or with `token_type_ids`. 5. Moving only `input_ids` to the GPU; `batch.to(model.device)` moves every tensor in the `BatchEncoding` at once.

  • What changes if you pass padding='max_length', max_length=128 instead of padding=True?
    Every row is padded to exactly 128 tokens whatever the batch holds, which gives fixed tensor shapes but spends compute on padding. It does not cut longer rows: with truncation left unset they stay longer than 128 and the tensor conversion fails, so pair it with `truncation=True`.
  • Why does the output include token_type_ids for some checkpoints and not for others?
    A tokenizer returns the inputs its model was built to take. BERT-style tokenizers add `token_type_ids`, which mark the first and second sentence of a pair; most decoder-only tokenizers return only `input_ids` and `attention_mask`. `return_token_type_ids=True` or `False` overrides the default.

  • The attention_mask marks special tokens rather than padding.
  • Setting return_tensors='pt' pads the batch automatically.
  • truncation=True always cuts input to 512 tokens.
  • Passing input_ids alone is enough; the mask is optional decoration.
  • Truncated tokens stay in the batch with a mask value of 0.
open as a page

In Hugging Face Transformers, what do apply_chat_template's add_generation_prompt and continue_final_message do, and when do you set each?

level: middleimportance: must knowfreq 28%

basics

~20 s

add_generation_prompt=True appends the template's assistant-turn header so the model replies as the assistant; use it at inference, not for training data. continue_final_message=True leaves the last message open so the model continues a prefill. Combining them raises.

open as a page

Batched generation with a Transformers decoder-only model returns garbled text for every prompt except the longest — which tokenizer settings cause it, and how do you fix them?

level: seniorimportance: should knowfreq 20%

basics

~20 s

The tokenizer padded on the right, its class default, so shorter prompts end in pad tokens and generate() continues from them. Load it with padding_side='left', give it a pad_token, and pass the attention_mask to generate().

open as a page

Tokenizer code written for Transformers v4 passes use_fast=False, calls encode_plus and treats apply_chat_template's result as ids — what changes under v5, and what replaces each?

level: seniorimportance: should knowfreq 8%

basics

~10 s

Transformers v5 drops the slow/fast split: AutoTokenizer picks a named backend, ignores use_fast, and takes backend='sentencepiece' to force SentencePieceBackend. encode_plus gives way to calling the tokenizer, and apply_chat_template returns a BatchEncoding, not bare ids.

open as a page

With a Hugging Face Transformers tokenizer, how do you map each token back to its character span in the original text, for example to highlight entities?

level: middleimportance: nice to knowfreq 9%

basics

~10 s

Pass return_offsets_mapping=True: the BatchEncoding gains offset_mapping, one (start, end) character pair per token, with (0, 0) for added special tokens. It comes from the Rust-backed TokenizersBackend; word_ids() and char_to_token() add word and character lookups.

open as a page

In Hugging Face Transformers, user text containing a literal special-token string such as '<|im_end|>' reaches a chat model — how is it tokenized, and what does split_special_tokens change?

level: seniorimportance: nice to knowfreq 5%

basics

~20 s

By default the tokenizer matches the string as the real special token, so user text can end a turn or forge a role marker. split_special_tokens=True encodes it as plain text, but set globally it also splits the template's markers.

open as a page