In Hugging Face Transformers, what does calling a tokenizer on a batch with padding, truncation and return_tensors='pt' return, and what is the attention_mask for?
answer
- a dict-like result, not a list
- BatchEncoding holding ids and a mask
- 1 for real tokens, 0 for pads
- padding and truncation both off by default
- model_max_length when max_length is unset
basics
~10 sIt returns a BatchEncoding: a dict-like object whose input_ids and attention_mask are tensors of one shape. The attention_mask is 1 on real tokens and 0 on padding, so the model ignores the padded positions.
solid answer
~40 s`tokenizer(texts, padding=True, truncation=True, return_tensors="pt")` returns a `BatchEncoding` holding `input_ids` and `attention_mask` (plus `token_type_ids` for BERT-style models), all PyTorch tensors of shape `(batch, seq_len)`. `padding=True` means `'longest'`: every row is padded to the longest row in the batch; `'max_length'` pads to a fixed length instead. `truncation=True` cuts to `max_length`, or to the tokenizer's `model_max_length` when `max_length` is unset. Both are off by default, so with `return_tensors` set a ragged batch cannot become a tensor and the call raises a `ValueError`. The `attention_mask` is 1 for real tokens and 0 for padding; pass it with `model(**batch)` so padded positions do not change the real tokens' outputs.
go deeper
Name the two keys you get back, input_ids and attention_mask, and say what a 0 in the mask means. Know that padding and truncation are off until you ask for them.
Explain longest against max_length padding, where truncation takes its limit from when max_length is unset, and why the mask must be passed with model(**batch).
Show you have debugged it: a checkpoint with no model_max_length that never truncates, a max_length ignored by padding=True, outputs that shift when the mask is dropped.
Talk about making preprocessing explicit and reviewable: fixed padding and truncation settings in shared code, checked against each checkpoint's limits, rather than trusting per-checkpoint defaults.
## What the call returns A **tokenizer** in Hugging Face Transformers is the object `AutoTokenizer.from_pretrained(...)` gives you for a model. Calling it on text runs the whole encoding path in one step: split the text into tokens, map each token to an integer **id** from the model's vocabulary, add the model's **special tokens** (`add_special_tokens=True` is the default), and, if asked, pad, truncate and convert the result to tensors. The result is a **`BatchEncoding`**, a dictionary-like object. You read it by key (`batch["input_ids"]`) or by attribute (`batch.input_ids`), and you can unpack it straight into a model call. Its keys are the inputs the model expects: - **`input_ids`** — the token ids, one row per input text. - **`attention_mask`** — 1 where a position holds a real token, 0 where it holds padding. - **`token_type_ids`** — only for models that use them, such as BERT, to mark which sentence of a pair each token belongs to. ```python from transformers import AutoTokenizer tokenizer = AutoTokenizer.from_pretrained("google-bert/bert-base-uncased") batch = tokenizer( ["A short one.", "A noticeably longer second sentence goes here."], padding=True, truncation=True, return_tensors="pt", ) print(batch["input_ids"].shape) # (2, length of the longer row) print(batch["attention_mask"]) # trailing zeros on the short row ``` `return_tensors="pt"` asks for PyTorch tensors; `"np"` asks for NumPy arrays. Without it you get plain Python lists, which may have different lengths. ## Padding: making a ragged batch rectangular Texts tokenize to different lengths, but a tensor needs one length per row. **Padding** appends the tokenizer's `pad_token` id to the short rows. | `padding=` | What happens | |---|---| | `False` / `"do_not_pad"` (default) | No padding; rows keep their own lengths | | `True` / `"longest"` | Pad every row to the longest row in this batch | | `"max_length"` | Pad every row to `max_length` (or `model_max_length`) | Two details catch people. First, with padding off and `return_tensors` set, a ragged batch raises a `ValueError` that suggests `padding=True` and `truncation=True`. Second, `padding=True` together with `max_length` but no truncation logs a warning that `max_length` is ignored; use `padding="max_length"` for fixed-length rows. `pad_to_multiple_of=8` rounds the padded length up, which some GPUs prefer. ## Truncation and max_length **Truncation** removes tokens beyond a limit. Truncation is off by default. - `truncation=True` (strategy `"longest_first"`) cuts to `max_length`; when `max_length` is unset it uses the tokenizer's **`model_max_length`**, read from the checkpoint's tokenizer configuration. - If the checkpoint never set `model_max_length`, the tokenizer stores a huge sentinel value and `truncation=True` silently cuts nothing. - Passing only `max_length`, with padding and truncation left alone, switches truncation on, for backward compatibility. - Without truncation, an input longer than `model_max_length` only logs a warning that the sequence is longer than the model's maximum; the model then fails or misbehaves at forward time. - `truncation_side` defaults to `"right"`, so the end of the text is what gets dropped. Truncated tokens are gone from the output entirely. They are not kept and masked. ## Sentence pairs Some tasks feed the model two texts at once, such as a question and a passage. Pass both lists, `tokenizer(questions, passages, truncation="only_second", padding=True, return_tensors="pt")`, and each pair is encoded into one row with the model's separator tokens between the two texts. The truncation strategy decides what gets cut: `"longest_first"` (what `True` means) trims whichever text is longer, token by token, while `"only_second"` trims only the passage and keeps the question whole. For BERT-style models the `token_type_ids` then mark which text each token came from. ## The attention mask The **attention mask** tells the model which positions to attend to. Padding exists only to make the batch rectangular, so it must not influence the real tokens; the model reads the mask and gives padded positions no attention weight. Pass the whole `BatchEncoding` with `model(**batch)` so the mask travels with the ids. Feeding `input_ids` alone makes the model treat pad ids as text, and the outputs for the short rows change. Special tokens such as `[CLS]` and `[SEP]` are real tokens and get a 1. If you need to know which positions are special tokens, ask for `return_special_tokens_mask=True`, a separate output. ## Mistakes interviewers listen for 1. Setting `return_tensors="pt"` on a batch without `padding`, then being surprised by the `ValueError`. 2. Passing only `input_ids` to the model and dropping the mask. 3. Assuming `truncation=True` always enforces a limit, when the checkpoint may define none. 4. Confusing the attention mask with the special-tokens mask or with `token_type_ids`. 5. Moving only `input_ids` to the GPU; `batch.to(model.device)` moves every tensor in the `BatchEncoding` at once.
- What changes if you pass padding='max_length', max_length=128 instead of padding=True?Every row is padded to exactly 128 tokens whatever the batch holds, which gives fixed tensor shapes but spends compute on padding. It does not cut longer rows: with truncation left unset they stay longer than 128 and the tensor conversion fails, so pair it with `truncation=True`.
- Why does the output include token_type_ids for some checkpoints and not for others?A tokenizer returns the inputs its model was built to take. BERT-style tokenizers add `token_type_ids`, which mark the first and second sentence of a pair; most decoder-only tokenizers return only `input_ids` and `attention_mask`. `return_token_type_ids=True` or `False` overrides the default.
- The attention_mask marks special tokens rather than padding.
- Setting return_tensors='pt' pads the batch automatically.
- truncation=True always cuts input to 512 tokens.
- Passing input_ids alone is enough; the mask is optional decoration.
- Truncated tokens stay in the batch with a mask value of 0.