skip to content

What are Kafka delegation tokens (KIP-48), and what problem do they solve for distributed workers and jobs?

level: seniorimportance: should knowfreq 30%

answer

  1. KIP-48: broker-issued shared secret after you authenticate
  2. tokenId=username, HMAC=password over SCRAM, tokenauth=true
  3. kafka-delegation-tokens.sh create/renew/expire/describe
  4. delegation.token.secret.key shared across brokers
  5. max.lifetime hard cap; avoid shipping keytabs to executors

basics

~20 s

Delegation tokens are lightweight shared-secret credentials a client first authenticates (e.g. via Kerberos or OAUTHBEARER), then requests from the broker. Workers/tasks use the token to authenticate to Kafka without distributing the original credential, simplifying secret distribution in distributed jobs.

solid answer

~50 s

Delegation tokens (KIP-48) let an already-authenticated principal obtain a short-lived, broker-issued shared secret that it can hand to its distributed workers. A Spark/Flink driver or Connect worker authenticates once with its strong credential (Kerberos keytab, mTLS, or OAUTHBEARER), then calls `createDelegationToken` (CLI: `kafka-delegation-tokens.sh --create`). The broker returns a tokenId + HMAC. Workers authenticate with SASL/SCRAM using the tokenId as username and the HMAC as password — no need to ship keytabs or client secrets to every executor. Tokens are stored and replicated across brokers (historically via ZooKeeper, now KRaft metadata) so any broker can verify them. They have a configurable max lifetime, can be renewed up to that cap, and can be expired/revoked immediately via the CLI. This bounds the blast radius: a leaked worker token is short-lived and individually revocable, unlike a shared keytab.

go deeper

for a junior

Know it's a lightweight token you can hand to worker tasks so they don't need the main credential.

for a middle

Explain create-via-CLI after authenticating, and that workers use it as a SCRAM username/password.

for a senior

Cover lifecycle (create/renew/expire), shared secret key, max-lifetime cap, and the impersonation/secret-distribution rationale.

for a principal

Design token issuance and rotation for large Spark/Flink/Connect fleets; reason about revocation blast radius, broker key management, and OAUTHBEARER-to-delegation-token bridging.

## The distribution problem In a distributed framework — Spark, Flink, MapReduce, Kafka Connect — a driver/coordinator spawns many worker tasks across a cluster, each needing to talk to Kafka. Distributing the *primary* credential (a Kerberos keytab or an OAuth client secret) to every executor is risky: it multiplies exposure, and revoking it kills everything. **Delegation tokens (KIP-48)** solve this. ## What a delegation token is A delegation token is a **shared-secret credential issued by the Kafka brokers** to an already-authenticated principal. It consists of: - a **tokenId** (public identifier) - an **HMAC** (the secret) - metadata: owner principal, renewers, issue/expiry/max-lifetime timestamps Workers authenticate using **SASL/SCRAM-SHA-256/512**, supplying the `tokenId` as the username and the `HMAC` as the password, with the JAAS config setting `tokenauth=true`. So the token rides on the SCRAM mechanism — no separate transport. ## Lifecycle and the CLI The `kafka-delegation-tokens.sh` tool drives the lifecycle: - `--create --renewer-principal User:... --max-life-time-period <ms>` — mint a token (the requester must already be authenticated). - `--renew --hmac <hmac> --renew-time-period <ms>` — extend, up to the max lifetime. - `--expire --hmac <hmac> --expiry-time-period -1` — revoke immediately. - `--describe` — list tokens. Broker configs: - `delegation.token.secret.key` (legacy `delegation.token.master.key`) — the broker secret used to generate/verify token HMACs; must be identical across all brokers. - `delegation.token.max.lifetime.ms` (default 7 days) — hard cap a token can live even with renewals. - `delegation.token.expiry.time.ms` (default 1 day) — default validity before renewal needed. - `delegation.token.expiry.check.interval.ms` — how often expired tokens are purged. ## Why it helps workers/jobs (impersonation) The driver authenticates **once** with its strong identity, then requests tokens that act on its behalf. Tasks use tokens, so: - The keytab/secret never leaves the driver. - Tokens are **short-lived** and **individually revocable** — leaking one token is far less damaging than leaking a keytab. - Renewal lets long jobs keep going up to the max lifetime without re-distributing the primary credential. ## Important constraints and pitfalls - **You must already be authenticated** to create a token, and (historically) tokens could not be created over a connection that itself authenticated *with* a delegation token (no token-from-token chaining over delegation-token auth). - **Master/secret key consistency**: all brokers must share `delegation.token.secret.key`. A mismatch makes tokens unverifiable on some brokers. - **TLS required in practice**: the HMAC is a bearer-style secret; use SASL_SSL. - **Max lifetime is a hard ceiling**: renewal cannot push a token past `delegation.token.max.lifetime.ms`; long jobs must obtain new tokens. - **Relationship to OAUTHBEARER**: a principal authenticated via OAUTHBEARER can mint delegation tokens, which workers then redeem over SCRAM — a common pattern to avoid handing OAuth client secrets to every executor.

  • Which SASL mechanism do workers use to authenticate with a delegation token, and how are the credentials mapped?
    SASL/SCRAM (SHA-256 or SHA-512). The tokenId is supplied as the username and the HMAC as the password, with tokenauth=true in the JAAS config.
  • Can a delegation token be renewed indefinitely?
    No. Renewal extends validity only up to delegation.token.max.lifetime.ms (default 7 days). Past that hard ceiling the client must create a fresh token.

saying these in an interview costs you the question

  • Saying delegation tokens replace the need to authenticate at all — you must already be authenticated to create one.
  • Claiming tokens authenticate via the OAUTHBEARER mechanism — workers redeem them over SCRAM.
  • Forgetting brokers must share delegation.token.secret.key.
  • Believing renewal can extend a token past its max lifetime.

context

resolving links…