skip to content

Walk through the common built-in SMTs (InsertField, ReplaceField, MaskField, RegexRouter, TimestampRouter, ExtractField, Cast, Flatten) and what each does.

level: middleimportance: must knowfreq 65%

answer

  1. InsertField=add, ReplaceField=rename/drop, MaskField=redact
  2. RegexRouter+TimestampRouter rewrite the TOPIC
  3. ExtractField=unwrap to single field
  4. Cast spec=field:type; Flatten=dotted nesting
  5. include/exclude/renames (old: whitelist/blacklist)

basics

~20 s

InsertField adds a field; ReplaceField renames/drops fields; MaskField hides values; RegexRouter and TimestampRouter rewrite the target topic name; ExtractField pulls one field up to be the whole key/value; Cast changes field types; Flatten collapses nested structs into dotted top-level fields.

solid answer

~50 s

These are the workhorse built-in SMTs in `org.apache.kafka.connect.transforms`: - **InsertField** — adds a static or metadata field (timestamp, topic, partition, offset, static value) to key/value. - **ReplaceField** — `include`/`exclude` (whitelist/blacklist) fields, or `renames` to rename them. - **MaskField** — replaces a field's value with a null-equivalent or a configured `replacement`, for PII redaction. - **RegexRouter** — rewrites the topic name via `regex` + `replacement` (capture groups), e.g. stripping a prefix. - **TimestampRouter** — appends a formatted timestamp to the topic name, useful for time-based partitioned sinks. - **ExtractField** — replaces the whole key or value with one of its fields (un-wrapping a Struct). - **Cast** — converts field types (e.g. string→int32), or the whole value for primitives. - **Flatten** — flattens nested structures into a single level with a delimiter, e.g. `address.city`. Most have `$Key`/`$Value` variants. RegexRouter/TimestampRouter act on the topic, not the payload.

go deeper

for a junior

Recognize each SMT name and its one-line purpose (add/rename/mask/route/extract/cast/flatten).

for a middle

Configure each correctly: ReplaceField modes, Cast spec syntax, RegexRouter capture groups, Flatten delimiter.

for a senior

Reason about ordering interactions and schema propagation; pick the right SMT for CDC un-wrapping and PII.

for a principal

Set standards for PII handling (MaskField vs upstream tokenization) and topic-routing conventions across connectors.

All of these live in `org.apache.kafka.connect.transforms` and ship with Apache Kafka (no extra plugin needed). Most are split into `$Key` and `$Value` inner classes; the router SMTs are exceptions because they rewrite the **topic name** rather than the record's key or value. ## Field-shaping SMTs - **InsertField** (`InsertField$Value`): inserts a field. Sources can be **static** (`static.field` + `static.value`) or **record metadata** via `timestamp.field`, `topic.field`, `partition.field`, `offset.field`. Useful to stamp lineage like `ingestedAt` or `sourceTopic`. - **ReplaceField** (`ReplaceField$Value`): three modes — `include` (keep only these fields), `exclude` (drop these fields), and `renames` (a comma list of `old:new` pairs). Common for dropping internal columns or renaming `id`→`user_id`. (Older Kafka used `whitelist`/`blacklist`, now `include`/`exclude`.) - **MaskField** (`MaskField$Value`): blanks listed `fields`. By default replaces with the type's zero value (empty string, 0, false); with `replacement` you set a literal masked value. Core PII tool, but note it produces a fixed value — it is **not** reversible hashing. - **ExtractField** (`ExtractField$Key`/`$Value`): given `field`, replaces the entire key or value with just that field's content. Classic use: a CDC value is a Struct `{id, name}` but the sink needs the bare `id` as the key — `ExtractField$Key` with `field=id`. - **Cast** (`Cast$Value`): `spec` like `field1:int32,field2:string`, or a bare type for a primitive whole-value. Supports int8/16/32/64, float32/64, boolean, string. Handy when a source emits strings that a typed sink needs as numbers. - **Flatten** (`Flatten$Value`): collapses nested Structs/Maps into one level, joining names with `delimiter` (default `.`), so `{address:{city:X}}` → `{address.city:X}`. Needed for sinks (e.g. JDBC) that cannot represent nesting. ## Routing SMTs (rewrite the topic) - **RegexRouter**: applies `regex` to the topic and substitutes `replacement` (with `$1` capture groups). Example: `regex=(.*)\.public\.(.*)`, `replacement=$2` to strip a CDC schema prefix. The downstream topic/partition is recomputed. - **TimestampRouter**: builds a new topic name from `topic.format` (default `${topic}-${timestamp}`) and `timestamp.format` (a `SimpleDateFormat` like `yyyyMMdd`). Used so a sink writes daily-named topics/objects. ## Practical notes - Order matters: e.g. Cast before Flatten, or RegexRouter after field edits. - Schemaful records get both **schema and value** transformed; schemaless `Map` records just transform the value. - Routing SMTs change which topic a record is associated with, which affects sink table/partition mapping downstream — but for a **source** connector they affect the destination Kafka topic.

  • Which of these SMTs do NOT have $Key/$Value variants, and why?
    RegexRouter and TimestampRouter. They operate on the record's topic name rather than its key or value, so a Key/Value split makes no sense for them.
  • You need to turn a CDC value Struct into a bare scalar key. Which SMT and config?
    ExtractField$Key with `field=<the field name>`. It replaces the entire key with that single field's value, un-wrapping the Struct.

saying these in an interview costs you the question

  • Claiming MaskField hashes or encrypts values — it replaces them with a fixed/zero value and is not reversible.
  • Thinking RegexRouter edits the payload — it only rewrites the topic name.
  • Forgetting that ReplaceField's modern option names are include/exclude/renames (whitelist/blacklist deprecated).

context

resolving links…