When designing an SMT chain (e.g. ExtractField + Cast + Flatten + RegexRouter with predicates), why does ordering matter and what are common pitfalls?
answer
- each SMT sees prior SMT's output (incl. schema)
- rename/Flatten changes field names → reorder dependents
- router changes topic → TopicNameMatches sees new topic
- one predicate per SMT; negate for skip
- Filter early but after needed inputs; tombstone safety
basics
~20 sSMTs run in listed order and each consumes the previous one's output, so an SMT that renames or removes a field changes what later SMTs can see. You must order field-shaping before routing or extraction that depends on those fields, mind schema changes, predicate scoping, and tombstone handling.
solid answer
~50 sBecause the chain is a left-to-right pipeline where each SMT transforms the output of the prior one, ordering determines correctness. If `ReplaceField` renames or drops a field that a later `ExtractField`, `Cast`, or routing SMT references by name, the later SMT must see the post-rename name — order accordingly. `Flatten` changes field names to dotted paths, so anything referencing nested fields must run before or after Flatten consistently. Routing SMTs (`RegexRouter`, `TimestampRouter`) change the topic; if a downstream predicate uses `TopicNameMatches`, place it to match the intended (pre- or post-route) topic. Schema-changing SMTs (Cast, Flatten, ReplaceField) alter the record schema, so a strict-schema sink or a later schema-dependent SMT can break. Other pitfalls: tombstones (null values) flowing into value SMTs, predicates only gating one SMT, and `Filter` placement. Test the chain with `connect-runtime`'s validate endpoint and representative records.
go deeper
Know the chain runs in order and later SMTs see earlier changes.
Order field renames/flatten before SMTs that reference those fields; handle Cast-before-sink needs.
Reason about schema evolution, predicate-after-router scoping, and tombstone safety across the chain.
Establish chain-design standards, validate via the REST endpoint, and decide SMT-chain vs stream-processing boundaries for the org.
## The mental model An SMT chain is a **deterministic pipeline**: record → SMT₁ → SMT₂ → … → SMTₙ → out. Each SMT's input is the **previous SMT's output**, including any schema changes. So 'ordering' is really 'what does each stage see?'. ## Why order changes behavior 1. **Name dependencies.** `ReplaceField` with `renames=id:user_id` changes the field name. A later `ExtractField` with `field=user_id` works only if it runs *after* the rename; `field=id` works only *before*. Get this wrong and the SMT silently no-ops or errors on a missing field. 2. **Flatten renames everything.** `Flatten` turns `address.city` into a top-level `address.city` field. A `MaskField` on `city` must reference the flattened name `address.city` and therefore must run *after* Flatten — or mask `city` while still nested, *before* Flatten. 3. **Routing affects topic-based predicates.** `RegexRouter`/`TimestampRouter` rewrite the topic. If a later transform is gated by `TopicNameMatches`, it sees the **new** topic. Decide whether you want to match the original or routed topic and place the router accordingly (predicates evaluate against the record at that point in the chain). 4. **Type dependencies.** `Cast` to a numeric type may be required before a downstream SMT or sink that assumes numbers. Conversely, casting after Flatten lets you target a flattened field name. 5. **Schema evolution.** Cast, Flatten, ReplaceField, InsertField all change the schema. A sink with strict schema compatibility (or a later SMT validating field presence) can break if the schema arrives in an unexpected shape. ## Predicate scoping pitfalls - A predicate gates exactly **one** SMT. If you want several SMTs to share a condition, each needs its own `predicate=` (and possibly `negate`). Forgetting one leaves an SMT running unconditionally. - Because predicates see the record **at that chain position**, a `TopicNameMatches` placed after a router matches the routed topic — a subtle source of 'my mask didn't apply' bugs. ## Tombstone and null pitfalls Value SMTs can NPE or no-op on tombstones (null value). For CDC/compacted topics, consider a `RecordIsTombstone` predicate with `negate` to skip value transforms on tombstones, or ensure each SMT is null-safe. ## Filter placement `Filter` + predicate drops records; place it **early** to avoid wasting later SMT work on records you'll discard, but **after** any SMT whose output the predicate needs. ## Operational guidance - Keep chains short and readable; long chains are hard to reason about and debug. - Use the Connect REST `PUT /connector-plugins/{type}/config/validate` to catch config errors before deploy. - Test with representative records including tombstones, missing fields, and both schemaful and schemaless payloads. - Document the intended order and the contract each stage assumes — ordering bugs are silent. - Consider whether heavy reshaping belongs in a stream processor instead of a long SMT chain.
- You Flatten then try to MaskField on `city` and nothing is masked. Why?After Flatten the field is named `address.city`, not `city`. MaskField references field names literally, so it found no `city` and no-oped. Either reference `address.city` post-Flatten or mask before flattening.
- A predicate using TopicNameMatches isn't firing for a routed record. What's likely wrong?The predicate is evaluated after the RegexRouter, so it sees the rewritten topic, not the original. Either match the new topic name or move the gated SMT before the router.
- How do you avoid wasting SMT work on records you'll drop?Place the Filter (with its predicate) as early in the chain as possible — but still after any SMT whose output the filtering predicate depends on.
saying these in an interview costs you the question
- Assuming SMT order doesn't matter because they're 'independent' — each consumes the prior output.
- Referencing a pre-rename / pre-flatten field name in a later SMT.
- Expecting one predicate to gate multiple SMTs at once.
- Forgetting that topic-based predicates after a router see the rewritten topic.
- Running value SMTs over tombstones without null safety.