skip to content

What do errors.log.enable and errors.log.include.messages do, and what is the security consideration with the latter?

level: middleimportance: should knowfreq 45%

answer

  1. log.enable: log the error context
  2. include.messages: log key+value contents
  3. both default false
  4. include.messages = PII/secret leak risk
  5. logging is separate from DLQ

basics

~10 s

errors.log.enable=true logs each failed record's error context to the Connect worker log. errors.log.include.messages=true additionally logs the record's key/value contents. Including messages can leak sensitive payload data into logs.

solid answer

~50 s

These configs control error logging, independent of the DLQ. errors.log.enable (default false) turns on logging of failures during processing: with it on, each tolerated/failed operation writes a log entry to the Connect worker's log describing the error context — topic, partition, offset, stage, and exception. errors.log.include.messages (default false) goes further and includes the actual record key and value contents in those log lines. That is useful for debugging but is a real security/PII concern: payloads may contain personal data, credentials, or regulated information that you do not want persisted in plaintext worker logs (which are often shipped to centralized log stores with broad access). So enable errors.log.enable broadly for observability, but gate errors.log.include.messages carefully — typically only in dev or with log redaction. Logging is orthogonal to the DLQ: you can use either, both, or neither, and both only emit when an error actually occurs (under tolerance=all, or on the failing record under none before the task dies).

go deeper

for a junior

Know log.enable logs the error and include.messages also logs the record contents; both default off.

for a middle

Explain logging is separate from the DLQ, works for source+sink, and that include.messages risks leaking sensitive payloads.

for a senior

Discuss log aggregation/retention exposure, GDPR/PCI implications, and preferring an ACL-restricted DLQ over payload logging.

for a principal

Set org policy: observability via log.enable everywhere, payload visibility gated/redacted, recovery via ACL-controlled DLQ; treat error logs as a data-governance surface.

## Two independent logging switches Kafka Connect's error handling offers logging that is **separate from** the dead-letter queue. You can log without a DLQ and DLQ without logging; they answer different questions ('what happened, in my logs' vs 'give me the failed record back, in a topic'). ### errors.log.enable (default false) When `true`, Connect writes a log entry for each error encountered during processing to the **worker log** (e.g. `connect.log` / stdout). The entry includes the **error context**: the source topic/partition/offset, the processing stage (converter, transform, put), and the exception type and message. This gives operators visibility into *that* and *why* records are failing without inspecting a DLQ topic. ### errors.log.include.messages (default false) When `true` (and only meaningful when `errors.log.enable=true`), the log entry **additionally contains the record's key and value contents**. This makes debugging much easier — you can see the exact bad payload inline — but it also writes potentially sensitive data into logs. ## The security consideration Worker logs are frequently: - shipped to centralized aggregation (Splunk, ELK, Datadog) with broad team access, - retained for long periods, - stored unencrypted at the application layer. If record payloads contain **PII, secrets, tokens, or regulated data**, `errors.log.include.messages=true` leaks that data into all those places — a compliance (GDPR/PCI) and security exposure. Best practice: keep it **off** in production, or enable it only temporarily for debugging, or rely on the DLQ (which keeps payloads in a Kafka topic you can ACL-restrict) instead of log lines. ## When do these fire These log entries are emitted **only when an error occurs**: - Under `errors.tolerance=all`, every tolerated error can be logged. - Under `errors.tolerance=none`, the failing record's error is logged before the task fails. ## Relationship to the DLQ - **DLQ** = preserves the original record bytes in a Kafka topic for replay; sink-only; ACL-controllable. - **Logging** = human-readable diagnostics in worker logs; works for source and sink. Using both gives observability (logs) plus recoverability (DLQ).

  • How is errors.log.enable different from configuring a DLQ?
    Logging writes human-readable error context (and optionally payloads) to the worker log for observability; a DLQ republishes the original record bytes to a Kafka topic for recovery/replay. Logging works for source and sink; DLQ is sink-only and ACL-controllable.
  • Your records contain customer emails. Should you set errors.log.include.messages=true in production?
    No — it writes record payloads (including the PII) into worker logs that are often centrally aggregated and broadly readable. Prefer the DLQ (ACL-restricted topic), or enable include.messages only temporarily in a controlled environment.

saying these in an interview costs you the question

  • Saying errors.log.include.messages is on by default (it's false)
  • Claiming logging and DLQ are the same mechanism
  • Ignoring the PII/secret exposure of logging payloads
  • Thinking errors.log.enable also writes to a Kafka topic (it writes to the worker log)

context

resolving links…