Telemetry spend is doubling every quarter and legal wants user email addresses removed from span attributes. How would you use OpenTelemetry Collector processors — filter, transform (OTTL), attributes — and connectors to shape volume and scrub data, and what breaks if you get it wrong?

## Separate the two goals Cost shaping and data scrubbing get lumped together because both are "processors that remove things", but they have different placements, different owners and different failure modes. Decide them separately. ## Scrubbing: as early as possible The defensible position is that sensitive values never leave the machine that produced them. That argues for the **agent tier**: a node-local Collector applies the rule before anything crosses a network boundary, so a gateway compromise or a vendor misconfiguration cannot expose what was never sent. The tools: the `transform` processor runs OTTL statements scoped to a context (`resource`, `span`, `spanevent`, `metric`, `datapoint`, `log`), so you can `delete_key(attributes, "user.email")`, `replace_pattern(...)` to mask, or hash a value where you still need to correlate on it without reading it. The `attributes` processor offers a simpler delete/update/hash surface, and the `redaction` processor supports an allow-list of permitted keys plus value-pattern blocking. The design point that matters: prefer an **allow-list** of attribute keys to a deny-list. A deny-list is open-world — it stops the leaks you have already thought of, and every new instrumentation library or hand-written attribute is a fresh chance to add a field nobody reviewed. An allow-list is closed and owned by the config, at the cost of friction when teams add legitimate attributes. Whichever you pick, the rule is fleet-wide config, not something each team re-implements. Remember what scrubbing cannot reach: an email embedded in a span *name*, in a URL path captured as `http.route`, in a log message body, or in an exception stack trace. Those need their own statements, and high-cardinality identifiers in span names quietly wreck backend indexing too. ## Shaping volume: measure, then cut The first move is not a filter, it is a measurement. A `count` connector (or a filter running in a parallel pipeline to a `debug`/`nop` exporter) tells you how many records a candidate rule matches, so you can rank rules by bytes saved rather than by intuition. Cost concentrates in surprising places: a chatty health-check endpoint, a client library spanning every retry, a debug log level left on in one service, a metric labelled with a request id. The ordered menu of interventions, cheapest first in terms of information lost: 1. **Prune attributes**, not records. Fat, low-value attributes (full SQL text, whole request headers, base64 payloads) often dominate bytes. You keep every span and lose some detail. 2. **Drop genuinely worthless records.** Readiness/liveness probe spans, `DEBUG`-severity logs in production, metrics nobody queries. `filter` with OTTL conditions does this. 3. **Aggregate instead of storing.** A `spanmetrics` or `servicegraph` connector converts spans into rate/error/duration metrics and service topology — a tiny fraction of the bytes, computed over 100% of traffic. Then keep only a sample of raw traces as exemplars. This is the single biggest lever in most estates. 4. **Sample**, last, because it is the only step that removes evidence you might need. ## What breaks **Trace continuity.** Dropping spans by condition is not the same as dropping traces. Filter out a middleware's spans and every child of those spans becomes an orphan referencing a missing parent; backends render broken waterfalls, and some drop the orphans entirely. Drop whole traces (sampling) or drop leaf spans that have no children — never a span in the middle. **Metric series continuity.** A cumulative series carries a start timestamp and monotonically increasing values; consumers detect resets by watching for a value decrease or start-time change. Intermittently dropping data points from a cumulative series produces gaps that rate functions interpolate over, and dropping and later resuming a series can look like a reset. Filter metrics by *whole series*, deterministically, not by conditions that flip over time. **Downstream queries.** Deleting an attribute breaks every dashboard, alert rule and saved query that grouped or filtered by it — silently, because the query still runs and simply returns fewer or unlabelled series. Attribute removal needs the same change-management as an API break: announce, measure usage where the backend supports it, stage it. **Reproducibility of the past.** Volume rules change what the archive contains, so historical comparisons across a rule change are invalid. Record when each rule shipped where incident reviewers will see it. ## Governance Because this is central config that silently changes what everyone can see, treat it as a reviewed artifact: version-controlled, code-reviewed by the owning teams, rolled out gradually (agent tiers hit every node at once), and each rule carrying an owner and a rationale. Pair every drop rule with a counter of what it dropped, so "why is this missing?" has an answer that does not require reading the config. And revisit annually — rules written for a service that no longer exists quietly cost nothing and confuse everyone. ## The trap to name explicitly Aggressive shaping optimises the bill and degrades exactly the data you need during the incident that justifies the bill. State the trade-off in the terms the business understands: the goal is not minimum telemetry, it is the minimum spend that still answers the questions you will actually ask under pressure.