X-Ray Distributed Tracing

## Two attachment points, one difference that matters The X-Ray SDKs give you two ways to hang your own data on a segment or subsegment: ```python from aws_xray_sdk.core import xray_recorder segment = xray_recorder.current_segment() segment.put_annotation("tenant_id", "acme") # indexed, searchable segment.put_metadata("request_body", payload) # stored, not searchable ``` The two land in different places in the segment document — an `annotations` object and a `metadata` object — and X-Ray treats them completely differently on ingest. Annotations are extracted and **indexed**; metadata is stored as an opaque blob attached to the segment. ## What indexing buys you X-Ray's console and the `GetTraceSummaries` API take a **filter expression**. Built-in fields are always available: ``` service("checkout-api") { fault } http.status = 500 AND responsetime > 2 ``` Indexed annotations extend that vocabulary with your own domain: ``` annotation.tenant_id = "acme" AND responsetime > 2 annotation.order_id = "A-1234" ``` That is the whole point. Distributed tracing is only operationally useful if you can get from a *business* fact — this customer, this order, this experiment arm — to the trace. Without an annotation you are reduced to guessing at a time window and scanning. Annotation values are restricted to **string, number or boolean**, because they have to be indexable. Keys are restricted to alphanumerics and underscore. An object will not be accepted as an annotation value. ## What metadata is for Metadata takes arbitrary JSON — nested objects, arrays, long strings — organised into namespaces. You cannot search it, but it is fully visible when someone opens the trace, which makes it the right home for: - the request or response body (redacted), - the inputs to a pricing or routing decision, - feature-flag evaluation results, - anything you would otherwise have logged "just in case". The SDK's default namespace is `default`; the `aws` namespace is reserved for data the SDK itself records. ## Why you cannot annotate everything The obvious junior instinct is to annotate every field, since annotations are strictly more useful. Two things stop you. First, **the index is bounded**. X-Ray indexes up to 50 annotations per trace; beyond that the extra ones are still stored but you cannot rely on searching them. Fifty across an entire multi-service trace is a small budget once every service starts annotating freely. Second, **high-cardinality keys are a trap in the other direction**. An annotation whose value is unique per request — a full request id, a timestamp — is perfectly valid and genuinely useful for pinpoint lookup, but it is useless for grouping. Keys you want to *group* by (tenant, region, api version) should be low-cardinality; keys you want to *look up* by (order id) can be unique. Deciding which of the two a key is, before you add it, is the design conversation. A sensible convention is to reserve annotations for a short, reviewed list agreed across services — so that `annotation.tenant_id` means the same thing everywhere — and let metadata absorb everything else. ## Subsegments count too Both calls work on subsegments as well as segments, and annotations on a subsegment are indexed for the trace. Annotating the subsegment is often better hygiene: the fact that *this particular downstream call* was for tenant `acme` is more precise than a segment-level annotation when a request fans out. ## The rule that overrides both Neither annotations nor metadata are an appropriate place for **secrets or personal data**. Trace data is retained by X-Ray (traces are kept for 30 days) and is readable by anyone with X-Ray read access in the account, which is typically a much wider group than has access to production data stores. Annotating `customer_email` gives you a beautiful search experience and a compliance problem; annotate a pseudonymous customer id instead and join to the real identity in a system that has the right controls. If a field is sensitive and you truly need it in the trace, redact or hash it before it is attached. ## Reading it back When you find the trace, the console shows annotations in a dedicated section of the segment view and metadata beneath it; `BatchGetTraces` returns the whole segment document including both. So metadata costs you nothing at retrieval time — only at search time, where it does not exist.