AI Engineering

The harness owns everything durable and enforceable: the session event log, the context builder that assembles each call, the tool router that actually executes calls, checkpoints, permissions and observability. The model only reasons over whatever context the harness hands it.

Quality decays as a window fills with stale turns and superseded tool output — context rot — well inside the nominal limit. The fix is context engineering: compact the session into a structured summary and reinitialize the window, keeping only what the remaining work needs.

You do not. Store the payload outside the window and pass a reference — a file path, object key or result ID — plus a short digest such as size, shape and the first lines. The agent reopens or searches it on demand if the digest is not enough.

Because the context window is derived state, not the system of record. An append-only event log lets the harness resume after a crash, rewind to a checkpoint, re-derive a compacted window, change the context policy without losing history, and reconstruct exactly what the model saw.

Once every vector is normalized to unit length, all three produce the same ordering: cosine equals the dot product, and squared L2 distance is 2 minus twice the cosine. Without normalization, the dot product favours long vectors and the three diverge.

m sets how many neighbour links each node keeps, efConstruction how hard the builder searches while inserting, and ef how wide the search is at query time. Raising them raises recall while costing memory, build time and query latency respectively.

Dense embeddings match meaning but smear rare exact tokens such as an identifier, a part number or a docket number; BM25 matches those literally. Reciprocal rank fusion merges the two result lists by rank position, so the systems' incomparable raw scores never have to be reconciled.

A selective filter applied after the search leaves almost nothing, because the top-k nearest vectors rarely satisfy a narrow predicate. Applied before the search, it deletes most nodes from the proximity graph, breaking the connectivity that greedy traversal depends on.

A cross-encoder reads the query and the candidate together, so it can judge fine-grained relevance that independently-encoded vectors cannot. It raises precision at the top of the list but cannot recover anything the first stage failed to retrieve, and its cost grows with the number of candidates scored.

A public benchmark is a shared, fixed dataset that ranks models against each other on a general capability. A task eval runs your own inputs through your own system and scores your own success criterion. Only the task eval predicts what your users will see.

LLM-as-judge means prompting a model with a rubric to score another model's output. The score is a noisy estimate, not truth: the judge has its own biases and blind spots, so it must be validated against human labels before anyone trusts the number.

Offline evaluation scores a fixed, frozen set of saved examples in a harness before release, so it is repeatable and cheap. Online evaluation measures real user traffic after release, where actual behaviour and business outcomes decide whether the change helped.

Sample by strata, not uniformly. Bucket traces by the dimensions you care about — request type, customer segment, known failure mode — then take a quota from each bucket, so rare but costly cases appear in numbers large enough to score.

The recurring four are position bias (order of presentation sways pairwise verdicts), verbosity bias (longer wins), self-preference (a judge rates its own family higher), and formatting halo (bullets and confident tone read as quality). Controls: swap orders, normalise or cap length, judge across families, and strip presentation from the rubric.

Explicit ratings cover a tiny, self-selected slice of traffic, often under 1% of sessions, and the people who bother to click skew angry or delighted. Implicit signals such as retries, abandonment and human edits are emitted by every session, so they carry the trend.

Reasoning (thinking) tokens, which are billed at generation rates but never shown, and the cache-write versus cache-read split of input tokens, which carry very different unit prices. Without those three extra counters, a dashboard cannot explain the invoice.

Make the turn one trace: a root run span, agent spans for each reasoning cycle, and child spans for every model call, tool call and retrieval. Each span's parent is whatever caused it and outlives it.

Score a fixed small sample of live sessions each hour, compare the rolling mean against a same-hour-of-week baseline rather than a flat threshold, and fire only when the deviation persists across several windows. Pin the scorer version, or scorer drift will masquerade as quality drift.

Open the turn's trace and split the question in two: did the availability fact ever reach the model? The retrieval and tool spans answer that; the model-call span's recorded input answers whether the fact was present and ignored.

A 429 means you crossed the provider's rate limit, usually a per-minute cap on requests or on tokens. Retrying instantly adds load to an already-throttled account; wait, honour any Retry-After, and back off exponentially with random jitter.

Streaming does not make generation faster. It makes the wait visible: the user starts reading the first sentence while the rest is still being produced, so an 18-second answer feels responsive instead of frozen.

A cascade only pays when the cheap tier answers enough traffic outright. Escalated requests are billed for both tiers plus the escalation check, so savings disappear once the escalation rate approaches the point where the small tier's spend no longer offsets the large tier's.

Track three separately: time to the first token the user can actually see, inter-token latency once the stream is flowing, and total time to the last token. Measure the first one on the client, at p50 and p99.

Use it when the job has a deadline but no single item needs a fast answer. Batch endpoints trade a completion window of up to a day for roughly half price and much higher throughput off the provider's spare capacity.

Different harms need different responses. Per-category scores let a system hard-block one category, route self-harm signals to a crisis flow, and only warn on mild insults. A single unsafe flag forces one action and one cutoff for every kind of harm.

Any path where model output causes a fetch is an egress channel: rendered image and link URLs, outbound tool or webhook calls, redirect targets, even hostname lookups. Controls only work once every such sink is enumerated.

Anything the model can influence can be widened by text that reaches it. Derive the tenant scope from the authenticated session and bind it at the query layer, so the model contributes only a search string and never the filter that decides which corpus is readable.

A model receives one flat token sequence. System text, user text and retrieved text carry no enforced privilege difference — only a learned tendency to prefer operator wording. Prompt-level separation shifts odds; the enforceable boundary has to live outside the model.

Input screening only sees what the user typed. A model can still emit harmful text from a benign prompt, from retrieved documents, or from another user's content pulled into context, so the output is a separate surface that needs its own classifier.

Parsing only proves the text is syntactically JSON. Validation proves the fields your code reads actually exist, with the right types and allowed values — so bad output fails loudly at the boundary instead of corrupting logic downstream.

The decoder tracks its position in a state machine compiled from the schema and, at every step, drives the logits of all tokens that cannot legally continue to negative infinity. An illegal token has zero probability, so it is never sampled.

JSON mode guarantees only that the text parses as JSON. A strict schema mode additionally guarantees the object conforms to your schema — required keys, types, enums. A custom grammar constrains any formal language, JSON or otherwise.

Prefer flat objects over deep nesting, enums over free text, and a single shape with a discriminator field over unions of alternative shapes. Give fields self-describing names and descriptions that state units and format. Compliance is a property of the schema's design, not only of the model.

Tool arguments are generated as text, so a model fills in a plausible order id, category or amount rather than leaving a field blank. Validate every call at the tool boundary — types, enum membership, units, and whether the referenced record actually exists — before executing anything.