In PyRIT, how does earlier conversation history reach the endpoint on turn five of a multi-turn run, and what changes when the prompt target wraps a service that keeps its own server-side session?
answer
- chat target replays memory each call
- conversation id keys memory
- context grows, cost grows, front truncation
- session id target = context is server state
- single-shot target = repeated first turns
basics
~20 sFor a chat-completion style endpoint the target replays the stored turns: it reads the conversation out of memory and sends the whole message list each call. If the service keeps its own session and accepts only the newest message, the history lives on the server, so the tool's copy and the real context can drift apart.
solid answer
~50 sPyRIT keys every request and response to a conversation id in memory. A chat-style target replays that conversation on each send, so turn five is one call carrying five turns; the context grows every turn and so does the per-call token bill. A service with its own session inverts that. The target holds a session identifier and posts only the new message; the accumulated context is server state you can neither read nor rebuild. Resuming a run against a session that has since expired restarts the endpoint's context while your transcript still shows five turns, and replaying a stored conversation against a fresh session is not the same experiment. Stored history is what you sent, not what was in the window. A single-shot target that ignores history is worse still: a multi-turn escalation strategy run through one is just repeated first turns.
go deeper
Knows the run stores conversations and that history is sent back to a chat endpoint, without necessarily reasoning about server-side sessions.
Explains replay from memory keyed by conversation id, the growing context and cost, and the difference when the endpoint holds its own session.
Diagnoses truncation and expired sessions from transcripts, and refuses to report a multi-turn result when the target could not guarantee the model saw the earlier turns.
Requires runs to record which history mode the target used so results are comparable across engagements, and rejects cross-tool comparisons that mix replay and session targets.
The mechanism deserves to be stated exactly, because it decides whether "a five-turn escalation" describes anything at all. ## Replay targets (the stateless case) PyRIT's memory stores every request and response piece keyed by a `conversation_id`, each tagged with a role - user or assistant. A chat-style target such as `OpenAIChatTarget` is **stateless on the wire**, so on every send it reads that conversation back out of memory and rebuilds the full message list: system message if any, then user/assistant alternating, then the new prompt. Turn five is therefore *one* HTTP call carrying roughly nine or ten messages. Nothing about the endpoint remembers anything; the illusion of a dialogue is manufactured locally, from your own log, on each call. Two consequences follow directly. - **First, cost.** Input tokens per call grow with turn number, and the total input billed over a run is the sum of all the prefixes - roughly **quadratic in the turn budget**. If each turn adds about 500 tokens, turn one bills ~500 input tokens and turn five bills ~4,500; doubling `max_turns` from 5 to 10 roughly quadruples the input side of the bill, not doubles it. This is why a turn budget in a multi-turn strategy is a cost control and not merely a stopping rule, and why long runs are the ones that surprise a finance owner. - **Second, the context limit.** Once the replayed conversation exceeds the model's window, something gives. Some providers error; some silently window or truncate, and truncation is usually from the front - exactly where a multi-turn escalation put its groundwork. Your transcript still shows ten turns of careful setup; the model may have seen four. The compensating virtue is **fidelity**: the stored conversation *is* what was sent, so a run can be paused, resumed, re-read and re-scored later against identical input. ## Session-holding targets (the stateful case) Many real applications hand you a thread, session or conversation identifier and keep the context themselves. Wrapped as a target - typically a hand-configured `HTTPTarget` carrying that id in a header or body field - the target posts only the newest message. Now there are **two histories**: yours, in memory, and theirs, on their server. They can diverge without any error surfacing. - Session TTL expiry mid-run resets their side to empty. - Server-side summarisation or a sliding window silently drops early turns. - A load-balanced backend may not find the session at all. - A resumed run may quietly start from a fresh, empty context while your log continues at turn six. ## Single-shot and non-chat targets Some surfaces have no dialogue: a completion endpoint, an image or audio generation call, a one-shot classification API. Point a multi-turn escalation strategy at one and you get a run that looks busy and is a set of repeated first turns. ## Where the number misleads "Objective achieved on turn 6 of 10" means two different things in the two modes. - **Under replay** it is a claim you can defend, because you can show the exact message list that was in the request. - **Under a session target** it is a claim about *what you sent*, not about what was in the window - and if the session expired at turn 3, the "escalation" was a cold-start prompt that happened to work. Comparing attack-success rates across a replay run and a session run, or across engagements where nobody recorded which mode was used, is a denominator swap dressed as a trend. Front truncation produces the mirror error in the other direction: a run that reports ten turns of pressure and delivered four, then gets read as evidence the model resists sustained escalation. ## What I check - Pull one conversation out of memory and count the messages actually sent on the last turn. - Read the provider's reported **prompt-token count** per call and confirm it rises with turn number - a flat curve on a supposedly replaying target means history is not going. - For a session target, compare the **session TTL** against the run's wall-clock time, and confirm whether a resumed run re-establishes or reuses the session. - Sample the late-turn responses for the tell of a lost context: an assistant answering as if the topic were new. - And record the **history mode** next to every number, because "the same strategy against the same model" is not the same experiment in the two modes.
- Why does a turn budget in a multi-turn PyRIT run act as a cost control rather than just a stop condition?Because a replaying target resends the whole conversation each turn, so per-turn input tokens grow with turn number and the last turns cost far more than the first.
- You resume a stopped run against a session-holding endpoint and the attempts suddenly all fail. What do you suspect first?That the session expired, so the endpoint restarted with an empty context while the tool kept sending late-stage turns that only make sense after the earlier setup.
saying these in an interview costs you the question
- Assuming any target automatically gives the endpoint the full dialogue
- Comparing multi-turn results across a replay target and a session-holding target as if they were the same experiment
- Not noticing front truncation at the context limit in long runs
- Running a multi-turn escalation strategy through a single-shot endpoint and reporting turn counts