Your threat-modeling assistant is a hosted LLM: what must you settle before pasting an unreleased design?

## The decision you are actually making Asking a hosted model to draft a threat model means handing a third party two things: the design of a system, and — in the reply — a structured enumeration of where that system is weak. For an unreleased product, say a bidding service for an ad auction that has not shipped, the design document is trade-secret intellectual property whose value is partly that competitors do not have it. This is a **confidentiality decision with a named adversary set**, and it should be modeled the way you would model any other data flow that crosses an organisational trust boundary. Who is in that adversary set? Not the internet attacker your other models are about: - The **vendor** itself, under whatever terms actually apply to your account. - Its **sub-processors** — the hosting, logging and abuse-review systems the data passes through. - **Staff with support or abuse-review access**, under whatever internal controls you cannot see. - **Legal process** reaching retained content. - A **future compromise** of the vendor, applied retroactively to everything retained. - Your own **account sprawl**: a personal account used on a work laptop puts the document somewhere your organisation cannot even enumerate. ## The output is the more sensitive artifact Teams reason carefully about the design document and casually about the reply. That is backwards. A design document describes what the system does; a threat model lists, in priority order, what is wrong with it and has not been fixed yet. Anything that treats the input as sensitive must treat the output — and the chat transcript containing it — at least as strictly, including where it is stored afterwards, who it is forwarded to, and whether it ends up pasted into a ticket system with wider access than the model deserves. ## What to settle, in order **1. The contract, as written.** Retention period, whether inputs are used to improve models, whether that differs by tier, processing region, sub-processor list, deletion semantics, and what "delete" means for logs and abuse-review copies. "They said they don't train on it" is not a control; the terms that bind your account are. This is a procurement and legal question, and the honest engineering answer in an interview is that you escalate it rather than guess. **2. A classification rule.** Decide once, organisationally, which classes of material may leave the tenant: public and internal-general, yes; unreleased product design, regulated personal data, credentials, and completed threat models, no — or only to an approved in-tenant service. A rule stated in terms of *document class* is enforceable and teachable; "use judgement" is neither. **3. A sanctioned alternative.** If the answer is only "no", engineers will paste anyway, on personal accounts, and you will have lost even the ability to know. The strongest version of this answer names what people may use instead: a model running inside your own tenant or on your own infrastructure, or an approved **abstraction rule**. ## Abstraction that keeps the draft useful Much of a threat model's value survives redaction, because the model reasons over shapes. Keep: the topology, the trust boundaries, the direction of each flow, the authentication scheme in generic terms, and the data *classes* at each store. Strip: product and project names, real hostnames and identifiers, the auction or pricing logic that constitutes the actual secret, customer names, volumes and revenue figures. A draft against "a public bidding API, an internal matching service and a store of counterparty records" is nearly as good as one against the real document — and it is no longer a disclosure of an unreleased product. The cost is honest: abstraction removes precisely the business detail that would have made the draft less generic, and it takes an experienced person a few minutes to do properly. Say both things out loud rather than pretending redaction is free. ## Governance that survives contact with engineers - **Decide by class, not by request.** Per-case approval queues get routed around. - **Make the sanctioned path the easy path.** If the approved in-tenant assistant is slower and worse, the policy is decorative. - **Log the decision, not the content.** Recording that a model for a given service was drafted with assistance, and on which platform, is enough for later audit. - **Watch for shadow use rather than assuming compliance.** Egress telemetry and, more usefully, asking teams without penalty, tells you whether the rule is real. - **Revisit when tiers or vendors change.** The terms you approved are attached to a specific contract, and both sides of that change. ## The framing that lands in an interview Threat-model the threat modeling. Draw the boundary between your organisation and the assistant vendor, name the asset crossing it — unreleased design, then the weakness list coming back — name the adversaries above, and pick controls: contractual, technical (in-tenant model), or procedural (abstraction). That answer shows you apply the method to your own tooling instead of exempting it.