A solution architecture must satisfy a security NFR such as 'all sensitive data encrypted at rest and in transit, with defense in depth against a compromised application server.' What architectural decisions does this drive, and what does it cost elsewhere in the design?

## An absolute rather than a budget Security NFRs differ from most other non-functional requirements in that they're often expressed as an absolute ('all sensitive data must be encrypted') rather than a percentile or budget, but translating them into architecture still requires the same discipline: - identifying the concrete **mechanisms**; - the **layers** they apply to; - and the **cost paid elsewhere** in the system. ## Encryption in transit Encryption in transit means every network hop carrying sensitive data uses TLS (or an equivalent transport-layer encryption), and the common architectural mistake is stopping at the public edge — terminating TLS at a load balancer or API gateway and then running plaintext HTTP between internal services 'because it's a private network.' A defense-in-depth security posture treats the internal network as untrusted too, because internal networks get compromised more often than architects assume when the NFR was written casually. The mechanism for doing this properly at scale is usually a service mesh or sidecar proxy pattern that automatically wraps service-to-service calls in mutual TLS (mTLS) without every service team hand-rolling certificate handling, because manual TLS setup per service is exactly the kind of thing that gets skipped under deadline pressure. ## Encryption at rest Encryption at rest means data is encrypted on physical/logical storage — database volumes, backups, object storage, and any derived caches or search indexes that might also hold a copy of sensitive fields. The mechanism is typically delegated to a managed key-management service (KMS): the storage layer encrypts with a data key, that data key is itself encrypted (enveloped) by a master key held in the KMS, and the KMS enforces access control and audit logging on who/what can request key material. This matters architecturally because 'encryption at rest' is trivially satisfiable at the top-level datastore but easy to miss in secondary copies — a nightly backup, a data warehouse ETL pipeline, or a search index built from the same records — each of which needs the same guarantee independently, and a security review that only checks the primary database misses these. ## Assume the app server is already compromised 'Defense in depth against a compromised application server' is the mechanism that most directly shapes the architecture beyond just 'turn on encryption,' because it assumes a specific breach scenario — an attacker gaining code execution on an app instance — and asks what they can reach from there. The architectural response layers several independent controls so no single one being defeated (which is the working assumption) gives full access: - **network segmentation** so the app tier's security group/firewall rules can reach only the specific database port and no lateral access to other internal systems; - **database credentials scoped to least privilege** (the app's service account can read/write the tables it needs, not administer the database or read unrelated tenants' data); - **secrets fetched at runtime** from a secrets manager rather than stored in environment variables or config files that persist in the compromised instance's filesystem or process memory indefinitely; - and, for the most sensitive fields, **application-level (field-level) encryption** so that even a compromised app server with valid database credentials only retrieves ciphertext for the most critical fields unless it also has the specific decryption key, which can be scoped even more narrowly than general DB access. ## Blast radius, rather than a checkbox Why this exists as an architectural discipline rather than a checkbox: security NFRs are stated as absolutes because a partial implementation (encrypt the primary DB but not backups; TLS at the edge but not internally; broad database credentials with encryption bolted on top) provides much less real protection than the stated requirement implies, while looking compliant on a surface-level audit. The 'compromised app server' framing specifically forces architects to think in terms of blast radius rather than perimeter — assuming the perimeter will eventually be breached and designing so that breach doesn't cascade into full data exposure. ## What it costs elsewhere The costs are concrete and worth naming honestly rather than treating security as free. - **TLS everywhere**, including internal mesh traffic, adds real latency from handshake overhead and CPU cost from encryption/decryption, which matters when internal call chains are already latency-budgeted tightly. - **Field-level encryption** on frequently-queried columns can make those columns unindexable or unqueryable in their encrypted form, forcing either application-side filtering (worse performance) or accepting that field can't be efficiently searched at all, sometimes requiring a redesign of the query pattern. - **Key rotation and secrets-manager integration** add operational surface area and a new class of outage risk (an app that can't fetch a secret at startup fails to boot), which needs its own resilience design (caching secrets with a sane TTL, graceful degradation). - **Least-privilege credential scoping**, done thoroughly, means many more distinct service accounts and roles to provision and maintain rather than one shared broad-access account, which is genuinely more operational work. ## How it fails in production Failure modes in production include: 1. TLS terminated at the edge with plaintext internal hops, discovered only during a security audit or, worse, after a lateral-movement incident. 2. Encrypted primary databases with an unencrypted nightly backup or an unencrypted downstream analytics copy of the same data. 3. Overly broad database credentials on the app server meaning a code-execution compromise immediately grants full data access regardless of how well encryption was implemented elsewhere. 4. Secrets baked into container images or long-lived environment variables that persist in the compromised instance far longer than a short-lived, rotated credential would. ## A worked example on a healthcare records API A worked example: a healthcare records API satisfies its security NFR with mTLS between all internal services via a service mesh, KMS-backed encryption at rest on the primary database and its backups, field-level encryption specifically on the most sensitive clinical-notes column (accepting that column can no longer be full-text searched, and building a separate, access-controlled search index instead), a database service account scoped to only the tables the API needs with no administrative rights, and secrets fetched at boot from a secrets manager with a short-lived cache and automatic rotation — accepting the added internal mTLS overhead per hop and the operational cost of managing the additional service accounts and key rotation schedule as the deliberate price of the defense-in-depth requirement.