When designing a system, when is RSA encryption the right asymmetric primitive in Java, and what alternatives or pitfalls should drive that decision?
answer
- RSA = wrap a small secret / PKI interop
- ECC: 256-bit EC ≈ 3072-bit RSA, smaller+faster
- always hybrid + AES-GCM for bulk; pin algorithms
- keys in HSM/KMS, plan rotation + cert-bound authenticity
- post-quantum: RSA & ECC both fall to Shor - track PQC
basics
~20 sUse RSA when you must encrypt a small secret (like a symmetric key) to someone's public key. For most data, use hybrid encryption, and for modern systems consider elliptic-curve options (ECDH/ECIES) which are smaller and faster.
solid answer
~40 sRSA encryption is appropriate for protecting a small payload - in practice, wrapping a symmetric key in an envelope scheme - to a recipient identified by a public key, especially where RSA interop is already required (existing PKI, X.509 certs, legacy peers). For new green-field systems, elliptic-curve primitives are usually preferable: ECDH/ECIES give equivalent security with far smaller keys (a 256-bit EC key ≈ 3072-bit RSA), faster operations, and smaller ciphertext. Key design decisions: mandate RSA-OAEP-SHA256 (never PKCS1v1.5 or NoPadding), use >=2048-bit (ideally 3072) keys, always go hybrid for bulk data with an authenticated symmetric mode (AES-GCM), keep private keys in an HSM/KMS, plan rotation, and consider post-quantum migration since RSA and ECC both fall to a sufficiently large quantum computer. Pin algorithms explicitly rather than relying on provider defaults.
go deeper
Knows RSA is for small secrets/keys and that bigger data uses a symmetric key; aware faster alternatives exist.
Can say RSA suits key-wrapping/interop and that elliptic-curve options are smaller/faster for new systems.
Compares RSA-OAEP vs ECDH/ECIES on key size, speed, and interop, and enforces hybrid + authenticated encryption + algorithm pinning.
Sets org crypto policy: primitive choice by threat model/lifetime, key sizes, HSM/KMS custody, rotation, FIPS constraints, public-key authenticity, and a post-quantum migration plan.
## The decision frame Choosing an asymmetric primitive is an **architecture** decision, not a coding detail. The questions: *what* are you protecting (a key, or bulk data?), *who* holds the keys, *what interop* is required, and *what threat model and lifetime* apply. ## What RSA is genuinely good for - **Encrypting a small secret to a public key** - the canonical use is **key wrapping** in hybrid encryption (encrypt data with AES, wrap the AES key with RSA-OAEP). - **Interop with existing PKI**: X.509 certificates, TLS endpoints, S/MIME, and many enterprise systems are RSA-based. If you must talk to them, RSA is the pragmatic choice. - It is **widely supported** and battle-tested when used correctly (OAEP, big keys). ## Why elliptic curve is often the better default for new systems **Elliptic-curve cryptography (ECC)** achieves the same security with much smaller keys because the underlying math is harder per bit: - A **256-bit EC key ≈ a 3072-bit RSA key** in security strength. - Smaller keys -> faster key generation, smaller ciphertext/signatures, less bandwidth and storage. - For encryption you typically use **ECIES** (Integrated Encryption Scheme) or, more commonly, **ECDH** (Elliptic-Curve Diffie-Hellman) **key agreement** to derive a shared symmetric key, then encrypt with AES. Java exposes ECDH via `KeyAgreement.getInstance("ECDH")`. Note: plain Java/JCE does not ship a one-call `ECIES` `Cipher`; you assemble ECDH + a KDF + AES (or use a provider like Bouncy Castle). So for green-field: **prefer ECDH/ECIES**; reach for RSA mainly for interop. ## Non-negotiable RSA hygiene (if you do use it) - **Padding:** RSA-OAEP with SHA-256. Never `PKCS1Padding` (Bleichenbacher), never `NoPadding` (textbook RSA). - **Key size:** >= 2048 bits; **3072** for longer-lived data (NIST puts 2048 at ~112-bit strength, fine through ~2030; 3072 = 128-bit). - **Hybrid always** for anything but a tiny secret, with an **authenticated** symmetric mode (AES-GCM) - confidentiality without integrity is a frequent mistake. - **Pin algorithms explicitly** in transformation strings and `OAEPParameterSpec`; do not rely on provider defaults that vary across JDKs/providers/FIPS modes. ## Operational concerns a principal owns - **Private-key custody:** store in an **HSM/KMS**, expose a handle (sign/decrypt as a service) rather than raw key bytes; this also enables auditing and centralized rotation. - **Rotation & revocation:** plan key rotation, certificate lifetimes, and how old ciphertext is re-wrapped to new keys. - **Authenticity of public keys:** encryption to an *attacker's* public key gives them, not you, the plaintext - bind public keys to identities via certificates / a trust store. - **FIPS / compliance:** if the platform must be FIPS-validated, your algorithm and provider choices are constrained. - **Post-quantum horizon:** both RSA and ECC are broken by a large quantum computer (Shor's algorithm). For data with a long confidentiality lifetime, track NIST PQC (e.g. ML-KEM/Kyber) and consider hybrid classical+PQ key establishment. This is increasingly a real planning input, not a theoretical aside. ## Bottom line RSA-OAEP for **small-secret/interop** cases; **hybrid** for bulk; **ECDH/ECIES** as the modern default for new systems; explicit algorithm pinning, HSM-backed keys, rotation, and a post-quantum plan as the surrounding architecture.
- Why might a new system prefer ECDH over RSA?Equivalent security at much smaller key sizes (256-bit EC ≈ 3072-bit RSA), faster operations, smaller ciphertext/bandwidth - all favorable unless RSA interop is required.
- Why does post-quantum risk matter for an RSA design decision today?Shor's algorithm breaks RSA and ECC on a large quantum computer. Data with long confidentiality lifetime is 'harvest now, decrypt later' exposed, so long-lived secrets should plan PQC (e.g. ML-KEM) migration.
saying these in an interview costs you the question
- Defaulting to RSA for everything when ECDH/ECIES would be smaller and faster
- Picking RSA key size by habit without a strength/lifetime rationale
- Confidentiality without integrity (no authenticated mode in the hybrid layer)
- Assuming a plain 'ECIES' Cipher ships in the JDK (it generally doesn't)
- Ignoring post-quantum risk for long-lived secrets
- Trusting a public key without certificate/identity verification